feat: raw packet capture, browse, and classification (v0.13.0)

Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.

Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
  composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
  reuses the decoder's per-hex cache (no second decode), one row per observer
  reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
  cleanup runs regardless of capture so disabling drains the table. Raw-packet
  observers retained in the is_observer recompute union.

API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
  cache key, and channel-visibility redaction (restricted-channel packets are
  returned metadata-only, not hidden, so pagination counts stay stable).

Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
  cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
  precedes Members; new packets icon + colour.

Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
  types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
  control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
  safety net.

Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
  exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
  Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
  on and the row has a stored hash.

Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Louis King
2026-06-12 22:40:31 +01:00
parent deb5af508c
commit 76f3dfa7eb
50 changed files with 3637 additions and 539 deletions
+17
View File
@@ -238,6 +238,20 @@ DATA_RETENTION_DAYS=30
# Applies to both event data and node cleanup
DATA_RETENTION_INTERVAL_HOURS=24
# -------------------
# Raw Packet Capture
# -------------------
# Capture every inbound packets-feed packet into the raw_packets table.
# Off by default. In Docker Compose this is derived from FEATURE_PACKETS, so
# setting FEATURE_PACKETS=true enables both capture and the Packets page.
# RAW_PACKET_CAPTURE_ENABLED=false
# Days to retain raw packets. Defaults to DATA_RETENTION_DAYS when unset.
# The raw_packets table grows fastest of all; lower this on busy meshes or
# constrained storage. Retention runs regardless of capture being enabled, so
# disabling capture lets existing rows drain.
# RAW_PACKET_RETENTION_DAYS=30
# -------------------
# Node Cleanup Settings
# -------------------
@@ -497,6 +511,9 @@ NETWORK_ANNOUNCEMENT=
# FEATURE_PAGES=true
# FEATURE_CHANNELS=true
# FEATURE_RADIO_CONFIG=true
# Packets page is OFF by default. Enabling it also drives raw-packet capture
# on the collector via Compose (RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS}).
# FEATURE_PACKETS=false
# -------------------
# Contact Information
+11 -1
View File
@@ -681,7 +681,7 @@ Key variables:
- `WEB_AUTO_REFRESH_SECONDS` - Auto-refresh interval in seconds for list pages (default: `30`, `0` to disable)
- `WEB_DEBUG` - Enable debug mode in the web dashboard (default: `false`)
- `TZ` - Timezone for web dashboard date/time display (default: `UTC`, e.g., `America/New_York`, `Europe/London`)
- `FEATURE_DASHBOARD`, `FEATURE_NODES`, `FEATURE_ADVERTISEMENTS`, `FEATURE_MESSAGES`, `FEATURE_MAP`, `FEATURE_MEMBERS`, `FEATURE_PAGES`, `FEATURE_CHANNELS`, `FEATURE_RADIO_CONFIG` - Feature flags to enable/disable specific web dashboard pages (default: all `true`). Dependencies: Dashboard auto-disables when all of Nodes/Advertisements/Messages are disabled. Map auto-disables when Nodes is disabled.
- `FEATURE_DASHBOARD`, `FEATURE_NODES`, `FEATURE_ADVERTISEMENTS`, `FEATURE_MESSAGES`, `FEATURE_MAP`, `FEATURE_MEMBERS`, `FEATURE_PAGES`, `FEATURE_CHANNELS`, `FEATURE_RADIO_CONFIG` - Feature flags to enable/disable specific web dashboard pages (default: all `true`). Dependencies: Dashboard auto-disables when all of Nodes/Advertisements/Messages are disabled. Map auto-disables when Nodes is disabled. `FEATURE_PACKETS` - enables the Packets page (default: `false`); in Compose it also drives `RAW_PACKET_CAPTURE_ENABLED` on the collector.
- `NETWORK_DOMAIN` - Network domain name (default: none)
- `NETWORK_NAME` - Network display name (default: `MeshCore Network`)
- `NETWORK_CITY` - Network city location (default: none)
@@ -763,6 +763,16 @@ When enabled, the collector automatically deletes event data older than the rete
- Telemetry
- Trace paths
- Event logs
- Raw packets (using `RAW_PACKET_RETENTION_DAYS`)
**Raw Packet Capture:**
| Variable | Description |
|----------|-------------|
| `RAW_PACKET_CAPTURE_ENABLED` | Capture every packets-feed packet into `raw_packets` (default: false). In Compose, derived from `FEATURE_PACKETS`. |
| `RAW_PACKET_RETENTION_DAYS` | Days to retain raw packets (default: falls back to `DATA_RETENTION_DAYS`). |
When enabled, the collector writes one `RawPacket` row per observer reception from the LetsMesh `packets` feed, reusing the decode the normalizer already performs (decoder per-hex cache). The `/packets` API serves these with channel-visibility redaction (restricted-channel packets returned metadata-only) and a role-aware Redis cache key. The web Packets page is gated behind `FEATURE_PACKETS` (default off). Raw-packet retention cleanup runs whenever data cleanup runs, regardless of the capture flag.
**Node Cleanup:**
+55
View File
@@ -461,6 +461,28 @@ Notification that routing path to a node has changed.
---
### Per-Payload-Type Classifications
Packets on the LetsMesh `packets` feed that no structured handler claims are classified by their MeshCore payload type rather than the generic `letsmesh_packet`. These are informational (logged to `events_log`, captured to `raw_packets`), with no dedicated table:
| Payload type | `event_type` | Notes |
|---|---|---|
| `0x00 REQ` | `req` | Encrypted request, metadata only |
| `0x01 RESPONSE` | `response` | Response with no structured content |
| `0x02 TXT_MSG` | `encrypted_direct` | Direct text that did not decrypt |
| `0x03 ACK` | `ack` | Acknowledgment |
| `0x04 ADVERT` | `advert` | Advert lacking identity metadata |
| `0x05 GRP_TXT` | `encrypted_channel` | Channel text with unknown key |
| `0x06 GRP_DATA` | `grp_data` | Group datagram |
| `0x07 ANON_REQ` | `anon_req` | Anonymous request |
| `0x08 PATH` | `path` | Returned path (unparsed) |
| `0x09 TRACE` | `trace` | Trace (unparsed) |
| `0x0A MULTIPART` | `multipart` | Fragment of a sequence (no reassembly) |
| `0x0B CONTROL` | `control` | Control packet not matched to contact/status |
| `0x0F RAW_CUSTOM` | `raw_custom` | Custom-encrypted, opaque |
`letsmesh_packet` remains as the safety-net label when the payload type cannot be resolved.
### Other Informational Events
The following events are logged but have varying or device-specific payloads:
@@ -478,6 +500,39 @@ The following events are logged but have varying or device-specific payloads:
---
## Raw Packets
When `RAW_PACKET_CAPTURE_ENABLED` is set (Compose derives it from `FEATURE_PACKETS`), the collector stores **every** inbound packet from the LetsMesh `packets` feed exactly as received, independent of how it is classified. One row is written per observer reception (no deduplication), reusing the decode the normalizer already performs.
**Database Table**: `raw_packets`
| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key |
| `observer_node_id` | FK `nodes.id` (SET NULL) | Receiving interface, from the MQTT topic public key |
| `packet_hash` | string(32) | LetsMesh packet hash; links to structured records, groups multi-observer receptions |
| `raw_hex` | text | On-air bytes from `payload["raw"]` |
| `packet_type` | integer | Wire packet type |
| `payload_type` | integer | Decoder payload type |
| `event_type` | string(50) | How the collector classified the packet (`advertisement`, `channel_msg_recv`, `letsmesh_packet`, …) |
| `channel_idx` | integer | `int(channelHash, 16)` for channel-message packets, else NULL; drives visibility redaction |
| `source_pubkey_prefix` | string(12) | Sender prefix from decoder `sourceHash` / `senderPublicKey` |
| `route_type` | string(20) | `flood`, `transport_flood`, `direct`, `transport_direct` |
| `path_len` | integer | Hop count |
| `snr` | float | Signal-to-noise ratio reported by the observer |
| `decoded` | JSON | Decoder summary (so detail views need no re-decode) |
| `received_at` | datetime | When received by the observing interface |
**Linking from structured tables**: `advertisements` and `messages` carry a nullable `packet_hash` (the LetsMesh wire hash) populated at ingest. Because the structured tables are deduplicated across observers, one row links to *all* the per-observer `raw_packets` sharing that hash — reachable via `GET /api/v1/packets?packet_hash=<hash>`. Only rows ingested while capture was enabled have a populated `packet_hash` (no backfill).
**REST API**: `GET /api/v1/packets`, `GET /api/v1/packets/{id}`
**`RawPacketRead` shape** mirrors the columns above plus a `redacted: bool` field and observer details (`observed_by`, `observer_name`, `observer_tag_name`). Channel-message packets on a channel above the requesting role's visibility level are returned with `redacted=true` and `raw_hex`/`decoded`/`source_pubkey_prefix` nulled; non-channel and visible-channel packets are returned in full. Responses are cached in Redis with a role-aware key.
Filters: `search` (hash/raw_hex/observer, bounded to a recent window when no `since` given), `event_type`, `packet_type`, `channel_idx`, `route_type`, `public_key`/`pubkey_prefix`, `observed_by`, `decryptable`, `min_snr`/`max_snr`, `min_path_len`/`max_path_len`, `redacted`, `since`/`until`, `sort`/`order`, `limit`/`offset`.
---
## Webhook Payload Format
All webhook events are wrapped in a standard envelope before being sent:
@@ -0,0 +1,117 @@
"""add raw_packets table
Revision ID: e9f0c4079540
Revises: 20260610_1200
Create Date: 2026-06-12 19:47:12.406423+00:00
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import sqlite
# revision identifiers, used by Alembic.
revision: str = "e9f0c4079540"
down_revision: Union[str, None] = "20260610_1200"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table(
"raw_packets",
sa.Column("observer_node_id", sa.String(), nullable=True),
sa.Column("packet_hash", sa.String(length=32), nullable=True),
sa.Column("raw_hex", sa.Text(), nullable=True),
sa.Column("packet_type", sa.Integer(), nullable=True),
sa.Column("payload_type", sa.Integer(), nullable=True),
sa.Column("event_type", sa.String(length=50), nullable=True),
sa.Column("channel_idx", sa.Integer(), nullable=True),
sa.Column("source_pubkey_prefix", sa.String(length=12), nullable=True),
sa.Column("route_type", sa.String(length=20), nullable=True),
sa.Column("path_len", sa.Integer(), nullable=True),
sa.Column("snr", sa.Float(), nullable=True),
sa.Column("decoded", sqlite.JSON(), nullable=True),
sa.Column("received_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("id", sa.String(), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("(CURRENT_TIMESTAMP)"),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("(CURRENT_TIMESTAMP)"),
nullable=False,
),
sa.ForeignKeyConstraint(
["observer_node_id"], ["nodes.id"], ondelete="SET NULL"
),
sa.PrimaryKeyConstraint("id"),
)
with op.batch_alter_table("raw_packets", schema=None) as batch_op:
batch_op.create_index(
batch_op.f("ix_raw_packets_channel_idx"), ["channel_idx"], unique=False
)
batch_op.create_index(
"ix_raw_packets_channel_idx_received_at",
["channel_idx", "received_at"],
unique=False,
)
batch_op.create_index(
batch_op.f("ix_raw_packets_event_type"), ["event_type"], unique=False
)
batch_op.create_index(
"ix_raw_packets_event_type_received_at",
["event_type", "received_at"],
unique=False,
)
batch_op.create_index(
batch_op.f("ix_raw_packets_observer_node_id"),
["observer_node_id"],
unique=False,
)
batch_op.create_index(
batch_op.f("ix_raw_packets_packet_hash"), ["packet_hash"], unique=False
)
batch_op.create_index(
batch_op.f("ix_raw_packets_packet_type"), ["packet_type"], unique=False
)
batch_op.create_index(
"ix_raw_packets_received_at", ["received_at"], unique=False
)
batch_op.create_index(
batch_op.f("ix_raw_packets_source_pubkey_prefix"),
["source_pubkey_prefix"],
unique=False,
)
batch_op.create_index(
"ix_raw_packets_source_pubkey_prefix_received_at",
["source_pubkey_prefix", "received_at"],
unique=False,
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("raw_packets", schema=None) as batch_op:
batch_op.drop_index("ix_raw_packets_source_pubkey_prefix_received_at")
batch_op.drop_index(batch_op.f("ix_raw_packets_source_pubkey_prefix"))
batch_op.drop_index("ix_raw_packets_received_at")
batch_op.drop_index(batch_op.f("ix_raw_packets_packet_type"))
batch_op.drop_index(batch_op.f("ix_raw_packets_packet_hash"))
batch_op.drop_index(batch_op.f("ix_raw_packets_observer_node_id"))
batch_op.drop_index("ix_raw_packets_event_type_received_at")
batch_op.drop_index(batch_op.f("ix_raw_packets_event_type"))
batch_op.drop_index("ix_raw_packets_channel_idx_received_at")
batch_op.drop_index(batch_op.f("ix_raw_packets_channel_idx"))
op.drop_table("raw_packets")
# ### end Alembic commands ###
@@ -0,0 +1,52 @@
"""add packet_hash to messages and advertisements
Revision ID: e8eb47c49062
Revises: e9f0c4079540
Create Date: 2026-06-12 21:11:17.664733+00:00
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "e8eb47c49062"
down_revision: Union[str, None] = "e9f0c4079540"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("advertisements", schema=None) as batch_op:
batch_op.add_column(
sa.Column("packet_hash", sa.String(length=32), nullable=True)
)
batch_op.create_index(
batch_op.f("ix_advertisements_packet_hash"), ["packet_hash"], unique=False
)
with op.batch_alter_table("messages", schema=None) as batch_op:
batch_op.add_column(
sa.Column("packet_hash", sa.String(length=32), nullable=True)
)
batch_op.create_index(
batch_op.f("ix_messages_packet_hash"), ["packet_hash"], unique=False
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("messages", schema=None) as batch_op:
batch_op.drop_index(batch_op.f("ix_messages_packet_hash"))
batch_op.drop_column("packet_hash")
with op.batch_alter_table("advertisements", schema=None) as batch_op:
batch_op.drop_index(batch_op.f("ix_advertisements_packet_hash"))
batch_op.drop_column("packet_hash")
# ### end Alembic commands ###
+6
View File
@@ -197,6 +197,10 @@ services:
- DATA_RETENTION_INTERVAL_HOURS=${DATA_RETENTION_INTERVAL_HOURS:-24}
- NODE_CLEANUP_ENABLED=${NODE_CLEANUP_ENABLED:-true}
- NODE_CLEANUP_DAYS=${NODE_CLEANUP_DAYS:-30}
# Raw packet capture (derived from FEATURE_PACKETS so one var drives both
# capture and the web Packets page). Retention defaults to DATA_RETENTION_DAYS.
- RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS:-false}
- RAW_PACKET_RETENTION_DAYS=${RAW_PACKET_RETENTION_DAYS:-${DATA_RETENTION_DAYS:-30}}
command: ["collector"]
healthcheck:
test: ["CMD", "meshcore-hub", "health", "collector"]
@@ -340,6 +344,8 @@ services:
- FEATURE_PAGES=${FEATURE_PAGES:-true}
- FEATURE_CHANNELS=${FEATURE_CHANNELS:-true}
- FEATURE_RADIO_CONFIG=${FEATURE_RADIO_CONFIG:-true}
# Packets page is off by default; enabling it also turns on collector capture.
- FEATURE_PACKETS=${FEATURE_PACKETS:-false}
command: ["web"]
healthcheck:
test:
+25
View File
@@ -16,6 +16,31 @@ The collector subscribes to packets published by [meshcore-packet-capture](https
- Decoder payload type `1` can map to native response events (`telemetry_response`, `battery`, `path_updated`, `status_response`) when decrypted structured content is available.
- `packet_type=5` packets are mapped to `channel_msg_recv`.
- `packet_type=1`, `2`, and `7` packets are mapped to `contact_msg_recv` when decryptable text is available.
- Packets that no structured handler claims are no longer all labelled `letsmesh_packet`. They are classified by their MeshCore payload type so the `event_type` is specific:
| Payload type | `event_type` |
|---|---|
| `0x00 REQ` | `req` |
| `0x01 RESPONSE` | `response` |
| `0x02 TXT_MSG` (undecryptable) | `encrypted_direct` |
| `0x03 ACK` | `ack` |
| `0x04 ADVERT` (no identity) | `advert` |
| `0x05 GRP_TXT` (unknown key) | `encrypted_channel` |
| `0x06 GRP_DATA` | `grp_data` |
| `0x07 ANON_REQ` | `anon_req` |
| `0x08 PATH` | `path` |
| `0x09 TRACE` | `trace` |
| `0x0A MULTIPART` | `multipart` |
| `0x0B CONTROL` | `control` |
| `0x0F RAW_CUSTOM` | `raw_custom` |
`letsmesh_packet` is retained only as a safety net for packets whose payload type cannot be resolved. Reaching these fallbacks for `TXT_MSG`/`GRP_TXT` means the payload did not decrypt, hence the `encrypted_*` labels (decryptable ones become `contact_msg_recv` / `channel_msg_recv`).
## Raw Packet Capture
- When `RAW_PACKET_CAPTURE_ENABLED` is set (Compose derives it from `FEATURE_PACKETS`), every packet on the `packets` feed is also stored verbatim in the `raw_packets` table — one row per observer reception, independent of structured classification. The `status` and `internal` feeds carry no on-air `raw` hex and are **not** captured as raw packets.
- Capture reuses the single decode the normalizer already performs (the decoder caches per raw hex), so it adds only an insert plus an observer upsert to the ingest path.
- The `/packets` API and Packets page apply channel-visibility rules: channel-message packets on a channel above the viewer's role are returned **metadata-only with the payload redacted** (`redacted=true`, `raw_hex`/`decoded` nulled), not hidden. Non-channel and visible-channel packets are returned in full.
## Channel Keys
@@ -0,0 +1,509 @@
# Raw Packets: Capture, Store, and Browse Decoded Wire Packets
## Summary
Add a first-class **Raw Packets** feature that captures every inbound MeshCore
packet exactly as it arrives over the LetsMesh `packets` feed, independent of how
the collector later classifies it. Today the collector decodes each packet and
routes it to a structured handler (advertisement, message, trace, telemetry,
contact); the original on-air bytes are then discarded for everything that
matches a structured handler, and only unclassified leftovers reach the
`events_log` table. There is no complete, searchable record of raw traffic.
This plan introduces a `raw_packets` table populated **unconditionally** at
ingest from the already-decoded packet, a `/packets` API with rich filtering and
channel-visibility-aware redaction, and a new SPA **Packets** page (disabled by
default) for display/search/filter. The structured tables remain derived views,
linkable back to raw packets by `packet_hash`.
## Background & Motivation
### Current State
1. The collector subscribes to three LetsMesh upload feeds
(`subscriber.py:512`): `<prefix>/+/+/packets`, `/status`, `/internal`.
2. Each `packets` payload carries the on-air packet as hex in `payload["raw"]`.
3. `LetsMeshNormalizer._normalize_letsmesh_event` (`letsmesh_normalizer.py:23`)
decodes the packet once via `LetsMeshPacketDecoder.decode_payload`
(`letsmesh_decoder.py:219`) and classifies it into an event type: message,
advertisement, trace, contact, telemetry, path/status, or the catch-all
`letsmesh_packet`.
4. The normalized event is dispatched to a handler (`subscriber.py:213`).
Structured handlers (`handlers/__init__.py:26-31`) extract specific columns
into their own tables and **drop** the raw hex. Only unmatched/informational
events fall through to `handle_event_log`, which stores the full payload JSON
(raw hex included) in `events_log`.
### Problems
- **Lossy capture**: The most interesting packets (adverts, channel messages,
traces, telemetry) are parsed and their raw form is not retained. The raw
record only survives for the packets that *don't* map to a structured handler.
- **No raw browse/search**: Operators debugging the mesh (decode failures,
unexpected packet types, path/SNR anomalies) have no way to view or filter the
actual wire packets.
- **No per-observer record**: Structured tables dedupe by event hash across
observers; there is no one-row-per-reception log of what each observer heard.
### Why Now
The decoder already runs on every packet and already attempts decryption with the
DB-backed channel key store (`letsmesh_decoder.py:240`). The channel-visibility
machinery (`channel_visibility.py`) and API-key auth (`auth.py`) already exist and
are used by the messages feed. Capturing the raw packet is therefore a cheap
addition that reuses the existing decode, key store, and visibility model — no new
crypto and no second decode.
## Goals
- Introduce a `RawPacket` SQLAlchemy model capturing every `packets`-feed packet,
one row per observer reception (no dedup).
- Persist the raw hex, classification, decoded summary, channel index, source
identity, and link metadata (`packet_hash`) at ingest, reusing the decode the
normalizer already performs.
- Add a separate `RAW_PACKET_RETENTION_DAYS` knob (defaulting to the global
retention default) and wire `raw_packets` into the existing cleanup scheduler.
- Provide a `/packets` API with filtering, sorting, and pagination consistent
with `/advertisements` and `/messages`.
- Apply channel-visibility rules to channel-message packets: packets on channels
above the user's role are returned as **metadata-only with the payload
redacted** (not hidden); `raw_hex` is returned in full for any non-redacted
packet.
- Provide a SPA **Packets** page behind a feature flag that is **off by default**.
## Non-Goals
- Capturing the `status` / `internal` feeds (they carry station housekeeping, not
on-air packets — see Requirements). These already land in `events_log`.
- Deduplicating raw packets across observers (explicitly one row per reception).
- Changing the `meshcoredecoder` library or any decryption logic.
- Decrypting direct/contact (node-to-node ECDH) messages — the hub holds no node
private keys; those remain ciphertext with no decoded text.
- Encrypting `raw_hex` at rest, or per-user (non-role) access control.
- Backfilling raw packets for traffic received before this feature ships.
## Requirements
### Functional Requirements
- **FR-1**: A `RawPacket` database model with fields:
- `id` (UUID primary key)
- `observer_node_id` (FK `nodes.id`, `ondelete=SET NULL`, nullable, indexed) —
the receiving interface, resolved from the MQTT topic public key
- `packet_hash` (String(32), nullable, indexed) — LetsMesh `hash`; links rows to
structured-table records and groups multi-observer receptions
- `raw_hex` (Text, nullable) — the on-air bytes from `payload["raw"]`
- `packet_type` (Integer, nullable, indexed) — wire packet type
- `payload_type` (Integer, nullable) — decoder payload type
- `event_type` (String(50), nullable, indexed) — how the collector classified
the packet (`advertisement`, `channel_msg_recv`, `contact_msg_recv`,
`trace_data`, `telemetry_response`, `letsmesh_packet`, ...)
- `channel_idx` (Integer, nullable, indexed) — `int(channelHash, 16)` for
channel-message packets; drives visibility filtering. `NULL` for non-channel
packets.
- `source_pubkey_prefix` (String(12), nullable, indexed) — sender prefix derived
from the decoder `sourceHash` / `senderPublicKey`, for efficient
"packets from this node" filtering
- `route_type` (String(20), nullable)
- `path_len` (Integer, nullable)
- `snr` (Float, nullable)
- `decoded` (JSON, nullable) — decoder summary, so the detail view needs no
re-decode
- `received_at` (DateTime(tz), default `utc_now`, indexed)
- `created_at`, `updated_at` (timestamps via `TimestampMixin`)
- **FR-2**: When capture is enabled (FR-13), for **every** message on the
`packets` feed the collector writes exactly one `RawPacket` row, regardless of
how the packet is subsequently classified or whether a structured handler also
persists it. Capture happens before/independent of structured dispatch so the
table is complete. When capture is disabled, no `RawPacket` rows are written and
the structured handlers continue unchanged.
- **FR-3**: The `status` and `internal` feeds are **not** captured as raw packets
(they carry no `raw` hex). No change to their existing `letsmesh_status` /
`letsmesh_internal` handling.
- **FR-4**: No deduplication. If N observers report the same packet, N rows are
stored (grouped by `packet_hash`).
- **FR-5**: `RawPacket` rows are subject to data-retention cleanup using a
dedicated `RAW_PACKET_RETENTION_DAYS` setting, defaulting to the existing global
`DATA_RETENTION_DAYS` value. Cleanup runs in the existing scheduler.
- **FR-6**: `GET /packets` lists raw packets with the following filters, all
optional and combinable:
- `search` (string) — `ilike` across `packet_hash`, `raw_hex`, and observer
name/public key
- `event_type` (string, comma-separated allowed) — classification filter
- `packet_type` (int, comma-separated allowed) — wire type filter
- `channel_idx` (int) — single channel filter
- `route_type` (string, comma-separated; `all`/`none`/`""` disables) — same
semantics as `/advertisements`
- `public_key` / `pubkey_prefix` (string) — filter by `source_pubkey_prefix`
- `observed_by` (list[str]) — filter by receiver node public keys
- `decryptable` (bool) — only packets whose `decoded` contains decrypted text
(vs ciphertext-only)
- `min_snr` / `max_snr` (float) — SNR range
- `min_path_len` / `max_path_len` (int) — hop-count range
- `redacted` (bool) — include only / exclude redacted (metadata-only) rows
- `since` / `until` (datetime) — `received_at` window
- `sort` / `order``sort` in `VALID_PACKET_SORT_COLUMNS`, `order` in
`asc`/`desc`
- `limit` (1100, default 50) / `offset` (>= 0)
- **FR-7**: `GET /packets/{id}` returns a single raw packet with its decoded
summary, subject to the same visibility/redaction rules.
- **FR-8**: **Channel-visibility redaction**. Using the user's resolved role
(`channel_visibility.resolve_user_role` + `get_visible_channel_indices`):
- Non-channel packets (`channel_idx IS NULL`): returned in full.
- Channel packets whose `channel_idx` is visible to the role: returned in full,
including `raw_hex` and `decoded`.
- Channel packets whose `channel_idx` is **above** the role's level: returned
with `redacted=true` and `raw_hex`, `decoded`, and any decoded text nulled
out. Non-sensitive metadata is retained: `packet_hash`, `packet_type`,
`event_type`, `channel_idx`, `route_type`, `path_len`, `snr`,
`observer`, `received_at`.
- The built-in Public channel (idx 17) is always visible.
- **FR-9**: `raw_hex` is **always included** in the response for any non-redacted
packet the user is permitted to see (no extra role gate beyond visibility).
- **FR-10**: A SPA **Packets** page at `/packets`, controlled by a
`FEATURE_PACKETS` flag that defaults to **off**. The page provides:
- text search box (`search`)
- event-type chips / dropdown (`event_type`)
- channel dropdown (`channel_idx`) showing only channels the role can see
- observer multi-select (`observed_by`)
- route-type toggle group (`route_type`)
- time-range picker (`since` / `until`)
- a collapsible "advanced" panel for `packet_type`, SNR range, path-len range,
`decryptable`, `redacted`
- **responsive layout matching the other list pages**: a zebra `<table>` on
desktop (`hidden lg:block`) with `sortableTableHeader` columns, and a stacked
**card** list on mobile (`lg:hidden`) with a `mobileSortSelect`
- pagination
- a per-row lock badge when `redacted=true`
- **FR-11**: A node-detail "packets from this node" view can reuse `/packets`
with `pubkey_prefix` (out of scope to build the node-detail UI here, but the
filter and index must support it).
- **FR-12**: Navigation entry points for the Packets page, all gated on
`features.packets` (so they stay hidden while the flag is off by default),
inserted **immediately after Messages** on every surface (i.e. `... → Messages →
Packets → Map → Members ...`), consistently across all three:
- **Homepage hero nav card** in `home.js` (`renderNavCard`) with a
`--color-packets` accent
- **Sidebar / mobile menu**: the dynamic nav in `app.js` and the static nav in
`spa.html`
- a matching page title in `updatePageTitle`
- **FR-13**: A **collector-side capture flag** `RAW_PACKET_CAPTURE_ENABLED`
(default `false`) controls whether the collector writes `RawPacket` rows. It is
independent of the web `FEATURE_PACKETS` flag (the collector and web are separate
processes with separate settings), but Compose links them by default so a single
source var drives both:
`RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS:-false}`. When disabled, the
collector performs no raw-packet inserts (eliminating the write-amplification
cost) while normalization/structured handling is unaffected. Capture has no
backfill, so the web page only shows packets captured while the flag was on.
Retention cleanup of `raw_packets` runs regardless of this flag, so turning
capture off lets existing rows drain via `RAW_PACKET_RETENTION_DAYS`.
### Technical Requirements
- **TR-1**: New model `RawPacket` in
`src/meshcore_hub/common/models/raw_packet.py`, exported from
`models/__init__.py`. Follows the `Advertisement` pattern (`UUIDMixin`,
`TimestampMixin`). Single-column indexes: `received_at`, `event_type`,
`packet_hash`, `channel_idx`, `source_pubkey_prefix`, `observer_node_id`.
Composite indexes to serve the common "filter then sort by newest" pattern
without a full scan + filesort: `(event_type, received_at)`,
`(channel_idx, received_at)`, `(source_pubkey_prefix, received_at)`. See TR-17
for the write-cost trade-off.
- **TR-2**: Alembic migration to create the `raw_packets` table, datestamped per
convention (`alembic/versions/2026XXXX_XXXX_add_raw_packets.py`).
- **TR-3**: New handler `src/meshcore_hub/collector/handlers/raw_packet.py`
exposing `store_raw_packet(...)`. It performs the find-or-create observer node
pattern used by `handle_event_log` (`handlers/event_log.py:34-50`) and inserts a
`RawPacket` row. It does not dedup.
- **TR-4**: Capture hook in `Subscriber._handle_mqtt_message`
(`subscriber.py:189`), guarded by `self._raw_packet_capture_enabled` (a cheap
boolean short-circuit when disabled — note it saves the insert, not the decode,
which normalization performs regardless). When capture is enabled and the parsed
feed is `packets`, call
`store_raw_packet` with the raw payload, the decoder output already produced by
the normalizer, the resolved `event_type`, and the observer public key. To avoid
decoding twice, `_normalize_letsmesh_event` is refactored to also return (or the
subscriber to reuse) the `decoded_packet` and `event_type`; capture uses that
single decode. Capture failures are logged and never block event dispatch.
- **TR-5**: Channel index and source identity are derived at capture time:
`channel_idx = int(channelHash, 16)` from
`_extract_letsmesh_decoder_channel_hash`; `source_pubkey_prefix` from the
decoder `sourceHash` / `senderPublicKey` via the existing
`_normalize_pubkey_prefix` helper.
- **TR-6**: `RAW_PACKET_RETENTION_DAYS` config field added to `CollectorSettings`
in `config.py` (alongside `data_retention_*` at `config.py:118`), defaulting to
the value of `data_retention_days`. Threaded through `create_subscriber` /
`run_collector` like the other cleanup params.
- **TR-6a**: `raw_packet_capture_enabled` (`RAW_PACKET_CAPTURE_ENABLED`) bool field
added to `CollectorSettings`, default `false`, following the
`data_retention_enabled` / `node_cleanup_enabled` convention. Threaded through
`create_subscriber` / `run_collector` to `Subscriber.__init__` and stored as
`self._raw_packet_capture_enabled` for the TR-4 guard. The collector logs the
capture state at startup. Compose wires it on the **collector** service as
`RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS:-false}` (in `docker-compose.yml`
and any prod/traefik overrides); `FEATURE_PACKETS=${FEATURE_PACKETS:-false}`
stays on the web service. Both env vars added to `.env.example`.
- **TR-7**: `collector/cleanup.py` gains a `raw_packets_deleted` stat and a cleanup
step for `RawPacket`. Because `_cleanup_table` (`cleanup.py:185`) deletes by
`created_at < cutoff` with a single retention value, add support for a per-table
retention override (or a dedicated call) so `raw_packets` can use
`RAW_PACKET_RETENTION_DAYS` independently of `DATA_RETENTION_DAYS`. The override
falls back to `DATA_RETENTION_DAYS` when `RAW_PACKET_RETENTION_DAYS` is unset.
Add `RawPacket.observer_node_id` to the observer union in
`_observer_node_id_union()` (`cleanup.py:124`) so a node that appears only as a
raw-packet observer still counts as an observer, and `recompute_observer_flags`
does not clear its `is_observer` flag while it has live raw packets.
- **TR-8**: Pydantic schemas `RawPacketRead` / `RawPacketList` in
`src/meshcore_hub/common/schemas/` (mirroring `messages.py`
`AdvertisementRead`/`AdvertisementList`). `RawPacketRead` includes a
`redacted: bool` field.
- **TR-9**: New route module `src/meshcore_hub/api/routes/raw_packets.py`,
registered in `api/routes/__init__.py` with `prefix="/packets"`, `tags=["Packets"]`.
Uses `RequireRead`, `DbSession`, and `@cached("packets",
key_builder=_packets_key_builder)`. `_packets_key_builder` includes the resolved
role (`resolve_user_role(request) or "anonymous"`) in the cache key — exactly
like `messages.py:30` — so role-redacted responses are **never** served across
roles. TTL uses the default `redis_cache_ttl` setting (`REDIS_CACHE_TTL`, default
30s); no dedicated TTL var. `VALID_PACKET_SORT_COLUMNS =
{"time", "event_type", "packet_type", "snr", "path_len"}`, default `time`/`desc`.
Observer hydration follows the `aliased(Node)` + `selectinload(Node.tags)` pattern
in `advertisements.py`.
- **TR-10**: The route applies redaction (FR-8) after fetching rows: compute the
visible channel-index set once via `get_visible_channel_indices`, then for each
channel-message row above the level, null the sensitive fields and set
`redacted=true`. The SQL query is **not** pre-filtered by visibility (rows are
returned but redacted), so pagination counts are stable across roles.
- **TR-11**: Web proxy access in `web/app.py` `_build_endpoint_access()`:
`"v1/packets": { "GET": _OPEN }` (read gated only by `RequireRead` /
channel-visibility redaction, consistent with `v1/messages` and
`v1/advertisements`).
- **TR-12**: `FEATURE_PACKETS` feature flag in `WebSettings` (`config.py:397`),
**default `False`**, registered in the `features` property (`config.py:428`) as
`"packets": self.feature_packets` (no OIDC gate).
- **TR-13**: New SPA page module
`src/meshcore_hub/web/static/js/spa/pages/packets.js` (lit-html), modelled on
`pages/advertisements.js`, including its **responsive layout**: a `hidden
lg:block` zebra `<table>` with `sortableTableHeader` on desktop and a
`lg:hidden` stacked-card list with `mobileSortSelect` on mobile. Wire into
`app.js`: lazy import entry, route
`router.addRoute('/packets', pageHandler(pages.packets))` guarded by
`features.packets`, dynamic nav item (`items.push`), and page title
(`updatePageTitle`). Renders the `redacted` lock badge.
- **TR-13a**: Add the Packets entry to **all three nav surfaces**, each guarded by
`features.packets` and inserted immediately **after the Messages entry** (before
Map): the static `spa.html` sidebar/mobile `<li>` (`{% if features.packets %}`),
the dynamic `app.js` nav (`items.push`), and the `home.js` hero card grid
(`renderNavCard`).
- **TR-13b**: Correct the `home.js` hero-card order to match the sidebar/dynamic
nav: it currently renders **Members before Map**; reorder to **Map before
Members** so every surface follows the canonical order `Dashboard → Nodes →
Advertisements → Channels → Messages → Packets → Map → Members`. Add a packets
icon function to `icons.js` (imported by `app.js`, `home.js`, and `packets.js`).
Add `--color-packets` (light + dark blocks), a `.nav-icon-packets` rule, and the
`.navbar .menu li:has(.nav-icon-packets)` accent in `app.css`.
- **TR-14**: i18n keys for packets UI strings (including `entities.packets`,
`entities.packet`, filter labels, and the redacted badge) added to **both**
locale files — `en.json` and `nl.json` — and documented per the existing i18n
docs.
- **TR-15**: `node build.js` rebuild of the SPA bundle after frontend changes.
- **TR-16**: Documentation updates:
- `AGENTS.md` — new model, API route, `FEATURE_PACKETS`,
`RAW_PACKET_RETENTION_DAYS`.
- `SCHEMAS.md` — the `raw_packets` table and the `RawPacketRead` shape.
- `.env.example` — add `RAW_PACKET_RETENTION_DAYS` and
`RAW_PACKET_CAPTURE_ENABLED` near `DATA_RETENTION_DAYS` (commented, noting the
retention default = `DATA_RETENTION_DAYS` and capture default = `false`), and
`# FEATURE_PACKETS=false` in the feature-flags block, with a comment that
Compose derives `RAW_PACKET_CAPTURE_ENABLED` from `FEATURE_PACKETS`.
- `docker-compose.yml` (+ prod/traefik overrides) — `RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS:-false}`
on the collector service; `FEATURE_PACKETS=${FEATURE_PACKETS:-false}` on the
web service.
- `docs/upgrading.md` — add a new **`## v0.13.0`** section at the top (above
`v0.12.0`) with the upgrade steps (see TR-18).
- `docs/letsmesh.md` — already covered in Phase 6.
- **TR-17**: **Performance / indexing.** The `raw_packets` table is the
highest-volume table in the system (one row per packet per observer), so:
- The composite indexes in TR-1 cover the dominant query shapes (filter by
`event_type` / `channel_idx` / `source_pubkey_prefix`, sorted by
`received_at DESC`). Do **not** add further composite indexes speculatively —
each one is paid on every insert in the hot ingest path.
- `search` is an unanchored `ilike '%term%'` over `raw_hex` / `packet_hash` and
**cannot use an index** (full scan). The route should require it to combine
with a narrowing filter (a `since` default window) so substring search runs
against a bounded row set; document this limitation. Hash lookups should use a
prefix (`ilike 'abc%'`) where possible so the `packet_hash` index applies.
- Pagination uses an exact `COUNT(*)` over the filtered subquery (matching the
other list routes); on a large table this is the most expensive part of the
request. Accept it for parity now; note approximate/capped counts as a possible
follow-up if it becomes a hotspot.
- **Write amplification:** capture adds one `INSERT` per packet to the ingest
path. SQLite is single-writer, so this competes with the structured handlers'
writes already happening per packet. The capture handler must keep its
transaction minimal (single insert + observer upsert, no extra reads) and must
not hold the session longer than needed. The separate (and typically shorter)
`RAW_PACKET_RETENTION_DAYS` is the primary mechanism keeping the table — and
its index size — bounded.
- **TR-18**: `docs/upgrading.md` **v0.13.0** section content:
- Run `meshcore-hub db upgrade` to create the `raw_packets` table.
- New optional env vars, all safe to omit: `FEATURE_PACKETS` (defaults to
`false` — Packets page and nav hidden until enabled), `RAW_PACKET_CAPTURE_ENABLED`
(collector capture; Compose derives it from `FEATURE_PACKETS`, default `false`),
and `RAW_PACKET_RETENTION_DAYS` (defaults to `DATA_RETENTION_DAYS`).
- Explain the capture↔page split: setting `FEATURE_PACKETS=true` enables both
capture and the page via the Compose wiring; advanced operators can set the two
independently. No backfill — only packets captured after enabling appear.
- Note the `raw_packets` table grows fastest of all; recommend tuning
`RAW_PACKET_RETENTION_DAYS` for busy meshes / constrained storage, and that
disabling capture lets it drain via retention.
- Note Redis caching of `/packets` is role-aware and honours the existing
`REDIS_CACHE_TTL`.
## Implementation Plan
### Phase 1: Model & Migration
- Create `src/meshcore_hub/common/models/raw_packet.py` with the fields in FR-1 /
TR-1, following the `Advertisement` model.
- Export `RawPacket` from `models/__init__.py`.
- Generate the Alembic migration for the `raw_packets` table with all indexes.
- Unit tests for the model (column presence, defaults, indexes).
### Phase 2: Capture at Ingest
- Add `store_raw_packet(...)` in
`src/meshcore_hub/collector/handlers/raw_packet.py` (find-or-create observer
node, derive `channel_idx` and `source_pubkey_prefix`, insert one row, no dedup).
- Refactor `_normalize_letsmesh_event` / `_handle_mqtt_message` so the single
decode performed during normalization is reused for capture (return or expose
`decoded_packet` + resolved `event_type`).
- Add `raw_packet_capture_enabled` to `CollectorSettings`, thread it to
`Subscriber` as `self._raw_packet_capture_enabled`, and guard the capture hook
on it (log the state at startup).
- Call `store_raw_packet` for the `packets` feed only (when capture is enabled),
before/independent of structured dispatch. Wrap in try/except so capture never
blocks event handling.
- Tests: capture writes one row per packet; one row per observer for the same
hash; `channel_idx` / `source_pubkey_prefix` derivation; status/internal feeds
produce no raw rows; **capture disabled writes no rows while structured handlers
still run**.
### Phase 3: Retention & Config
- Add `RAW_PACKET_RETENTION_DAYS` to `CollectorSettings`, defaulting to
`data_retention_days`.
- Thread the value through `create_subscriber` / `run_collector` and the cleanup
scheduler.
- Extend `cleanup.py` with a `raw_packets_deleted` stat and a `RawPacket` cleanup
step using the dedicated retention value (per-table retention override on
`_cleanup_table`, or a dedicated call).
- Add `RawPacket.observer_node_id` to `_observer_node_id_union()` so raw-packet-only
observers retain their `is_observer` flag in `recompute_observer_flags`.
- Tests: raw packets older than `RAW_PACKET_RETENTION_DAYS` are deleted; default
falls back to the global retention value; stat is reported; a node observing
only raw packets keeps `is_observer=true` until those packets are pruned.
### Phase 4: Schemas & API
- Add `RawPacketRead` / `RawPacketList` schemas (with `redacted: bool`).
- Create `api/routes/raw_packets.py`:
- `GET /packets` with all filters in FR-6, sort whitelist, pagination, observer
hydration, and `@cached("packets", key_builder=_packets_key_builder)` — the
key builder folds the resolved role into the cache key (TR-9) so redacted
responses don't leak across roles; default TTL.
- `GET /packets/{id}`.
- Redaction pass (FR-8) using `get_visible_channel_indices`; `raw_hex` always
present for non-redacted rows (FR-9).
- Register the router in `api/routes/__init__.py` (`prefix="/packets"`).
- Add `"v1/packets": { "GET": _OPEN }` to `_build_endpoint_access()`.
- Tests: each filter; sort/pagination; redaction for a low role viewing a
member-channel packet (`redacted=true`, null `raw_hex`/`decoded`); non-channel
packets never redacted; Public (17) always visible.
### Phase 5: Web Packets Page
- Add `FEATURE_PACKETS` to `WebSettings` (default `False`) and the `features` map.
- Create `pages/packets.js` (lit-html) modelled on `advertisements.js`, with the
**desktop zebra table (`hidden lg:block`) + mobile stacked cards (`lg:hidden`)**
responsive split using `sortableTableHeader` / `mobileSortSelect`: search box,
event-type and channel filters, observer multi-select, route-type toggle,
time-range picker, collapsible advanced filters (`packet_type`, SNR range,
path-len range, `decryptable`, `redacted`), pagination, and a `redacted` lock
badge.
- Wire `app.js`: lazy import, route guarded by `features.packets`, page title,
i18n keys; add a packets icon function in `icons.js`.
- Add the Packets nav entry after Messages on **all three surfaces** (guarded by
`features.packets`): `spa.html` sidebar/mobile `<li>`, `app.js` dynamic nav, and
the `home.js` hero card grid (`renderNavCard`).
- Reorder the `home.js` hero cards so **Map precedes Members**, matching the
sidebar/dynamic nav (TR-13b).
- Add `--color-packets` (light + dark), `.nav-icon-packets`, and the navbar
`:has(.nav-icon-packets)` accent in `app.css`.
- Add i18n strings to `en.json` (incl. `entities.packets`, `entities.packet`) and
locale stubs.
- `node build.js` to rebuild the bundle.
- Tests in the web test suite (nav/route hidden when the flag is off, list render,
redacted badge).
### Phase 6: Docs
- Update `AGENTS.md` (model, route, `FEATURE_PACKETS`, `RAW_PACKET_RETENTION_DAYS`).
- Update `SCHEMAS.md` with the `raw_packets` table and `RawPacketRead`.
- Update `.env.example`: add `RAW_PACKET_RETENTION_DAYS` (commented, near
`DATA_RETENTION_DAYS`, noting the `DATA_RETENTION_DAYS` default) and
`# FEATURE_PACKETS=false` in the feature-flags block.
- Add a new `## v0.13.0` section to `docs/upgrading.md` (above `v0.12.0`) with the
TR-18 content: `db upgrade`, the two new optional env vars and their defaults,
the no-backfill / table-growth note, and the role-aware cache note.
- Update `docs/letsmesh.md` to note raw-packet capture is `packets`-feed only and
that channel-restricted packets are returned metadata-only/redacted.
- Confirm `entities.packets` / `entities.packet` and filter labels exist in both
`en.json` and `nl.json`.
## Open Decisions (Resolved)
- **Retention**: separate `RAW_PACKET_RETENTION_DAYS`, default = global
`DATA_RETENTION_DAYS`. (Resolved.)
- **Feeds**: `packets` only; `status`/`internal` excluded (no `raw` hex).
(Resolved.)
- **Dedup**: none — one row per observer reception. (Resolved.)
- **Feature flag**: `FEATURE_PACKETS` defaults to **off**. (Resolved.)
- **Restricted channel packets**: shown as metadata-only with payload redacted
(`redacted=true`, null `raw_hex`/`decoded`), not hidden. (Resolved.)
- **`raw_hex` exposure**: always returned for non-redacted, visible packets.
(Resolved.)
## Review
**Status**: Draft — pending review.
## References
- `src/meshcore_hub/collector/subscriber.py:189``_handle_mqtt_message` (capture
hook site); `:512` — feed subscriptions
- `src/meshcore_hub/collector/letsmesh_normalizer.py:23`
`_normalize_letsmesh_event` (single-decode + classification to reuse)
- `src/meshcore_hub/collector/letsmesh_decoder.py:219``decode_payload`;
`:174` — channel hash computation
- `src/meshcore_hub/collector/handlers/event_log.py:34-50` — find-or-create
observer node pattern to follow in the raw-packet handler
- `src/meshcore_hub/collector/handlers/__init__.py:26-37` — handler registration /
structured vs informational split
- `src/meshcore_hub/common/models/advertisement.py` — model pattern (UUIDMixin,
TimestampMixin, indexes)
- `src/meshcore_hub/collector/cleanup.py:54,185``cleanup_old_data` /
`_cleanup_table` (retention to extend for per-table override)
- `src/meshcore_hub/common/config.py:118-135` — data-retention settings; `:397-448`
— feature flags and the `features` map
- `src/meshcore_hub/api/routes/advertisements.py` — list/detail route pattern
(aliased node joins, observer hydration, sort whitelist, pagination, `@cached`)
- `src/meshcore_hub/api/routes/messages.py:53,83-106``channel_idx` filter and
channel-visibility filtering to mirror
- `src/meshcore_hub/api/channel_visibility.py``resolve_user_role`,
`get_visible_channel_indices`, `get_all_known_channel_indices`
- `src/meshcore_hub/api/auth.py:52,145``require_read` / `RequireRead`
- `src/meshcore_hub/web/app.py:68-129``_build_endpoint_access()` proxy access map
- `src/meshcore_hub/web/static/js/spa/pages/advertisements.js` — SPA list page to
model `packets.js` on
- `src/meshcore_hub/web/static/js/spa/app.js:21,64-99,161` — lazy import map, route
registration, nav, page titles
@@ -0,0 +1,192 @@
# Tasks: Raw Packets — Capture, Store, and Browse Decoded Wire Packets
> Generated from `plan.md` on 2026-06-12
## 1. Database Model & Migration
- [x] 1.1 Create `RawPacket` SQLAlchemy model
- [x] 1.1.1 Create `RawPacket` class in `src/meshcore_hub/common/models/raw_packet.py` following the `Advertisement` pattern (`Base`, `UUIDMixin`, `TimestampMixin`)
- [x] 1.1.2 Fields: `observer_node_id` (FK `nodes.id`, `ondelete="SET NULL"`, nullable), `packet_hash` (String(32), nullable), `raw_hex` (Text, nullable), `packet_type` (Integer, nullable), `payload_type` (Integer, nullable), `event_type` (String(50), nullable), `channel_idx` (Integer, nullable), `source_pubkey_prefix` (String(12), nullable), `route_type` (String(20), nullable), `path_len` (Integer, nullable), `snr` (Float, nullable), `decoded` (JSON, nullable), `received_at` (DateTime(tz), default `utc_now`)
- [x] 1.1.3 Single-column indexes on `received_at`, `event_type`, `packet_hash`, `channel_idx`, `source_pubkey_prefix`, `observer_node_id`
- [x] 1.1.4 Composite indexes `(event_type, received_at)`, `(channel_idx, received_at)`, `(source_pubkey_prefix, received_at)` for the "filter then sort by newest" pattern
- [x] 1.1.5 Add `__repr__`
- [x] 1.1.6 Export `RawPacket` from `models/__init__.py`
- [x] 1.2 Generate Alembic migration
- [x] 1.2.1 Run `meshcore-hub db revision --autogenerate -m "add raw_packets table"`
- [x] 1.2.2 Review generated migration: all columns, FK, and every index (single + composite) present
- [x] 1.2.3 Test migration: `meshcore-hub db upgrade` and verify table + indexes created
- [x] 1.3 Write unit tests for the model
- [x] 1.3.1 Test `RawPacket` instantiation and defaults (`received_at`, timestamps)
- [x] 1.3.2 Test nullable columns accept `None`
- [x] 1.3.3 Test index presence via table metadata
## 2. Collector Capture & Capture Flag
- [x] 2.1 Add `raw_packet_capture_enabled` setting
- [x] 2.1.1 Add `raw_packet_capture_enabled: bool` to `CollectorSettings` in `config.py` (env `RAW_PACKET_CAPTURE_ENABLED`, default `false`), following the `data_retention_enabled` convention
- [x] 2.1.2 Thread through `create_subscriber()` and `run_collector()` to `Subscriber.__init__`
- [x] 2.1.3 Store as `self._raw_packet_capture_enabled`; log capture state at startup
- [x] 2.2 Create the raw-packet handler
- [x] 2.2.1 Create `src/meshcore_hub/collector/handlers/raw_packet.py` with `store_raw_packet(...)`
- [x] 2.2.2 Find-or-create observer node from the topic public key (follow `handlers/event_log.py:34-50`), update `last_seen`
- [x] 2.2.3 Derive `channel_idx = int(channelHash, 16)` via `_extract_letsmesh_decoder_channel_hash`
- [x] 2.2.4 Derive `source_pubkey_prefix` from decoder `sourceHash` / `senderPublicKey` via `_normalize_pubkey_prefix`
- [x] 2.2.5 Insert exactly one `RawPacket` row (no dedup); keep the transaction minimal (single insert + observer upsert, no extra reads)
- [x] 2.3 Wire the capture hook into the subscriber
- [x] 2.3.1 Refactor `_normalize_letsmesh_event` / `_handle_mqtt_message` so the decode already performed during normalization is reused for capture (return/expose `decoded_packet` + resolved `event_type`)
- [x] 2.3.2 In `_handle_mqtt_message`, short-circuit on `self._raw_packet_capture_enabled` (saves the insert, not the decode)
- [x] 2.3.3 When enabled and feed is `packets`, call `store_raw_packet` before/independent of structured dispatch
- [x] 2.3.4 Wrap capture in try/except so it never blocks event dispatch
- [x] 2.3.5 Ensure `status` / `internal` feeds produce no raw rows
- [x] 2.4 Write collector tests
- [x] 2.4.1 Capture writes one row per packet (enabled)
- [x] 2.4.2 One row per observer for the same `packet_hash` (no dedup)
- [x] 2.4.3 `channel_idx` / `source_pubkey_prefix` derivation correct
- [x] 2.4.4 `status` / `internal` feeds write no raw rows
- [x] 2.4.5 Capture disabled writes no rows while structured handlers still run
- [x] 2.4.6 Capture failure is logged and does not interrupt event handling
## 3. Retention & Config
- [x] 3.1 Add `RAW_PACKET_RETENTION_DAYS` setting
- [x] 3.1.1 Add `raw_packet_retention_days: int` to `CollectorSettings` near `data_retention_*`, defaulting to `data_retention_days` when unset
- [x] 3.1.2 Thread through `create_subscriber()` / `run_collector()` and the cleanup scheduler
- [x] 3.2 Extend cleanup for raw packets
- [x] 3.2.1 Add a `raw_packets_deleted` counter to `CleanupStats` (`__init__` + `__repr__`)
- [x] 3.2.2 Add per-table retention override to `_cleanup_table` (or a dedicated call) so `raw_packets` uses `RAW_PACKET_RETENTION_DAYS` independently
- [x] 3.2.3 Add the `RawPacket` cleanup step to `cleanup_old_data`; include in `total_deleted`
- [x] 3.2.4 Ensure cleanup runs regardless of `raw_packet_capture_enabled` (so existing rows drain when capture is off)
- [x] 3.2.5 Add `RawPacket.observer_node_id` to `_observer_node_id_union()` so raw-packet-only observers keep `is_observer`
- [x] 3.3 Write retention tests
- [x] 3.3.1 Raw packets older than `RAW_PACKET_RETENTION_DAYS` are deleted; stat reported
- [x] 3.3.2 Default falls back to global `DATA_RETENTION_DAYS` when unset
- [x] 3.3.3 A node observing only raw packets keeps `is_observer=true` until those packets are pruned
## 4. Schemas & API
- [x] 4.1 Create Pydantic schemas
- [x] 4.1.1 Add `RawPacketRead` / `RawPacketList` (mirror `AdvertisementRead` / `AdvertisementList`)
- [x] 4.1.2 `RawPacketRead` includes a `redacted: bool` field
- [x] 4.2 Create the packets route module
- [x] 4.2.1 Create `src/meshcore_hub/api/routes/raw_packets.py` with `RequireRead`, `DbSession`
- [x] 4.2.2 Add `_packets_key_builder(request)` folding `resolve_user_role(request) or "anonymous"` into the key (per `messages.py:30`)
- [x] 4.2.3 Decorate list with `@cached("packets", key_builder=_packets_key_builder)` using the default `redis_cache_ttl`
- [x] 4.2.4 Define `VALID_PACKET_SORT_COLUMNS = {"time", "event_type", "packet_type", "snr", "path_len"}`, default `time`/`desc`
- [x] 4.2.5 Register router in `api/routes/__init__.py` with `prefix="/packets"`, `tags=["Packets"]`
- [x] 4.3 Implement `GET /packets` filtering
- [x] 4.3.1 `search``ilike` over `packet_hash`, `raw_hex`, observer name/public key
- [x] 4.3.2 `event_type` (comma-separated), `packet_type` (comma-separated)
- [x] 4.3.3 `channel_idx` (single)
- [x] 4.3.4 `route_type` (comma-separated; `all`/`none`/`""` disables — match `/advertisements`)
- [x] 4.3.5 `public_key` / `pubkey_prefix``source_pubkey_prefix`
- [x] 4.3.6 `observed_by` (list) → observer public keys
- [x] 4.3.7 `decryptable` (bool) — only packets whose `decoded` has decrypted text
- [x] 4.3.8 `min_snr` / `max_snr`, `min_path_len` / `max_path_len`
- [x] 4.3.9 `redacted` (bool) — include only / exclude redacted rows
- [x] 4.3.10 `since` / `until` on `received_at`; `sort` / `order`; `limit` (1100, default 50) / `offset`
- [x] 4.3.11 Observer hydration via `aliased(Node)` + `selectinload(Node.tags)` (per `advertisements.py`)
- [x] 4.3.12 Require `search` to combine with a narrowing window (default `since`) so the unindexed substring scan runs against a bounded set
- [x] 4.4 Implement channel-visibility redaction
- [x] 4.4.1 Compute the visible channel-index set once via `get_visible_channel_indices` (Public idx 17 always visible)
- [x] 4.4.2 Non-channel packets (`channel_idx IS NULL`) returned in full
- [x] 4.4.3 Visible channel packets returned in full incl. `raw_hex` and `decoded`
- [x] 4.4.4 Channel packets above the role: set `redacted=true`, null `raw_hex` / `decoded` / decoded text; keep hash/type/channel/path/snr/observer/timing
- [x] 4.4.5 Do not pre-filter the SQL by visibility (rows returned but redacted) so pagination counts stay stable across roles
- [x] 4.5 Implement `GET /packets/{id}`
- [x] 4.5.1 Single-row fetch with observer hydration
- [x] 4.5.2 Apply the same redaction rules; 404 when not found
- [x] 4.6 Web proxy access
- [x] 4.6.1 Add `"v1/packets": { "GET": _OPEN }` to `_build_endpoint_access()` in `web/app.py`
- [x] 4.7 Write API tests
- [x] 4.7.1 Each filter narrows results correctly
- [x] 4.7.2 Sort whitelist + pagination (count stable across roles)
- [x] 4.7.3 Low role viewing a member-channel packet → `redacted=true`, null `raw_hex`/`decoded`
- [x] 4.7.4 Non-channel packets never redacted; Public (17) always visible
- [x] 4.7.5 Cache key differs by role (no cross-role leakage of redacted content)
## 5. Web Packets Page
- [x] 5.1 Add `FEATURE_PACKETS` feature flag
- [x] 5.1.1 Add `feature_packets: bool` to `WebSettings` (default `False`, env `FEATURE_PACKETS`)
- [x] 5.1.2 Add `"packets": self.feature_packets` to the `features` property (no OIDC gate)
- [x] 5.2 Create the Packets page module
- [x] 5.2.1 Create `src/meshcore_hub/web/static/js/spa/pages/packets.js` (lit-html), modelled on `advertisements.js`
- [x] 5.2.2 Desktop layout: `hidden lg:block` zebra `<table>` with `sortableTableHeader` columns
- [x] 5.2.3 Mobile layout: `lg:hidden` stacked-card list with `mobileSortSelect`
- [x] 5.2.4 Filter controls: search box, event-type chips/dropdown, channel dropdown (visible channels only), observer multi-select, route-type toggle, time-range picker
- [x] 5.2.5 Collapsible "advanced" panel: `packet_type`, SNR range, path-len range, `decryptable`, `redacted`
- [x] 5.2.6 Pagination + per-row lock badge when `redacted=true`
- [x] 5.3 Wire routing, icon, and nav
- [x] 5.3.1 Add lazy import entry and `router.addRoute('/packets', pageHandler(pages.packets))` guarded by `features.packets` in `app.js`
- [x] 5.3.2 Add a packets `updatePageTitle` case
- [x] 5.3.3 Add a packets icon function to `icons.js` (imported by `app.js`, `home.js`, `packets.js`)
- [x] 5.3.4 Add nav entry **immediately after Messages** on all three surfaces (guarded by `features.packets`): `spa.html` sidebar/mobile `<li>`, `app.js` dynamic nav, `home.js` hero card grid
- [x] 5.3.5 Reorder `home.js` hero cards so **Map precedes Members** (canonical order `Dashboard → Nodes → Advertisements → Channels → Messages → Packets → Map → Members`)
- [x] 5.3.6 Add `--color-packets` (light + dark), `.nav-icon-packets`, and the `.navbar .menu li:has(.nav-icon-packets)` accent in `app.css`
- [x] 5.4 i18n
- [x] 5.4.1 Add keys to `en.json`: `entities.packets`, `entities.packet`, filter labels, redacted-badge label
- [x] 5.4.2 Add the same keys to `nl.json`
- [x] 5.4.3 Add i18n key tests if the suite covers them
- [x] 5.5 Build & web tests
- [x] 5.5.1 Run `node build.js` to rebuild the SPA bundle
- [x] 5.5.2 Test nav/route hidden when `FEATURE_PACKETS` is off
- [x] 5.5.3 Test list render and redacted lock badge
## 6. Docs, Env & Compose
- [x] 6.1 Config & compose wiring
- [x] 6.1.1 `.env.example`: add `RAW_PACKET_RETENTION_DAYS` and `RAW_PACKET_CAPTURE_ENABLED` near `DATA_RETENTION_DAYS` (commented; note retention default = `DATA_RETENTION_DAYS`, capture default = `false`)
- [x] 6.1.2 `.env.example`: add `# FEATURE_PACKETS=false` to the feature-flags block with a note that Compose derives capture from it
- [x] 6.1.3 `docker-compose.yml`: add `RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS:-false}` to the collector service env
- [x] 6.1.4 `docker-compose.yml`: add `FEATURE_PACKETS=${FEATURE_PACKETS:-false}` to the web service env
- [x] 6.1.5 Apply the same env additions to prod/traefik compose overrides
- [x] 6.2 Reference docs
- [x] 6.2.1 `AGENTS.md`: add `RawPacket` model, `/packets` route, `FEATURE_PACKETS`, `RAW_PACKET_CAPTURE_ENABLED`, `RAW_PACKET_RETENTION_DAYS`
- [x] 6.2.2 `SCHEMAS.md`: add the `raw_packets` table and the `RawPacketRead` shape
- [x] 6.2.3 `docs/letsmesh.md`: note raw capture is `packets`-feed only and channel-restricted packets are returned metadata-only/redacted
- [x] 6.3 `docs/upgrading.md` v0.13.0 section
- [x] 6.3.1 Add a new `## v0.13.0` section at the top (above `v0.12.0`)
- [x] 6.3.2 Step: run `meshcore-hub db upgrade` to create `raw_packets`
- [x] 6.3.3 Document new optional env vars and defaults: `FEATURE_PACKETS` (false), `RAW_PACKET_CAPTURE_ENABLED` (Compose-derived from `FEATURE_PACKETS`, false), `RAW_PACKET_RETENTION_DAYS` (= `DATA_RETENTION_DAYS`)
- [x] 6.3.4 Explain the capture↔page split and no-backfill behaviour
- [x] 6.3.5 Note `raw_packets` grows fastest; recommend tuning `RAW_PACKET_RETENTION_DAYS`; disabling capture drains via retention
- [x] 6.3.6 Note `/packets` Redis caching is role-aware and honours `REDIS_CACHE_TTL`
## 7. Verification
- [x] 7.1 Code quality
- [x] 7.1.1 Run `pre-commit run --all-files` and fix all issues
- [x] 7.2 Component tests
- [x] 7.2.1 `pytest tests/test_common/` (model)
- [x] 7.2.2 `pytest tests/test_collector/` (capture, flag, retention)
- [x] 7.2.3 `pytest tests/test_api/` (filters, redaction, role-aware cache)
- [x] 7.2.4 `pytest tests/test_web/` (flag-gated nav/route, render)
- [x] 7.2.5 Full `pytest` to verify no regressions
- [x] 7.3 Manual verification
- [x] 7.3.1 With `RAW_PACKET_CAPTURE_ENABLED=false` (default), confirm no `raw_packets` rows are written and structured handlers still work
- [x] 7.3.2 Enable capture, confirm one row per packet per observer with correct `event_type` / `channel_idx` / `source_pubkey_prefix`
- [x] 7.3.3 With `FEATURE_PACKETS=true`, confirm the Packets page and nav appear after Messages on sidebar, mobile menu, and home page; Map precedes Members
- [x] 7.3.4 Verify desktop table / mobile card layouts and all filters
- [x] 7.3.5 Verify redaction: a low-privilege role sees restricted-channel packets as metadata-only with a lock badge; raw_hex absent
- [x] 7.3.6 Verify retention prunes old raw packets per `RAW_PACKET_RETENTION_DAYS` and the table drains after disabling capture
+40
View File
@@ -2,6 +2,46 @@
This guide covers upgrading from a previous MeshCore Hub release to the current version. Check the relevant version section below before upgrading.
## v0.13.0
### Raw Packets (capture, browse, and search wire packets)
A new **Raw Packets** feature captures every inbound MeshCore packet exactly as it arrives over the LetsMesh `packets` feed into a dedicated `raw_packets` table, independent of how the collector later classifies it. A new `/packets` API and a SPA **Packets** page (table on desktop, cards on mobile) let operators browse, filter, and search the raw traffic.
**Database migration required:**
```
meshcore-hub db upgrade
```
This creates the `raw_packets` table and its indexes. On Docker deployments the migration runs automatically on startup.
**New optional environment variables (all safe to omit):**
| Variable | Default | Description |
| ---------------------------- | ---------------------- | -------------------------------------------------------------------------------------------- |
| `FEATURE_PACKETS` | `false` | Show the Packets page and nav entry. Off by default. |
| `RAW_PACKET_CAPTURE_ENABLED` | `false` | Collector-side capture of raw packets. In Compose this is **derived from `FEATURE_PACKETS`**. |
| `RAW_PACKET_RETENTION_DAYS` | = `DATA_RETENTION_DAYS` | Days to retain raw packets, independent of the global retention window. |
**Capture ↔ page split:** capture runs in the collector while the page is served by the web app — two separate processes with separate settings. Docker Compose links them: setting `FEATURE_PACKETS=true` enables **both** capture (`RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS}` on the collector) and the page. Advanced operators running the processes directly can set the two flags independently.
**No backfill:** only packets captured *after* enabling appear — historical traffic is not reconstructed.
**Storage:** `raw_packets` grows fastest of all tables (one row per packet per observer). On busy meshes or constrained storage, lower `RAW_PACKET_RETENTION_DAYS`. Retention cleanup runs regardless of whether capture is currently enabled, so turning capture off lets existing rows drain. Restricted-channel packets are stored in full but returned **metadata-only (redacted)** to roles that cannot see the channel.
**Caching:** `/packets` responses are cached in Redis (when enabled) using a **role-aware** cache key and honour the existing `REDIS_CACHE_TTL`, so redacted responses are never served across roles.
The `advertisements` and `messages` tables gain a nullable `packet_hash` column (added by the same `db upgrade`) so each event can link to its captured raw packets. When `FEATURE_PACKETS` is on, the Adverts and Messages list pages show a packet icon linking to the raw packets for that transmission. Only events ingested while capture was enabled carry the hash (no backfill), so the link is hidden for older rows.
**No action required** to keep current behaviour — the feature is off by default.
### Finer-Grained Packet Classification
Packets the collector previously could not categorise were all emitted as a single `letsmesh_packet` event. They are now classified by their MeshCore payload type — `req`, `response`, `ack`, `encrypted_direct`, `encrypted_channel`, `grp_data`, `anon_req`, `multipart`, `control`, `raw_custom`, plus `advert`/`path`/`trace` for malformed variants. `letsmesh_packet` remains only as a safety net for packets whose payload type can't be resolved.
**Action only if you consume `event_type`:** any external webhook filter, saved query, or dashboard keyed on `letsmesh_packet` should be updated to the specific type(s) it cares about. No database migration or config change is involved.
## v0.12.0
### Multi-Worker API (`API_WORKERS`)
+2
View File
@@ -12,6 +12,7 @@ from meshcore_hub.api.routes.dashboard import router as dashboard_router
from meshcore_hub.api.routes.user_profiles import router as user_profiles_router
from meshcore_hub.api.routes.adoptions import router as adoptions_router
from meshcore_hub.api.routes.channels import router as channels_router
from meshcore_hub.api.routes.raw_packets import router as raw_packets_router
api_router = APIRouter()
@@ -30,3 +31,4 @@ api_router.include_router(dashboard_router, prefix="/dashboard", tags=["Dashboar
api_router.include_router(user_profiles_router, prefix="/user", tags=["User"])
api_router.include_router(adoptions_router, prefix="/adoptions", tags=["Adoptions"])
api_router.include_router(channels_router, prefix="/channels", tags=["Channels"])
api_router.include_router(raw_packets_router, prefix="/packets", tags=["Packets"])
@@ -228,6 +228,7 @@ def list_advertisements(
"advert_timestamp": adv.advert_timestamp,
"received_at": adv.received_at,
"created_at": adv.created_at,
"packet_hash": adv.packet_hash,
"observers": (
observers_by_hash.get(adv.event_hash, []) if adv.event_hash else []
),
@@ -313,6 +314,7 @@ def get_advertisement(
"advert_timestamp": adv.advert_timestamp,
"received_at": adv.received_at,
"created_at": adv.created_at,
"packet_hash": adv.packet_hash,
"observers": observers,
}
return AdvertisementRead(**data)
+2
View File
@@ -200,6 +200,7 @@ def list_messages(
"sender_timestamp": m.sender_timestamp,
"received_at": m.received_at,
"created_at": m.created_at,
"packet_hash": m.packet_hash,
"observers": (
observers_by_hash.get(m.event_hash, []) if m.event_hash else []
),
@@ -266,6 +267,7 @@ def get_message(
"sender_timestamp": message.sender_timestamp,
"received_at": message.received_at,
"created_at": message.created_at,
"packet_hash": message.packet_hash,
"observers": observers,
}
return MessageRead(**data)
+320
View File
@@ -0,0 +1,320 @@
"""Raw packet API routes."""
from datetime import datetime, timedelta, timezone
from typing import Optional
from fastapi import APIRouter, HTTPException, Query, Request
from sqlalchemy import String, cast, func, or_, select
from sqlalchemy.orm import aliased, selectinload
from meshcore_hub.api.auth import RequireRead
from meshcore_hub.api.cache import cached, sorted_query_string
from meshcore_hub.api.channel_visibility import (
get_max_visibility_level,
get_visible_channel_indices,
resolve_user_role,
)
from meshcore_hub.api.dependencies import DbSession
from meshcore_hub.common.models import Node, RawPacket
from meshcore_hub.common.schemas.raw_packets import RawPacketList, RawPacketRead
router = APIRouter()
VALID_PACKET_SORT_COLUMNS = {"time", "event_type", "packet_type", "snr", "path_len"}
DISABLE_FILTER_VALUES = {"all", "none", ""}
# Unanchored substring search over raw_hex cannot use an index (full scan). When
# the caller supplies no explicit time window we bound the scan to this many days
# so search runs against a recent slice rather than the whole table (see TR-17).
SEARCH_DEFAULT_WINDOW_DAYS = 7
def _packets_key_builder(request: Request) -> str:
"""Role-aware cache key so redacted responses never leak across roles."""
role = resolve_user_role(request) or "anonymous"
return f"packets:role={role}:{sorted_query_string(request)}"
def _get_tag_name(node: Optional[Node]) -> Optional[str]:
"""Extract name tag from a node's tags."""
if not node or not node.tags:
return None
for tag in node.tags:
if tag.key == "name":
return tag.value
return None
def _csv_set(value: Optional[str]) -> set[str]:
"""Split a comma-separated filter value into a normalized set."""
if not value:
return set()
return {t.strip() for t in value.split(",") if t.strip()}
@router.get("", response_model=RawPacketList)
@cached("packets", key_builder=_packets_key_builder)
def list_raw_packets(
_: RequireRead,
session: DbSession,
request: Request,
search: Optional[str] = Query(
None, description="Search in packet hash, raw hex, or observer name/key"
),
event_type: Optional[str] = Query(
None, description="Comma-separated classification filter"
),
packet_type: Optional[str] = Query(
None, description="Comma-separated wire packet type filter"
),
packet_hash: Optional[str] = Query(
None,
description="Filter by exact wire packet hash (links from adverts/messages)",
),
channel_idx: Optional[int] = Query(None, description="Filter by channel index"),
route_type: Optional[str] = Query(
None,
description="Comma-separated route types. Use 'all'/'none' to disable.",
),
public_key: Optional[str] = Query(
None, description="Filter by source public key prefix"
),
pubkey_prefix: Optional[str] = Query(
None, description="Filter by source public key prefix"
),
observed_by: Optional[list[str]] = Query(
None, description="Filter by receiver node public keys"
),
decryptable: Optional[bool] = Query(
None, description="Only packets whose payload decrypted (or the inverse)"
),
min_snr: Optional[float] = Query(None, description="Minimum SNR"),
max_snr: Optional[float] = Query(None, description="Maximum SNR"),
min_path_len: Optional[int] = Query(None, description="Minimum hop count"),
max_path_len: Optional[int] = Query(None, description="Maximum hop count"),
redacted: Optional[bool] = Query(
None, description="Include only / exclude redacted (metadata-only) rows"
),
since: Optional[datetime] = Query(None, description="Start timestamp"),
until: Optional[datetime] = Query(None, description="End timestamp"),
sort: Optional[str] = Query(None, description="Sort column"),
order: Optional[str] = Query(None, description="Sort direction: asc or desc"),
limit: int = Query(50, ge=1, le=100, description="Page size"),
offset: int = Query(0, ge=0, description="Page offset"),
) -> RawPacketList:
"""List raw packets with filtering, redaction, and pagination."""
ObserverNode = aliased(Node)
query = select(
RawPacket,
ObserverNode.public_key.label("observer_pk"),
ObserverNode.name.label("observer_name"),
ObserverNode.id.label("observer_id"),
).outerjoin(ObserverNode, RawPacket.observer_node_id == ObserverNode.id)
# Bound an unindexed substring search to a recent window when unconstrained.
if search and since is None:
since = datetime.now(timezone.utc) - timedelta(days=SEARCH_DEFAULT_WINDOW_DAYS)
if search:
pattern = f"%{search}%"
query = query.where(
or_(
RawPacket.packet_hash.ilike(pattern),
RawPacket.raw_hex.ilike(pattern),
ObserverNode.name.ilike(pattern),
ObserverNode.public_key.ilike(pattern),
)
)
event_types = _csv_set(event_type)
if event_types:
query = query.where(RawPacket.event_type.in_(event_types))
packet_types = {int(t) for t in _csv_set(packet_type) if t.lstrip("-").isdigit()}
if packet_types:
query = query.where(RawPacket.packet_type.in_(packet_types))
if packet_hash:
query = query.where(RawPacket.packet_hash == packet_hash)
if channel_idx is not None:
query = query.where(RawPacket.channel_idx == channel_idx)
if route_type and route_type.strip().lower() not in DISABLE_FILTER_VALUES:
requested = {t.lower() for t in _csv_set(route_type)}
if requested:
query = query.where(RawPacket.route_type.in_(requested))
source_prefix = public_key or pubkey_prefix
if source_prefix:
query = query.where(RawPacket.source_pubkey_prefix == source_prefix)
if observed_by:
query = query.where(ObserverNode.public_key.in_(observed_by))
if decryptable is not None:
# Best-effort: the decoder only emits a "decrypted" object when
# decryption succeeds. Match on the serialized JSON text.
decoded_text = cast(RawPacket.decoded, String)
has_decrypted = decoded_text.ilike('%"decrypted":%') & ~decoded_text.ilike(
'%"decrypted": null%'
)
query = query.where(has_decrypted if decryptable else ~has_decrypted)
if min_snr is not None:
query = query.where(RawPacket.snr >= min_snr)
if max_snr is not None:
query = query.where(RawPacket.snr <= max_snr)
if min_path_len is not None:
query = query.where(RawPacket.path_len >= min_path_len)
if max_path_len is not None:
query = query.where(RawPacket.path_len <= max_path_len)
if since:
query = query.where(RawPacket.received_at >= since)
if until:
query = query.where(RawPacket.received_at <= until)
# Channel-visibility: a row is "redacted" when it is a channel packet on a
# channel the role cannot see. Apply the `redacted` filter at SQL level (not
# post-fetch) so pagination counts stay stable.
role = resolve_user_role(request)
max_level = get_max_visibility_level(role)
visible_indices = get_visible_channel_indices(session, max_level)
is_redacted = RawPacket.channel_idx.is_not(None) & RawPacket.channel_idx.not_in(
visible_indices
)
if redacted is True:
query = query.where(is_redacted)
elif redacted is False:
query = query.where(~is_redacted)
# Total count over the filtered subquery
count_query = select(func.count()).select_from(query.subquery())
total = session.execute(count_query).scalar() or 0
# Sorting
sort = sort if sort in VALID_PACKET_SORT_COLUMNS else "time"
order = order if order in ("asc", "desc") else "desc"
sort_column = {
"time": RawPacket.received_at,
"event_type": RawPacket.event_type,
"packet_type": RawPacket.packet_type,
"snr": RawPacket.snr,
"path_len": RawPacket.path_len,
}[sort]
query = query.order_by(sort_column.desc() if order == "desc" else sort_column.asc())
query = query.offset(offset).limit(limit)
results = session.execute(query).all()
# Hydrate observer tags
observer_ids = {row.observer_id for row in results if row.observer_id}
nodes_by_id: dict[str, Node] = {}
if observer_ids:
nodes = (
session.execute(
select(Node)
.where(Node.id.in_(observer_ids))
.options(selectinload(Node.tags))
)
.scalars()
.all()
)
nodes_by_id = {n.id: n for n in nodes}
items = [
_build_read(
row[0],
observer_pk=row.observer_pk,
observer_name=row.observer_name,
observer_node=nodes_by_id.get(row.observer_id) if row.observer_id else None,
visible_indices=visible_indices,
)
for row in results
]
return RawPacketList(items=items, total=total, limit=limit, offset=offset)
def _build_read(
packet: RawPacket,
observer_pk: Optional[str],
observer_name: Optional[str],
observer_node: Optional[Node],
visible_indices: set[int],
) -> RawPacketRead:
"""Build a RawPacketRead, applying channel-visibility redaction."""
redacted = (
packet.channel_idx is not None and packet.channel_idx not in visible_indices
)
return RawPacketRead(
id=packet.id,
observed_by=observer_pk,
observer_name=observer_name,
observer_tag_name=_get_tag_name(observer_node),
packet_hash=packet.packet_hash,
raw_hex=None if redacted else packet.raw_hex,
packet_type=packet.packet_type,
payload_type=packet.payload_type,
event_type=packet.event_type,
channel_idx=packet.channel_idx,
source_pubkey_prefix=None if redacted else packet.source_pubkey_prefix,
route_type=packet.route_type,
path_len=packet.path_len,
snr=packet.snr,
decoded=None if redacted else packet.decoded,
redacted=redacted,
received_at=packet.received_at,
created_at=packet.created_at,
)
@router.get("/{packet_id}", response_model=RawPacketRead)
def get_raw_packet(
_: RequireRead,
session: DbSession,
request: Request,
packet_id: str,
) -> RawPacketRead:
"""Get a single raw packet by ID, subject to redaction rules."""
ObserverNode = aliased(Node)
result = session.execute(
select(
RawPacket,
ObserverNode.public_key.label("observer_pk"),
ObserverNode.name.label("observer_name"),
ObserverNode.id.label("observer_id"),
)
.outerjoin(ObserverNode, RawPacket.observer_node_id == ObserverNode.id)
.where(RawPacket.id == packet_id)
).one_or_none()
if not result:
raise HTTPException(status_code=404, detail="Raw packet not found")
role = resolve_user_role(request)
max_level = get_max_visibility_level(role)
visible_indices = get_visible_channel_indices(session, max_level)
observer_node = None
if result.observer_id:
observer_node = (
session.execute(
select(Node)
.where(Node.id == result.observer_id)
.options(selectinload(Node.tags))
)
.scalars()
.one_or_none()
)
return _build_read(
result[0],
observer_pk=result.observer_pk,
observer_name=result.observer_name,
observer_node=observer_node,
visible_indices=visible_indices,
)
+23
View File
@@ -17,6 +17,7 @@ from meshcore_hub.common.models import (
Message,
Node,
NodeTag,
RawPacket,
Telemetry,
TracePath,
)
@@ -34,6 +35,7 @@ class CleanupStats:
self.telemetry_deleted: int = 0
self.trace_paths_deleted: int = 0
self.event_logs_deleted: int = 0
self.raw_packets_deleted: int = 0
self.nodes_deleted: int = 0
self.observers_cleared: int = 0
self.total_deleted: int = 0
@@ -46,6 +48,7 @@ class CleanupStats:
f"telemetry={self.telemetry_deleted}, "
f"trace_paths={self.trace_paths_deleted}, "
f"event_logs={self.event_logs_deleted}, "
f"raw_packets={self.raw_packets_deleted}, "
f"nodes={self.nodes_deleted}, "
f"observers_cleared={self.observers_cleared})"
)
@@ -55,6 +58,7 @@ async def cleanup_old_data(
db: AsyncSession,
retention_days: int,
dry_run: bool = False,
raw_packet_retention_days: int | None = None,
) -> CleanupStats:
"""Delete event data older than the retention period.
@@ -62,12 +66,22 @@ async def cleanup_old_data(
db: Database session
retention_days: Number of days to retain data
dry_run: If True, only count records without deleting
raw_packet_retention_days: Days to retain raw packets (falls back to
``retention_days`` when None). Raw-packet cleanup runs whenever data
cleanup runs, independent of whether capture is currently enabled.
Returns:
CleanupStats object with deletion counts
"""
stats = CleanupStats()
cutoff_date = datetime.now(timezone.utc) - timedelta(days=retention_days)
raw_packet_cutoff = datetime.now(timezone.utc) - timedelta(
days=(
raw_packet_retention_days
if raw_packet_retention_days is not None
else retention_days
)
)
logger.info(
"Starting data cleanup (dry_run=%s, retention_days=%d, cutoff=%s)",
@@ -101,12 +115,18 @@ async def cleanup_old_data(
db, EventLog, cutoff_date, "event_logs", dry_run
)
# Clean up raw packets (independent retention window)
stats.raw_packets_deleted = await _cleanup_table(
db, RawPacket, raw_packet_cutoff, "raw_packets", dry_run
)
stats.total_deleted = (
stats.advertisements_deleted
+ stats.messages_deleted
+ stats.telemetry_deleted
+ stats.trace_paths_deleted
+ stats.event_logs_deleted
+ stats.raw_packets_deleted
)
# Clear the is_observer flag for nodes whose events were all pruned above.
@@ -134,6 +154,9 @@ def _observer_node_id_union() -> CompoundSelect:
select(TracePath.observer_node_id).where(
TracePath.observer_node_id.is_not(None)
),
select(RawPacket.observer_node_id).where(
RawPacket.observer_node_id.is_not(None)
),
select(EventObserver.observer_node_id),
)
+2
View File
@@ -281,6 +281,8 @@ def _run_collector_service(
node_cleanup_enabled=settings.node_cleanup_enabled,
node_cleanup_days=settings.node_cleanup_days,
channel_refresh_interval_seconds=settings.channel_refresh_interval_seconds,
raw_packet_capture_enabled=settings.raw_packet_capture_enabled,
raw_packet_retention_days=settings.effective_raw_packet_retention_days,
)
@@ -91,6 +91,9 @@ def handle_advertisement(
if delta <= timedelta(hours=4):
advert_timestamp_for_hash = advert_timestamp_dt
# LetsMesh wire packet hash, for linking to raw_packets
packet_hash = payload.get("packet_hash") or payload.get("hash")
# Compute event hash for deduplication (5-minute time bucket)
event_hash = compute_advertisement_hash(
public_key=adv_public_key,
@@ -195,6 +198,7 @@ def handle_advertisement(
flags=flags,
received_at=now,
event_hash=event_hash,
packet_hash=packet_hash,
route_type=route_type,
advert_timestamp=advert_timestamp_dt,
)
@@ -86,6 +86,9 @@ def _handle_message(
except (ValueError, OSError):
pass
# LetsMesh wire packet hash, for linking to raw_packets
packet_hash = payload.get("packet_hash") or payload.get("hash")
# Compute event hash for deduplication
event_hash = compute_message_hash(
text=text,
@@ -151,6 +154,7 @@ def _handle_message(
sender_timestamp=sender_timestamp,
received_at=now,
event_hash=event_hash,
packet_hash=packet_hash,
)
session.add(message)
@@ -0,0 +1,140 @@
"""Raw packet capture handler.
Stores one ``RawPacket`` row per observer reception from the LetsMesh
``packets`` feed, independent of how the packet is later classified. The decode
is reused from the normalizer's per-hex decode cache (a second ``decode_payload``
call is a cache hit, not a re-decode), so capture adds only a single insert plus
the observer upsert to the ingest path.
"""
import logging
from datetime import datetime, timezone
from typing import Any, Optional
from sqlalchemy import select
from meshcore_hub.collector.letsmesh_normalizer import LetsMeshNormalizer
from meshcore_hub.common.database import DatabaseManager
from meshcore_hub.common.models import Node, RawPacket
logger = logging.getLogger(__name__)
# Wire route-type codes -> labels (mirrors LetsMeshNormalizer._ROUTE_TYPE_MAP).
_ROUTE_TYPE_MAP: dict[int, str] = {
0: "transport_flood",
1: "flood",
2: "direct",
3: "transport_direct",
}
def _extract_source_pubkey_prefix(
decoded_packet: Optional[dict[str, Any]],
) -> Optional[str]:
"""Derive a 12-char sender prefix from the decoder sourceHash/senderPublicKey."""
decoded = LetsMeshNormalizer._extract_letsmesh_decoder_payload(decoded_packet)
if not decoded:
return None
for key in ("sourceHash", "senderPublicKey"):
prefix = LetsMeshNormalizer._normalize_pubkey_prefix(decoded.get(key))
if prefix:
return prefix
return None
def store_raw_packet(
public_key: str,
payload: dict[str, Any],
decoded_packet: Optional[dict[str, Any]],
event_type: str,
db: DatabaseManager,
) -> None:
"""Capture a single raw packet from the LetsMesh ``packets`` feed.
Args:
public_key: Observer (receiver) node public key from the MQTT topic
payload: Original LetsMesh upload payload (carries ``raw``, ``hash``, SNR)
decoded_packet: Decoder output already produced during normalization
event_type: How the collector classified the packet
db: Database manager
"""
now = datetime.now(timezone.utc)
raw_value = payload.get("raw")
raw_hex = raw_value.strip() if isinstance(raw_value, str) else None
hash_value = payload.get("hash")
packet_hash = hash_value if isinstance(hash_value, str) else None
payload_type = LetsMeshNormalizer._extract_letsmesh_decoder_payload_type(
decoded_packet
)
packet_type = LetsMeshNormalizer._parse_int(payload.get("packet_type"))
if packet_type is None:
packet_type = payload_type
channel_idx = LetsMeshNormalizer._parse_int(payload.get("channel_idx"))
if channel_idx is None:
channel_hash = LetsMeshNormalizer._extract_letsmesh_decoder_channel_hash(
decoded_packet
)
if channel_hash:
channel_idx = LetsMeshNormalizer._parse_channel_hash_idx(channel_hash)
source_pubkey_prefix = _extract_source_pubkey_prefix(decoded_packet)
route_type: Optional[str] = None
if isinstance(decoded_packet, dict):
route_raw = LetsMeshNormalizer._parse_int(decoded_packet.get("routeType"))
if route_raw in _ROUTE_TYPE_MAP:
route_type = _ROUTE_TYPE_MAP[route_raw]
path_len = LetsMeshNormalizer._parse_path_length(payload.get("path"))
if path_len is None:
path_len = LetsMeshNormalizer._extract_letsmesh_decoder_path_length(
decoded_packet
)
snr = LetsMeshNormalizer._parse_float(payload.get("SNR"))
if snr is None:
snr = LetsMeshNormalizer._parse_float(payload.get("snr"))
with db.session_scope() as session:
observer_node = None
if public_key:
observer_node = session.execute(
select(Node).where(Node.public_key == public_key)
).scalar_one_or_none()
if not observer_node:
observer_node = Node(
public_key=public_key,
first_seen=now,
last_seen=now,
is_observer=True,
)
session.add(observer_node)
session.flush()
else:
observer_node.last_seen = now
if not observer_node.is_observer:
observer_node.is_observer = True
session.add(
RawPacket(
observer_node_id=observer_node.id if observer_node else None,
packet_hash=packet_hash,
raw_hex=raw_hex,
packet_type=packet_type,
payload_type=payload_type,
event_type=event_type,
channel_idx=channel_idx,
source_pubkey_prefix=source_pubkey_prefix,
route_type=route_type,
path_len=path_len,
snr=snr,
decoded=decoded_packet,
received_at=now,
)
)
logger.debug("Captured raw packet: %s (%s)", packet_hash or "unknown", event_type)
@@ -74,7 +74,10 @@ class LetsMeshNormalizer:
normalized_packet_payload["decoded_payload_type"] = (
decoded_payload_type
)
return observer_public_key, "letsmesh_packet", normalized_packet_payload
fallback_event_type = self._classify_fallback_event_type(
payload, decoded_packet
)
return observer_public_key, fallback_event_type, normalized_packet_payload
if feed_type == "internal":
return observer_public_key, "letsmesh_internal", payload
@@ -140,6 +143,8 @@ class LetsMeshNormalizer:
txt_type if txt_type is not None else packet_type
)
normalized_payload["signature"] = payload.get("signature") or packet_hash
if packet_hash_text:
normalized_payload["packet_hash"] = packet_hash_text
path_len = self._parse_path_length(payload.get("path"))
if path_len is None:
path_len = self._extract_letsmesh_decoder_path_length(decoded_packet)
@@ -549,6 +554,44 @@ class LetsMeshNormalizer:
3: "transport_direct",
}
# MeshCore payload type -> event_type for packets that reach the fallback
# (i.e. the structured/message/advert builders declined). Reaching this path
# for TXT_MSG (2) / GRP_TXT (5) means the payload did not decrypt, so they
# are labelled as encrypted rather than as messages. Unknown/unresolvable
# types keep the generic ``letsmesh_packet`` safety-net label.
_FALLBACK_EVENT_TYPES: dict[int, str] = {
0: "req", # PAYLOAD_TYPE_REQ
1: "response", # PAYLOAD_TYPE_RESPONSE
2: "encrypted_direct", # PAYLOAD_TYPE_TXT_MSG (undecryptable)
3: "ack", # PAYLOAD_TYPE_ACK
4: "advert", # PAYLOAD_TYPE_ADVERT (no identity metadata)
5: "encrypted_channel", # PAYLOAD_TYPE_GRP_TXT (unknown channel key)
6: "grp_data", # PAYLOAD_TYPE_GRP_DATA
7: "anon_req", # PAYLOAD_TYPE_ANON_REQ
8: "path", # PAYLOAD_TYPE_PATH
9: "trace", # PAYLOAD_TYPE_TRACE
10: "multipart", # PAYLOAD_TYPE_MULTIPART
11: "control", # PAYLOAD_TYPE_CONTROL
15: "raw_custom", # PAYLOAD_TYPE_RAW_CUSTOM
}
@classmethod
def _classify_fallback_event_type(
cls,
payload: dict[str, Any],
decoded_packet: dict[str, Any] | None,
) -> str:
"""Resolve a specific event type for an otherwise-unhandled packet.
Maps the MeshCore payload type to a precise label so uncategorised
packets are no longer all lumped under ``letsmesh_packet``. Falls back
to ``letsmesh_packet`` only when the payload type cannot be resolved.
"""
packet_type = cls._resolve_letsmesh_packet_type(payload, decoded_packet)
if packet_type is None:
return "letsmesh_packet"
return cls._FALLBACK_EVENT_TYPES.get(packet_type, "letsmesh_packet")
def _build_letsmesh_advertisement_payload(
self,
payload: dict[str, Any],
@@ -583,6 +626,10 @@ class LetsMeshNormalizer:
"public_key": public_key,
}
advert_hash = payload.get("hash")
if isinstance(advert_hash, str) and advert_hash:
normalized_payload["packet_hash"] = advert_hash
route_type_raw = self._parse_int(decoded_packet.get("routeType"))
if route_type_raw is not None and route_type_raw in self._ROUTE_TYPE_MAP:
normalized_payload["route_type"] = self._ROUTE_TYPE_MAP[route_type_raw]
+58
View File
@@ -48,6 +48,8 @@ class Subscriber(LetsMeshNormalizer):
node_cleanup_enabled: bool = False,
node_cleanup_days: int = 90,
channel_refresh_interval_seconds: int = 300,
raw_packet_capture_enabled: bool = False,
raw_packet_retention_days: int = 30,
):
"""Initialize subscriber.
@@ -61,6 +63,8 @@ class Subscriber(LetsMeshNormalizer):
node_cleanup_enabled: Enable automatic cleanup of inactive nodes
node_cleanup_days: Remove nodes not seen for this many days
channel_refresh_interval_seconds: Seconds between channel key refresh
raw_packet_capture_enabled: Capture every packets-feed packet to raw_packets
raw_packet_retention_days: Days to retain raw packets
"""
self.mqtt = mqtt_client
self.db = db_manager
@@ -83,6 +87,14 @@ class Subscriber(LetsMeshNormalizer):
self._node_cleanup_days = node_cleanup_days
self._cleanup_thread: Optional[threading.Thread] = None
self._last_cleanup: Optional[datetime] = None
# Raw packet capture
self._raw_packet_capture_enabled = raw_packet_capture_enabled
self._raw_packet_retention_days = raw_packet_retention_days
logger.info(
"Raw packet capture %s (retention=%d days)",
"enabled" if raw_packet_capture_enabled else "disabled",
raw_packet_retention_days,
)
# Channel key refresh
self._channel_refresh_interval_seconds = channel_refresh_interval_seconds
self._channel_refresh_thread: Optional[threading.Thread] = None
@@ -208,8 +220,43 @@ class Subscriber(LetsMeshNormalizer):
public_key, event_type, normalized_payload = parsed
logger.debug("Received event: %s from %s...", event_type, public_key[:12])
# Capture the raw packet (packets feed only) independent of, and before,
# structured dispatch so the raw_packets table is complete. The boolean
# short-circuit avoids the insert entirely when capture is disabled.
if self._raw_packet_capture_enabled:
self._maybe_capture_raw_packet(topic, public_key, event_type, payload)
self._dispatch_event(public_key, event_type, normalized_payload)
def _maybe_capture_raw_packet(
self,
topic: str,
public_key: str,
event_type: str,
payload: dict[str, Any],
) -> None:
"""Persist a raw_packets row for a packets-feed reception.
Reuses the decode the normalizer already performed (the decoder caches
per raw hex, so this ``decode_payload`` call is a cache hit). Capture
failures are logged and never block event dispatch.
"""
try:
parsed_topic = self.mqtt.topic_builder.parse_letsmesh_upload_topic(topic)
if not parsed_topic:
return
_, feed_type = parsed_topic
if feed_type != "packets":
return
from meshcore_hub.collector.handlers.raw_packet import store_raw_packet
decoded_packet = self._letsmesh_decoder.decode_payload(payload)
store_raw_packet(public_key, payload, decoded_packet, event_type, self.db)
except Exception as e:
logger.error("Error capturing raw packet: %s", e)
def _dispatch_event(
self,
public_key: str,
@@ -370,6 +417,9 @@ class Subscriber(LetsMeshNormalizer):
session,
self._cleanup_retention_days,
dry_run=False,
raw_packet_retention_days=(
self._raw_packet_retention_days
),
)
logger.info(
"Event cleanup completed: %s", stats
@@ -600,6 +650,8 @@ def create_subscriber(
node_cleanup_enabled: bool = False,
node_cleanup_days: int = 90,
channel_refresh_interval_seconds: int = 300,
raw_packet_capture_enabled: bool = False,
raw_packet_retention_days: int = 30,
) -> Subscriber:
"""Create a configured subscriber instance.
@@ -653,6 +705,8 @@ def create_subscriber(
node_cleanup_enabled=node_cleanup_enabled,
node_cleanup_days=node_cleanup_days,
channel_refresh_interval_seconds=channel_refresh_interval_seconds,
raw_packet_capture_enabled=raw_packet_capture_enabled,
raw_packet_retention_days=raw_packet_retention_days,
)
# Register handlers
@@ -680,6 +734,8 @@ def run_collector(
node_cleanup_enabled: bool = False,
node_cleanup_days: int = 90,
channel_refresh_interval_seconds: int = 300,
raw_packet_capture_enabled: bool = False,
raw_packet_retention_days: int = 30,
) -> None:
"""Run the collector (blocking).
@@ -718,6 +774,8 @@ def run_collector(
node_cleanup_enabled=node_cleanup_enabled,
node_cleanup_days=node_cleanup_days,
channel_refresh_interval_seconds=channel_refresh_interval_seconds,
raw_packet_capture_enabled=raw_packet_capture_enabled,
raw_packet_retention_days=raw_packet_retention_days,
)
# Set up signal handlers
+24
View File
@@ -143,6 +143,26 @@ class CollectorSettings(CommonSettings):
ge=10,
)
# Raw packet capture settings
raw_packet_capture_enabled: bool = Field(
default=False,
description="Capture every inbound packets-feed packet into raw_packets",
)
raw_packet_retention_days: Optional[int] = Field(
default=None,
description=(
"Days to retain raw packets (defaults to DATA_RETENTION_DAYS when unset)"
),
ge=1,
)
@property
def effective_raw_packet_retention_days(self) -> int:
"""Resolve raw-packet retention, falling back to the global default."""
if self.raw_packet_retention_days is not None:
return self.raw_packet_retention_days
return self.data_retention_days
@property
def collector_data_dir(self) -> str:
"""Get the collector data directory path."""
@@ -411,6 +431,9 @@ class WebSettings(CommonSettings):
feature_channels: bool = Field(
default=True, description="Enable the /channels page"
)
feature_packets: bool = Field(
default=False, description="Enable the /packets page (off by default)"
)
feature_pages: bool = Field(
default=True, description="Enable custom markdown pages"
)
@@ -444,6 +467,7 @@ class WebSettings(CommonSettings):
"map": self.feature_map and self.feature_nodes,
"members": self.feature_members and self.oidc_enabled,
"channels": self.feature_channels,
"packets": self.feature_packets,
"pages": self.feature_pages,
"radio_config": self.feature_radio_config,
}
@@ -8,6 +8,7 @@ from meshcore_hub.common.models.advertisement import Advertisement
from meshcore_hub.common.models.trace_path import TracePath
from meshcore_hub.common.models.telemetry import Telemetry
from meshcore_hub.common.models.event_log import EventLog
from meshcore_hub.common.models.raw_packet import RawPacket
from meshcore_hub.common.models.user_profile import UserProfile
from meshcore_hub.common.models.user_profile_node import UserProfileNode
from meshcore_hub.common.models.event_observer import EventObserver, add_event_observer
@@ -23,6 +24,7 @@ __all__ = [
"TracePath",
"Telemetry",
"EventLog",
"RawPacket",
"UserProfile",
"UserProfileNode",
"EventObserver",
@@ -63,6 +63,12 @@ class Advertisement(Base, UUIDMixin, TimestampMixin):
nullable=True,
unique=True,
)
# LetsMesh wire packet hash, links this advert to its raw_packets rows.
packet_hash: Mapped[Optional[str]] = mapped_column(
String(32),
nullable=True,
index=True,
)
route_type: Mapped[Optional[str]] = mapped_column(
String(20),
nullable=True,
@@ -81,6 +81,12 @@ class Message(Base, UUIDMixin, TimestampMixin):
nullable=True,
unique=True,
)
# LetsMesh wire packet hash, links this event to its raw_packets rows.
packet_hash: Mapped[Optional[str]] = mapped_column(
String(32),
nullable=True,
index=True,
)
__table_args__ = (
Index("ix_messages_message_type", "message_type"),
@@ -0,0 +1,122 @@
"""RawPacket model for storing raw decoded wire packets."""
from datetime import datetime
from typing import Any, Optional
from sqlalchemy import DateTime, Float, ForeignKey, Index, Integer, String, Text
from sqlalchemy.dialects.sqlite import JSON
from sqlalchemy.orm import Mapped, mapped_column
from meshcore_hub.common.models.base import Base, TimestampMixin, UUIDMixin, utc_now
class RawPacket(Base, UUIDMixin, TimestampMixin):
"""Raw on-air packet captured from the LetsMesh ``packets`` feed.
One row is stored per observer reception (no deduplication). Rows are
linkable back to the structured tables (messages, advertisements, ...) by
``packet_hash``. The raw hex and decoded summary are retained so the API can
serve a complete searchable record of wire traffic without re-decoding.
Attributes:
id: UUID primary key
observer_node_id: FK to nodes (the receiving interface)
packet_hash: LetsMesh packet hash, links rows to structured records and
groups multi-observer receptions
raw_hex: The on-air bytes from ``payload["raw"]``
packet_type: Wire packet type
payload_type: Decoder payload type
event_type: How the collector classified the packet
channel_idx: ``int(channelHash, 16)`` for channel-message packets, else
NULL; drives channel-visibility redaction
source_pubkey_prefix: Sender prefix derived from the decoder sourceHash /
senderPublicKey, for efficient "packets from this node" filtering
route_type: Mapped route type (flood, transport_flood, direct, ...)
path_len: Hop count
snr: Signal-to-noise ratio as reported by the observer
decoded: Decoder summary JSON, so the detail view needs no re-decode
received_at: When received by the observing interface
created_at: Record creation timestamp
"""
__tablename__ = "raw_packets"
observer_node_id: Mapped[Optional[str]] = mapped_column(
ForeignKey("nodes.id", ondelete="SET NULL"),
nullable=True,
index=True,
)
packet_hash: Mapped[Optional[str]] = mapped_column(
String(32),
nullable=True,
index=True,
)
raw_hex: Mapped[Optional[str]] = mapped_column(
Text,
nullable=True,
)
packet_type: Mapped[Optional[int]] = mapped_column(
Integer,
nullable=True,
index=True,
)
payload_type: Mapped[Optional[int]] = mapped_column(
Integer,
nullable=True,
)
event_type: Mapped[Optional[str]] = mapped_column(
String(50),
nullable=True,
index=True,
)
channel_idx: Mapped[Optional[int]] = mapped_column(
Integer,
nullable=True,
index=True,
)
source_pubkey_prefix: Mapped[Optional[str]] = mapped_column(
String(12),
nullable=True,
index=True,
)
route_type: Mapped[Optional[str]] = mapped_column(
String(20),
nullable=True,
)
path_len: Mapped[Optional[int]] = mapped_column(
Integer,
nullable=True,
)
snr: Mapped[Optional[float]] = mapped_column(
Float,
nullable=True,
)
decoded: Mapped[Optional[dict[str, Any]]] = mapped_column(
JSON,
nullable=True,
)
received_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
__table_args__ = (
Index("ix_raw_packets_received_at", "received_at"),
# Composite indexes serve the common "filter then sort by newest" shape
# without a full scan + filesort. See plan TR-17 for the write-cost
# trade-off; do not add further composites speculatively.
Index("ix_raw_packets_event_type_received_at", "event_type", "received_at"),
Index("ix_raw_packets_channel_idx_received_at", "channel_idx", "received_at"),
Index(
"ix_raw_packets_source_pubkey_prefix_received_at",
"source_pubkey_prefix",
"received_at",
),
)
def __repr__(self) -> str:
return (
f"<RawPacket(id={self.id}, event_type={self.event_type}, "
f"packet_hash={self.packet_hash})>"
)
@@ -28,6 +28,10 @@ from meshcore_hub.common.schemas.messages import (
from meshcore_hub.common.schemas.network import (
RadioConfig,
)
from meshcore_hub.common.schemas.raw_packets import (
RawPacketRead,
RawPacketList,
)
__all__ = [
# Events
@@ -54,4 +58,7 @@ __all__ = [
"MessageFilters",
# Network
"RadioConfig",
# Raw packets
"RawPacketRead",
"RawPacketList",
]
@@ -58,6 +58,9 @@ class MessageRead(BaseModel):
)
received_at: datetime = Field(..., description="When received by interface")
created_at: datetime = Field(..., description="Record creation timestamp")
packet_hash: Optional[str] = Field(
default=None, description="LetsMesh wire packet hash, links to raw packets"
)
observers: list[ObserverInfo] = Field(
default_factory=list, description="All observers that captured this message"
)
@@ -138,6 +141,9 @@ class AdvertisementRead(BaseModel):
)
received_at: datetime = Field(..., description="When received")
created_at: datetime = Field(..., description="Record creation timestamp")
packet_hash: Optional[str] = Field(
default=None, description="LetsMesh wire packet hash, links to raw packets"
)
observers: list[ObserverInfo] = Field(
default_factory=list,
description="All observers that captured this advertisement",
@@ -0,0 +1,67 @@
"""Pydantic schemas for raw packet API endpoints."""
from datetime import datetime
from typing import Any, Optional
from pydantic import BaseModel, Field
class RawPacketRead(BaseModel):
"""Schema for reading a raw packet.
When ``redacted`` is true the packet was observed on a channel above the
requesting role's visibility level: ``raw_hex`` and ``decoded`` are nulled
out and only non-sensitive metadata is retained.
"""
id: str = Field(..., description="Raw packet UUID")
observed_by: Optional[str] = Field(
default=None, description="Observing interface node public key"
)
observer_name: Optional[str] = Field(default=None, description="Observer node name")
observer_tag_name: Optional[str] = Field(
default=None, description="Observer name from tags"
)
packet_hash: Optional[str] = Field(
default=None, description="LetsMesh packet hash (links to structured records)"
)
raw_hex: Optional[str] = Field(
default=None, description="On-air packet bytes as hex (null when redacted)"
)
packet_type: Optional[int] = Field(default=None, description="Wire packet type")
payload_type: Optional[int] = Field(
default=None, description="Decoder payload type"
)
event_type: Optional[str] = Field(
default=None, description="How the collector classified the packet"
)
channel_idx: Optional[int] = Field(
default=None, description="Channel index for channel-message packets"
)
source_pubkey_prefix: Optional[str] = Field(
default=None, description="Sender public key prefix"
)
route_type: Optional[str] = Field(default=None, description="Route type")
path_len: Optional[int] = Field(default=None, description="Hop count")
snr: Optional[float] = Field(default=None, description="Signal-to-noise ratio")
decoded: Optional[dict[str, Any]] = Field(
default=None, description="Decoder summary (null when redacted)"
)
redacted: bool = Field(
default=False,
description="True when the payload was redacted for channel visibility",
)
received_at: datetime = Field(..., description="When received")
created_at: datetime = Field(..., description="Record creation timestamp")
class Config:
from_attributes = True
class RawPacketList(BaseModel):
"""Schema for paginated raw packet list response."""
items: list[RawPacketRead] = Field(..., description="List of raw packets")
total: int = Field(..., description="Total number of raw packets")
limit: int = Field(..., description="Page size limit")
offset: int = Field(..., description="Page offset")
+3
View File
@@ -105,6 +105,9 @@ def _build_endpoint_access(
"v1/advertisements": {
"GET": _OPEN,
},
"v1/packets": {
"GET": _OPEN,
},
"v1/dashboard": {
"GET": _OPEN,
},
+4
View File
@@ -24,6 +24,7 @@
--color-adverts: oklch(0.7 0.17 330); /* magenta */
--color-messages: oklch(0.75 0.18 180); /* teal */
--color-channels: oklch(0.72 0.15 300); /* purple */
--color-packets: oklch(0.72 0.17 145); /* green */
--color-map: oklch(0.8471 0.199 83.87); /* yellow (matches btn-warning) */
--color-members: oklch(0.72 0.17 50); /* orange */
--color-neutral: oklch(0.3 0.01 250); /* subtle dark grey */
@@ -37,6 +38,7 @@
--color-adverts: oklch(0.55 0.17 330);
--color-messages: oklch(0.55 0.18 180);
--color-channels: oklch(0.55 0.15 300);
--color-packets: oklch(0.52 0.17 145);
--color-map: oklch(0.58 0.16 45);
--color-members: oklch(0.55 0.18 25);
--color-neutral: oklch(0.85 0.01 250);
@@ -68,6 +70,7 @@
.nav-icon-nodes { color: var(--color-nodes); }
.nav-icon-adverts { color: var(--color-adverts); }
.nav-icon-messages { color: var(--color-messages); }
.nav-icon-packets { color: var(--color-packets); }
.nav-icon-map { color: var(--color-map); }
.nav-icon-members { color: var(--color-members); }
@@ -76,6 +79,7 @@
.navbar .menu li:has(.nav-icon-nodes) { --nav-color: var(--color-nodes); }
.navbar .menu li:has(.nav-icon-adverts) { --nav-color: var(--color-adverts); }
.navbar .menu li:has(.nav-icon-messages) { --nav-color: var(--color-messages); }
.navbar .menu li:has(.nav-icon-packets) { --nav-color: var(--color-packets); }
.navbar .menu li:has(.nav-icon-map) { --nav-color: var(--color-map); }
.navbar .menu li:has(.nav-icon-members) { --nav-color: var(--color-members); }
+11 -1
View File
@@ -9,7 +9,7 @@ import { Router } from './router.js';
import { isAbortError } from './api.js';
import { html, litRender, getConfig, hasRole, renderAuthSection } from './components.js';
import { loadLocale, t } from './i18n.js';
import { iconHome, iconDashboard, iconNodes, iconAdvertisements, iconMessages, iconMap, iconMembers, iconPage, iconChannel } from './icons.js';
import { iconHome, iconDashboard, iconNodes, iconAdvertisements, iconMessages, iconPackets, iconMap, iconMembers, iconPage, iconChannel } from './icons.js';
// Page modules (lazy-loaded)
const pages = {
@@ -19,6 +19,8 @@ const pages = {
nodeDetail: () => import('./pages/node-detail.js'),
messages: () => import('./pages/messages.js'),
advertisements: () => import('./pages/advertisements.js'),
packets: () => import('./pages/packets.js'),
packetDetail: () => import('./pages/packet-detail.js'),
map: () => import('./pages/map.js'),
members: () => import('./pages/members.js'),
channels: () => import('./pages/channels.js'),
@@ -83,6 +85,10 @@ if (features.messages !== false) {
if (features.advertisements !== false) {
router.addRoute('/advertisements', pageHandler(pages.advertisements));
}
if (features.packets === true) {
router.addRoute('/packets', pageHandler(pages.packets));
router.addRoute('/packets/:id', pageHandler(pages.packetDetail));
}
if (features.map !== false) {
router.addRoute('/map', pageHandler(pages.map));
}
@@ -159,6 +165,7 @@ function updatePageTitle(pathname) {
if (features.channels !== false) titles['/channels'] = composePageTitle('entities.channels');
if (features.messages !== false) titles['/messages'] = composePageTitle('entities.messages');
if (features.advertisements !== false) titles['/advertisements'] = composePageTitle('entities.advertisements');
if (features.packets === true) titles['/packets'] = composePageTitle('entities.packets');
if (features.map !== false) titles['/map'] = composePageTitle('entities.map');
if (features.members !== false) titles['/members'] = composePageTitle('entities.members');
titles['/profile'] = composePageTitle('links.profile');
@@ -212,6 +219,9 @@ function renderMobileNav(config) {
if (features.messages !== false) {
items.push(html`<li><a href="/messages" data-nav-link>${iconMessages('h-5 w-5 nav-icon-messages')} ${t('entities.messages')}</a></li>`);
}
if (features.packets === true) {
items.push(html`<li><a href="/packets" data-nav-link>${iconPackets('h-5 w-5 nav-icon-packets')} ${t('entities.packets')}</a></li>`);
}
if (features.map !== false) {
items.push(html`<li><a href="/map" data-nav-link>${iconMap('h-5 w-5 nav-icon-map')} ${t('entities.map')}</a></li>`);
}
@@ -26,6 +26,10 @@ export function iconMessages(cls = 'h-5 w-5') {
return html`<svg xmlns="http://www.w3.org/2000/svg" class=${cls} fill="none" viewBox="0 0 24 24" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 10h.01M12 10h.01M16 10h.01M9 16H5a2 2 0 01-2-2V6a2 2 0 012-2h14a2 2 0 012 2v8a2 2 0 01-2 2h-5l-5 5v-5z" /></svg>`;
}
export function iconPackets(cls = 'h-5 w-5') {
return html`<svg xmlns="http://www.w3.org/2000/svg" class=${cls} fill="none" viewBox="0 0 24 24" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M20 7l-8-4-8 4m16 0l-8 4m8-4v10l-8 4m0-10L4 7m8 4v10M4 7v10l8 4" /></svg>`;
}
export function iconHome(cls = 'h-5 w-5') {
return html`<svg xmlns="http://www.w3.org/2000/svg" class=${cls} fill="none" viewBox="0 0 24 24" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M3 12l2-2m0 0l7-7 7 7M5 10v10a1 1 0 001 1h3m10-11l2 2m-2-2v10a1 1 0 01-1 1h-3m-6 0a1 1 0 001-1v-4a1 1 0 011-1h2a1 1 0 011 1v4a1 1 0 001 1m-6 0h6" /></svg>`;
}
@@ -8,6 +8,23 @@ import {
observerIcons, observerDetailRow, toggleObserverDetail, toggleCardObserverDetail
} from '../components.js';
import { createAutoRefresh } from '../auto-refresh.js';
import { iconPackets } from '../icons.js';
function packetLink(packetHash, navigate) {
if (!packetHash) {
return nothing;
}
const icon = iconPackets('h-5 w-5 nav-icon-packets');
const title = t('packets.view_raw');
const url = `/packets?packet_hash=${packetHash}`;
if (navigate) {
// Inside a card that is itself an <a>: use a span to avoid nested anchors.
return html`<span class="inline-flex cursor-pointer" title="${title}"
@click=${(e) => { e.preventDefault(); e.stopPropagation(); navigate(url); }}>${icon}</span>`;
}
return html`<a href="${url}" class="inline-flex" title="${title}"
@click=${(e) => e.stopPropagation()}>${icon}</a>`;
}
function routeTypeBadge(routeType) {
if (!routeType) {
@@ -38,6 +55,8 @@ export async function render(container, params, router) {
const order = query.order || 'desc';
const config = getConfig();
const features = config.features || {};
const packetsEnabled = features.packets === true;
const tz = config.timezone || '';
const tzBadge = tz && tz !== 'UTC' ? html`<span class="text-sm opacity-60">${tz}</span>` : nothing;
const navigate = (url) => router.navigate(url);
@@ -143,6 +162,7 @@ ${displayContent}`, container);
<div class="flex items-center justify-end gap-1">
${routeTypeBadge(ad.route_type)}
${receiversBlock}
${packetsEnabled ? packetLink(ad.packet_hash, navigate) : nothing}
</div>
</div>
</div>
@@ -170,7 +190,7 @@ ${displayContent}`, container);
});
const tableRows = advertisements.length === 0
? html`<tr><td colspan="5" class="text-center py-8 opacity-70">${t('common.no_entity_found', { entity: t('entities.advertisements').toLowerCase() })}</td></tr>`
? html`<tr><td colspan=${packetsEnabled ? 6 : 5} class="text-center py-8 opacity-70">${t('common.no_entity_found', { entity: t('entities.advertisements').toLowerCase() })}</td></tr>`
: advertisements.map(ad => {
const adName = ad.node_tag_name || ad.node_name || ad.name;
const adDescription = ad.node_tag_description;
@@ -202,6 +222,7 @@ ${displayContent}`, container);
<td>${routeTypeBadge(ad.route_type)}</td>
<td class="text-sm whitespace-nowrap">${formatDateTime(ad.received_at)}</td>
<td>${receiversBlock}</td>
${packetsEnabled ? html`<td>${packetLink(ad.packet_hash)}</td>` : nothing}
</tr>${observerDetailRow(ad.observers || [], null, { hidePath: true })}`;
});
@@ -299,6 +320,7 @@ ${mobileSortSelect({
<th>${t('advertisements.col_route_type')}</th>
${sortable(t('common.time'), 'time')}
<th>${t('common.observers')}</th>
${packetsEnabled ? html`<th></th>` : nothing}
</tr>
</thead>
<tbody>
@@ -4,7 +4,7 @@ import {
getConfig, errorAlert, pageColors, renderStatCard, t,
} from '../components.js';
import {
iconDashboard, iconNodes, iconAdvertisements, iconMessages, iconMembers, iconMap,
iconDashboard, iconNodes, iconAdvertisements, iconMessages, iconPackets, iconMembers, iconMap,
iconPage, iconInfo, iconChart, iconAntenna, iconUsers, iconChannel,
iconSettings, iconFrequency, iconBandwidth, iconSpreadingFactor, iconCodingRate, iconTxPower,
} from '../icons.js';
@@ -105,11 +105,11 @@ function renderHeroSection({ networkName, logoUrl, logoInvertLight, networkCity,
label: t('entities.messages'),
colorVar: '--color-messages',
}) : nothing}
${features.members !== false ? renderNavCard({
href: '/members',
icon: iconMembers('w-full h-full'),
label: t('entities.members'),
colorVar: '--color-members',
${features.packets === true ? renderNavCard({
href: '/packets',
icon: iconPackets('w-full h-full'),
label: t('entities.packets'),
colorVar: '--color-packets',
}) : nothing}
${features.map !== false ? renderNavCard({
href: '/map',
@@ -117,6 +117,12 @@ function renderHeroSection({ networkName, logoUrl, logoInvertLight, networkCity,
label: t('entities.map'),
colorVar: '--color-map',
}) : nothing}
${features.members !== false ? renderNavCard({
href: '/members',
icon: iconMembers('w-full h-full'),
label: t('entities.members'),
colorVar: '--color-members',
}) : nothing}
</div>
${features.pages !== false && customPages.length > 0 ? html`
<div class="flex flex-wrap justify-center gap-3 mt-4">
@@ -9,6 +9,22 @@ import {
observerIcons, observerDetailRow, toggleObserverDetail, toggleCardObserverDetail
} from '../components.js';
import { createAutoRefresh } from '../auto-refresh.js';
import { iconPackets } from '../icons.js';
function packetLink(packetHash, navigate) {
if (!packetHash) {
return nothing;
}
const icon = iconPackets('h-5 w-5 nav-icon-packets');
const title = t('packets.view_raw');
const url = `/packets?packet_hash=${packetHash}`;
if (navigate) {
return html`<span class="inline-flex cursor-pointer" title="${title}"
@click=${(e) => { e.preventDefault(); e.stopPropagation(); navigate(url); }}>${icon}</span>`;
}
return html`<a href="${url}" class="inline-flex" title="${title}"
@click=${(e) => e.stopPropagation()}>${icon}</a>`;
}
export async function render(container, params, router) {
const { signal } = params || {};
@@ -25,6 +41,8 @@ export async function render(container, params, router) {
const order = query.order || 'desc';
const config = getConfig();
const features = config.features || {};
const packetsEnabled = features.packets === true;
let channelLabels = new Map();
const tz = config.timezone || '';
const tzBadge = tz && tz !== 'UTC' ? html`<span class="text-sm opacity-60">${tz}</span>` : nothing;
@@ -265,6 +283,7 @@ ${displayContent}`, container);
</div>
<div class="flex items-center gap-2 flex-shrink-0">
${receiversBlock}
${packetsEnabled ? packetLink(msg.packet_hash) : nothing}
</div>
</div>
<p class="text-sm mt-2 break-words whitespace-pre-wrap">${displayMessage}</p>
@@ -294,7 +313,7 @@ ${displayContent}`, container);
});
const tableRows = messages.length === 0
? html`<tr><td colspan="5" class="text-center py-8 opacity-70">${t('common.no_entity_found', { entity: t('entities.messages').toLowerCase() })}</td></tr>`
? html`<tr><td colspan=${packetsEnabled ? 6 : 5} class="text-center py-8 opacity-70">${t('common.no_entity_found', { entity: t('entities.messages').toLowerCase() })}</td></tr>`
: messages.map(msg => {
const isChannel = msg.message_type === 'channel';
const typeIcon = isChannel ? '\u{1F4FB}' : '\u{1F464}';
@@ -321,6 +340,7 @@ ${displayContent}`, container);
</td>
<td class="break-words max-w-md" style="white-space: pre-wrap;">${displayMessage}</td>
<td>${receiversBlock}</td>
${packetsEnabled ? html`<td>${packetLink(msg.packet_hash)}</td>` : nothing}
</tr>${observerDetailRow(msg.observers || [])}`;
});
@@ -427,6 +447,7 @@ ${mobileSortSelect({
${sortable(t('common.from'), 'from')}
${sortable(t('entities.message'), 'message')}
<th>${t('common.observers')}</th>
${packetsEnabled ? html`<th></th>` : nothing}
</tr>
</thead>
<tbody>
@@ -0,0 +1,119 @@
import { apiGet, isAbortError } from '../api.js';
import {
html, litRender, nothing, t,
getConfig, formatDateTime, warningBadge, copyToClipboard
} from '../components.js';
function field(label, value) {
return html`
<div class="flex flex-col gap-0.5 py-2 border-b border-base-200">
<span class="text-xs uppercase opacity-60">${label}</span>
<span class="text-sm">${value}</span>
</div>`;
}
export async function render(container, params, router) {
const { signal } = params || {};
const id = params.id;
const config = getConfig();
const tz = config.timezone || '';
const tzBadge = tz && tz !== 'UTC' ? html`<span class="text-sm opacity-60">${tz}</span>` : nothing;
function shell(content, leaf) {
litRender(html`
<div class="breadcrumbs text-sm mb-4">
<ul>
<li><a href="/">${t('entities.home')}</a></li>
<li><a href="/packets">${t('entities.packets')}</a></li>
<li>${leaf || t('packets.detail_title')}</li>
</ul>
</div>
<div class="flex items-center justify-between mb-4">
<h1 class="text-2xl font-bold">${t('packets.detail_title')}</h1>
${tzBadge}
</div>
${content}`, container);
}
shell(html`<div class="opacity-60">${t('common.loading')}</div>`);
try {
const [p, channelsData] = await Promise.all([
apiGet(`/api/v1/packets/${id}`, {}, { signal }),
apiGet('/api/v1/channels', { limit: 200 }, { signal }).catch(() => ({ items: [] })),
]);
const channelNames = new Map(
(channelsData.items || [])
.map(c => [parseInt(c.channel_hash, 16), c.name])
.filter(([idx]) => !Number.isNaN(idx))
);
let channelDisplay = html`<span class="opacity-50">—</span>`;
if (p.channel_idx != null) {
const name = channelNames.get(p.channel_idx);
channelDisplay = html`${name ? `${name} (${p.channel_idx})` : `${p.channel_idx}`}`;
}
const observerDisplay = p.observed_by
? html`<a href="/nodes/${p.observed_by}" class="link link-hover">${p.observer_tag_name || p.observer_name || p.observed_by}</a>`
: html`<span class="opacity-50">—</span>`;
const redactedNotice = p.redacted
? html`<div class="alert alert-warning mb-4">\u{1F512} ${t('packets.redacted_notice')}</div>`
: nothing;
const rawBlock = p.redacted
? nothing
: html`
<div class="mt-4">
<div class="flex items-center justify-between mb-1">
<span class="text-xs uppercase opacity-60">${t('packets.col_raw')}</span>
${p.raw_hex ? html`<button class="btn btn-xs btn-ghost" @click=${(e) => copyToClipboard(e, p.raw_hex)}>${t('packets.copy_raw')}</button>` : nothing}
</div>
<pre class="bg-base-200 rounded p-3 text-xs overflow-x-auto whitespace-pre-wrap break-all">${p.raw_hex || '—'}</pre>
</div>`;
const decodedBlock = (!p.redacted && p.decoded)
? html`
<div class="mt-4">
<span class="text-xs uppercase opacity-60">${t('packets.decoded')}</span>
<pre class="bg-base-200 rounded p-3 text-xs overflow-x-auto">${JSON.stringify(p.decoded, null, 2)}</pre>
</div>`
: nothing;
shell(html`
${redactedNotice}
<div class="card bg-base-100 shadow">
<div class="card-body">
<div class="grid grid-cols-1 md:grid-cols-2 gap-x-8">
${field(t('common.time'), formatDateTime(p.received_at))}
${field(t('common.observers'), observerDisplay)}
${field(t('packets.col_event_type'), p.event_type || '—')}
${field(t('entities.channel'), channelDisplay)}
${field(t('packets.col_source'), p.source_pubkey_prefix
? html`<code class="font-mono text-xs">${p.source_pubkey_prefix}</code>`
: html`<span class="opacity-50">—</span>`)}
${field(t('packets.packet_hash'), p.packet_hash
? html`<code class="font-mono text-xs">${p.packet_hash}</code>`
: html`<span class="opacity-50">—</span>`)}
${field(t('packets.packet_type'), p.packet_type != null ? p.packet_type : '—')}
${field(t('packets.payload_type'), p.payload_type != null ? p.payload_type : '—')}
${field(t('packets.col_route_type'), p.route_type || '—')}
${field(t('common.snr_db'), p.snr != null ? Number(p.snr).toFixed(1) : '—')}
${field(t('common.hops'), p.path_len != null ? p.path_len : '—')}
</div>
${rawBlock}
${decodedBlock}
</div>
</div>`, p.packet_hash || p.event_type);
} catch (e) {
if (isAbortError(e)) return;
if (e.status === 404) {
shell(html`<div class="alert alert-error">${t('common.entity_not_found_details', { entity: t('entities.packet').toLowerCase() })}</div>`);
return;
}
shell(warningBadge(e.message));
}
}
@@ -0,0 +1,292 @@
import { apiGet, isAbortError } from '../api.js';
import {
html, litRender, nothing, t,
getConfig, formatDateTime, formatDateTimeShort,
warningBadge,
pagination, sortableTableHeader, mobileSortSelect,
renderFilterCard, autoSubmit, submitOnEnter
} from '../components.js';
import { createAutoRefresh } from '../auto-refresh.js';
const EVENT_TYPES = [
// Structured classifications
'advertisement', 'channel_msg_recv', 'contact_msg_recv',
'trace_data', 'telemetry_response', 'path_updated', 'status_response',
// Per-payload-type classifications for otherwise-unhandled packets
'req', 'response', 'ack', 'encrypted_direct', 'encrypted_channel',
'grp_data', 'anon_req', 'multipart', 'control', 'raw_custom',
'advert', 'path', 'trace', 'letsmesh_packet',
];
function lockBadge() {
return html`<span class="opacity-60" title="${t('packets.redacted_title')}">\u{1F512}</span>`;
}
function channelLabel(packet, channelNames) {
if (packet.channel_idx == null) {
return html`<span class="opacity-50">—</span>`;
}
const name = channelNames.get(packet.channel_idx);
const text = name ? `${name} (${packet.channel_idx})` : `${packet.channel_idx}`;
return html`${text}${packet.redacted ? html` ${lockBadge()}` : nothing}`;
}
function observerCell(packet) {
const name = packet.observer_tag_name || packet.observer_name;
if (!packet.observed_by) {
return html`<span class="opacity-50">—</span>`;
}
const label = name || (packet.observed_by.slice(0, 12) + '…');
return html`<a href="/nodes/${packet.observed_by}" class="link link-hover" @click=${(e) => e.stopPropagation()}>${label}</a>`;
}
export async function render(container, params, router) {
const { signal } = params || {};
const query = params.query || {};
const search = query.search || '';
const packet_hash = query.packet_hash || '';
const event_type = query.event_type || '';
const channel_idx = query.channel_idx || '';
const route_type = query.route_type || 'all';
const observed_by = query.observed_by
? (Array.isArray(query.observed_by) ? query.observed_by : [query.observed_by])
: [];
const min_snr = query.min_snr || '';
const max_snr = query.max_snr || '';
const page = parseInt(query.page, 10) || 1;
const limit = parseInt(query.limit, 10) || 20;
const offset = (page - 1) * limit;
const sort = query.sort || 'time';
const order = query.order || 'desc';
const config = getConfig();
const tz = config.timezone || '';
const tzBadge = tz && tz !== 'UTC' ? html`<span class="text-sm opacity-60">${tz}</span>` : nothing;
const navigate = (url) => router.navigate(url);
let lastContent = nothing;
let lastTotal = null;
function renderPage(content, { total = null, error = null } = {}) {
if (!error) {
lastContent = content;
lastTotal = total;
}
const displayContent = error ? lastContent : content;
const displayTotal = error ? lastTotal : total;
litRender(html`
<div class="flex items-center justify-between mb-4">
<h1 class="text-3xl font-bold">${t('entities.packets')}</h1>
${tzBadge}
</div>
<div class="flex items-center gap-2 mb-4">
${displayTotal !== null
? html`<span class="badge badge-lg">${t('common.total', { count: displayTotal })}</span>`
: nothing}
<span id="auto-refresh-toggle"></span>
${error ? warningBadge(error) : nothing}
</div>
${displayContent}`, container);
}
renderPage(nothing);
async function fetchAndRenderData() {
try {
const apiParams = { limit, offset, search, sort, order };
if (packet_hash) apiParams.packet_hash = packet_hash;
if (event_type) apiParams.event_type = event_type;
if (channel_idx !== '') apiParams.channel_idx = channel_idx;
if (route_type && route_type !== 'all') apiParams.route_type = route_type;
if (observed_by.length > 0) apiParams.observed_by = observed_by;
if (min_snr !== '') apiParams.min_snr = min_snr;
if (max_snr !== '') apiParams.max_snr = max_snr;
const [data, nodesData, channelsData] = await Promise.all([
apiGet('/api/v1/packets', apiParams, { signal }),
apiGet('/api/v1/nodes', { limit: 500, observer: true }, { signal }),
apiGet('/api/v1/channels', { limit: 200 }, { signal }).catch(() => ({ items: [] })),
]);
const packets = data.items || [];
const total = data.total || 0;
const totalPages = Math.ceil(total / limit);
const sortedNodes = (nodesData.items || []).map(n => {
const tagName = n.tags?.find(tg => tg.key === 'name')?.value;
return { ...n, _sortName: (tagName || n.name || '').toLowerCase(), _displayName: tagName || n.name || n.public_key.slice(0, 12) + '...' };
}).sort((a, b) => a._sortName.localeCompare(b._sortName));
const channelList = (channelsData.items || []).map(c => ({
name: c.name,
idx: parseInt(c.channel_hash, 16),
})).filter(c => !Number.isNaN(c.idx));
const channelNames = new Map(channelList.map(c => [c.idx, c.name]));
const noneFound = html`<div class="text-center py-8 opacity-70">${t('common.no_entity_found', { entity: t('entities.packets').toLowerCase() })}</div>`;
const mobileCards = packets.length === 0
? noneFound
: packets.map(p => html`
<a href="/packets/${p.id}" class="card bg-base-100 shadow-sm block">
<div class="card-body p-3">
<div class="flex items-center justify-between gap-2">
<div class="min-w-0">
<div class="font-mono text-sm truncate">${p.event_type || '—'}</div>
<div class="text-xs opacity-60">${channelLabel(p, channelNames)}</div>
</div>
<div class="text-right flex-shrink-0">
<div class="text-xs opacity-60">${formatDateTimeShort(p.received_at)}</div>
<div class="text-xs opacity-60">${observerCell(p)}</div>
</div>
</div>
<div class="flex items-center gap-2 mt-1 text-xs opacity-60">
${p.snr != null ? html`<span>${Number(p.snr).toFixed(1)} dB</span>` : nothing}
${p.path_len != null ? html`<span>${p.path_len} ${t('common.hops').toLowerCase()}</span>` : nothing}
</div>
</div>
</a>`);
const tableRows = packets.length === 0
? html`<tr><td colspan="7" class="text-center py-8 opacity-70">${t('common.no_entity_found', { entity: t('entities.packets').toLowerCase() })}</td></tr>`
: packets.map(p => html`<tr class="hover cursor-pointer" @click=${() => navigate(`/packets/${p.id}`)}>
<td class="text-sm whitespace-nowrap">${formatDateTime(p.received_at)}</td>
<td>${p.packet_hash ? html`<code class="font-mono text-xs">${p.packet_hash}</code>` : html`<span class="opacity-50">—</span>`}</td>
<td class="text-sm">${observerCell(p)}</td>
<td class="font-mono text-xs">${p.event_type || '—'}</td>
<td class="text-sm">${channelLabel(p, channelNames)}</td>
<td class="text-sm">${p.snr != null ? Number(p.snr).toFixed(1) : '—'}</td>
<td class="text-sm">${p.path_len != null ? p.path_len : '—'}</td>
</tr>`);
const paginationBlock = pagination(page, totalPages, '/packets', {
search, packet_hash, event_type, channel_idx, route_type, observed_by,
min_snr, max_snr, limit, sort, order,
});
const filterFields = [
() => html`
<div class="flex flex-col gap-1">
<label class="flex items-center py-1"><span class="opacity-80 text-sm">${t('common.search')}</span></label>
<input type="text" name="search" .value=${search} placeholder="${t('common.search_placeholder')}" class="input input-bordered input-sm w-80" @keydown=${submitOnEnter} />
</div>`,
() => html`
<div class="flex flex-col gap-1 max-w-48">
<label class="flex items-center py-1"><span class="opacity-80 text-sm">${t('packets.filter_event_type')}</span></label>
<select name="event_type" class="select select-bordered select-sm" @change=${autoSubmit}>
<option value="" ?selected=${!event_type}>${t('common.all_types')}</option>
${EVENT_TYPES.map(et => html`<option value=${et} ?selected=${event_type === et}>${et}</option>`)}
</select>
</div>`,
() => html`
<div class="flex flex-col gap-1 max-w-48">
<label class="flex items-center py-1"><span class="opacity-80 text-sm">${t('entities.channel')}</span></label>
<select name="channel_idx" class="select select-bordered select-sm" @change=${autoSubmit}>
<option value="" ?selected=${channel_idx === ''}>${t('common.all_channels')}</option>
${channelList.map(c => html`<option value=${c.idx} ?selected=${String(channel_idx) === String(c.idx)}>${c.name} (${c.idx})</option>`)}
</select>
</div>`,
];
if (sortedNodes.length > 0) {
filterFields.push(() => html`
<div class="flex flex-col gap-1">
<label class="flex items-center py-1"><span class="opacity-80 text-sm">${t('common.filter_observer_label')}</span></label>
<select name="observed_by" multiple size="2" class="select select-bordered select-sm w-full max-w-xs">
${sortedNodes.map(n => html`<option value=${n.public_key} ?selected=${observed_by.includes(n.public_key)}>${n._displayName}</option>`)}
</select>
</div>`);
}
filterFields.push(() => html`
<div class="flex flex-col gap-1 max-w-32">
<label class="flex items-center py-1"><span class="opacity-80 text-sm">${t('packets.filter_min_snr')}</span></label>
<input type="number" step="0.1" name="min_snr" .value=${min_snr} class="input input-bordered input-sm" @keydown=${submitOnEnter} />
</div>`);
filterFields.push(() => html`
<div class="flex flex-col gap-1 max-w-32">
<label class="flex items-center py-1"><span class="opacity-80 text-sm">${t('packets.filter_max_snr')}</span></label>
<input type="number" step="0.1" name="max_snr" .value=${max_snr} class="input input-bordered input-sm" @keydown=${submitOnEnter} />
</div>`);
const hasActiveFilters = search !== '' || event_type !== '' || channel_idx !== '' || observed_by.length > 0 || min_snr !== '' || max_snr !== '';
const existingDetails = container.querySelector('details.collapse');
const isFilterOpen = existingDetails ? existingDetails.open : hasActiveFilters;
const filterCard = renderFilterCard({
fields: filterFields,
basePath: '/packets',
navigate,
collapsible: true,
defaultOpen: isFilterOpen,
});
const headerParams = { search, packet_hash, event_type, channel_idx, route_type, observed_by, min_snr, max_snr, limit };
const sortable = (label, sortKey) => sortableTableHeader(label, {
sortKey, currentSort: sort, currentOrder: order,
navigate, basePath: '/packets', params: headerParams,
});
const packetHashChip = packet_hash
? html`<div class="flex items-center gap-2 mb-3">
<span class="badge badge-neutral inline-flex items-center">
<code class="font-mono text-xs leading-none">${packet_hash}</code>
</span>
<a href="/packets" class="btn btn-xs btn-ghost">${t('common.clear_filters')}</a>
</div>`
: nothing;
renderPage(html`${filterCard}
${packetHashChip}
${mobileSortSelect({
currentSort: sort, currentOrder: order,
navigate, basePath: '/packets',
params: headerParams,
options: [
{ value: 'time:desc', label: t('packets.sort.newest') },
{ value: 'time:asc', label: t('packets.sort.oldest') },
{ value: 'event_type:asc', label: t('packets.sort.event_az') },
{ value: 'snr:desc', label: t('packets.sort.snr_high') },
{ value: 'path_len:desc', label: t('packets.sort.path_high') },
],
})}
<div class="lg:hidden space-y-3">
${mobileCards}
</div>
<div class="hidden lg:block overflow-x-auto overflow-y-visible bg-base-100 rounded-box shadow">
<table class="table table-zebra">
<thead>
<tr>
${sortable(t('common.time'), 'time')}
<th>${t('packets.packet_hash')}</th>
<th>${t('common.observers')}</th>
${sortable(t('packets.col_event_type'), 'event_type')}
<th>${t('entities.channel')}</th>
${sortable(t('common.snr_db'), 'snr')}
${sortable(t('common.hops'), 'path_len')}
</tr>
</thead>
<tbody>
${tableRows}
</tbody>
</table>
</div>
${paginationBlock}`, { total });
} catch (e) {
if (isAbortError(e)) return;
renderPage(nothing, { error: e.message });
}
}
await fetchAndRenderData();
const toggleEl = container.querySelector('#auto-refresh-toggle');
const { cleanup } = createAutoRefresh({
fetchAndRender: fetchAndRenderData,
toggleContainer: toggleEl,
});
return cleanup;
}
+326 -298
View File
@@ -1,301 +1,329 @@
{
"entities": {
"home": "Home",
"dashboard": "Dashboard",
"nodes": "Nodes",
"node": "Node",
"node_detail": "Node Detail",
"advertisements": "Adverts",
"advertisement": "Advert",
"messages": "Messages",
"message": "Message",
"map": "Map",
"members": "Members",
"member": "Member",
"tags": "Tags",
"tag": "Tag",
"channel": "Channel",
"channels": "Channels"
},
"common": {
"filter": "Filter",
"filters": "Filters",
"clear": "Clear",
"clear_filters": "Clear Filters",
"search": "Search",
"cancel": "Cancel",
"delete": "Delete",
"edit": "Edit",
"move": "Move",
"save": "Save",
"save_changes": "Save Changes",
"add": "Add",
"add_entity": "Add {{entity}}",
"add_new_entity": "Add New {{entity}}",
"edit_entity": "Edit {{entity}}",
"delete_entity": "Delete {{entity}}",
"delete_all_entity": "Delete All {{entity}}",
"move_entity": "Move {{entity}}",
"move_entity_to_another_node": "Move {{entity}} to Another Node",
"copy_entity": "Copy {{entity}}",
"copy_all_entity_to_another_node": "Copy All {{entity}} to Another Node",
"view_entity": "View {{entity}}",
"recent_entity": "Recent {{entity}}",
"total_entity": "Total {{entity}}",
"all_entity": "All {{entity}}",
"no_entity_found": "No {{entity}} found",
"no_entity_recorded": "No {{entity}} recorded",
"no_entity_defined": "No {{entity}} defined",
"no_entity_in_database": "No {{entity}} in database",
"no_entity_configured": "No {{entity}} configured",
"no_entity_yet": "No {{entity}} yet",
"entity_not_found_details": "{{entity}} not found: {{details}}",
"page_not_found": "Page not found",
"delete_entity_confirm": "Are you sure you want to delete {{entity}} <strong>{{name}}</strong>?",
"delete_all_entity_confirm": "Are you sure you want to delete all {{count}} {{entity}} from <strong>{{name}}</strong>?",
"cannot_be_undone": "This action cannot be undone.",
"entity_added_success": "{{entity}} added successfully",
"entity_updated_success": "{{entity}} updated successfully",
"entity_deleted_success": "{{entity}} deleted successfully",
"entity_moved_success": "{{entity}} moved successfully",
"all_entity_deleted_success": "All {{entity}} deleted successfully",
"copy_all_entity_description": "Copy all {{count}} {{entity}} from <strong>{{name}}</strong> to another node.",
"previous": "Previous",
"next": "Next",
"go_home": "Go Home",
"loading": "Loading...",
"error": "Error",
"failed_to_load_page": "Failed to load page",
"total": "{{count}} total",
"shown": "{{count}} shown",
"count_entity": "{{count}} {{entity}}",
"type": "Type",
"name": "Name",
"key": "Key",
"value": "Value",
"time": "Time",
"actions": "Actions",
"updated": "Updated",
"sign_in": "Sign In",
"sign_out": "Sign Out",
"view_details": "View Details",
"all_types": "All Types",
"all_channels": "All Channels",
"all_members": "All Members",
"all_operators": "All Operators",
"filter_member_label": "Member",
"filter_operator_label": "Operator",
"filter_observer_label": "Observer",
"node_type": "Node Type",
"show": "Show",
"search_placeholder": "Search by name, ID, or public key...",
"contact": "Contact",
"description": "Description",
"callsign": "Callsign",
"tags": "Tags",
"last_seen": "Last Seen",
"first_seen_label": "First seen:",
"last_seen_label": "Last seen:",
"location": "Location",
"public_key": "Public Key",
"received": "Received",
"received_by": "Received By",
"observers": "Observers",
"snr_db": "SNR (dB)",
"hops": "Hops",
"from": "From",
"close": "close",
"unnamed": "Unnamed",
"unnamed_node": "Unnamed Node",
"validation_invalid_number": "Value must be a valid number",
"validation_invalid_boolean": "Value must be true, false, yes, no, 1, or 0",
"sort_by": "Sort by"
},
"links": {
"website": "Website",
"github": "GitHub",
"discord": "Discord",
"youtube": "YouTube",
"profile": "Profile"
},
"auto_refresh": {
"pause": "Pause auto-refresh",
"resume": "Resume auto-refresh"
},
"time": {
"days_ago": "{{count}}d ago",
"hours_ago": "{{count}}h ago",
"minutes_ago": "{{count}}m ago",
"less_than_minute": "<1m ago",
"last_7_days": "Last 7 days",
"per_day_last_7_days": "Per day (last 7 days)",
"over_time_last_7_days": "Over time (last 7 days)",
"activity_per_day_last_7_days": "Activity per day (last 7 days)"
},
"node_types": {
"chat": "Chat",
"repeater": "Repeater",
"companion": "Companion",
"room": "Room Server",
"unknown": "Unknown"
},
"home": {
"welcome_default": "Welcome to the {{network_name}} mesh network dashboard. Monitor network activity, view connected nodes, and explore message history.",
"all_discovered_nodes": "All discovered nodes",
"network_info": "Network Info",
"network_activity": "Network Activity",
"frequency": "Frequency",
"bandwidth": "Bandwidth",
"spreading_factor": "Spreading Factor",
"coding_rate": "Coding Rate",
"tx_power": "TX Power"
},
"dashboard": {
"all_discovered_nodes": "All discovered nodes",
"recent_channel_messages": "Recent Channel Messages",
"channel": "Channel {{number}}"
},
"nodes": {
"scan_to_add": "Scan to add as contact",
"ownership": "Ownership",
"adopt": "Adopt",
"release": "Release",
"adopted_by": "Adopted by {{name}}",
"adopted_by_prefix": "Adopted by",
"not_adopted": "This node has not been adopted by any operator.",
"adopt_success": "Node adopted successfully",
"release_success": "Node released successfully",
"release_confirm": "Are you sure you want to release this node?",
"sort": {
"last_seen_newest": "Last Seen (newest)",
"last_seen_oldest": "Last Seen (oldest)",
"name_az": "Name (A\u2013Z)",
"name_za": "Name (Z\u2013A)",
"key_asc": "Public Key (ascending)",
"key_desc": "Public Key (descending)"
}
},
"advertisements": {
"filter_route_type_label": "Advert Type",
"route_type_all": "All",
"route_type_flood": "Flood & Relay",
"route_type_direct": "Zero-hop only",
"route_type_unknown": "Unknown",
"col_route_type": "Type",
"sort": {
"newest": "Time (newest)",
"oldest": "Time (oldest)",
"node_az": "Node (A\u2013Z)",
"node_za": "Node (Z\u2013A)",
"key_asc": "Public Key (ascending)",
"key_desc": "Public Key (descending)"
}
},
"messages": {
"type_direct": "Direct",
"type_channel": "Channel",
"type_contact": "Contact",
"type_public": "Public",
"sort": {
"newest": "Time (newest)",
"oldest": "Time (oldest)",
"type_az": "Type (A\u2013Z)",
"type_za": "Type (Z\u2013A)",
"from_az": "From (A\u2013Z)",
"from_za": "From (Z\u2013A)",
"message_az": "Message (A\u2013Z)",
"message_za": "Message (Z\u2013A)"
}
},
"map": {
"show_labels": "Show Labels",
"infrastructure_only": "Infrastructure Only",
"legend": "Legend:",
"infrastructure": "Infrastructure",
"public": "Public",
"nodes_on_map": "{{count}} nodes on map",
"nodes_none_have_coordinates": "{{count}} nodes (none have coordinates)",
"gps_description": "Nodes are placed on the map based on GPS coordinates from node reports or manual tags.",
"owner": "Owner:",
"role": "Role:",
"select_destination_node": "-- Select destination node --"
},
"members": {
"empty_state_description": "No members yet.",
"empty_description": "Members will appear here once users log in and adopt nodes."
},
"channels": {
"title": "Channels",
"add_channel": "Add Channel",
"edit_channel": "Edit Channel",
"delete_channel": "Delete Channel",
"delete_confirm": "Are you sure you want to delete channel {{name}}?",
"name_label": "Channel Name",
"key_label": "Channel Key (hex)",
"visibility_label": "Visibility",
"visibility_community": "Community",
"visibility_member": "Member",
"visibility_operator": "Operator",
"visibility_admin": "Admin",
"enabled_label": "Enabled",
"channel_hash_label": "Hash",
"disabled": "Disabled",
"optgroup_standard": "Standard",
"optgroup_custom": "Custom"
},
"not_found": {
"description": "The page you're looking for doesn't exist or has been moved."
},
"custom_page": {
"failed_to_load": "Failed to load page"
},
"auth": {
"login": "Login",
"logout": "Logout",
"login_required": "Login required",
"admin_required": "Admin access required",
"login_hint": "Log in to access admin features",
"logged_in_as": "Logged in as {{name}}",
"session_expired": "Session expired, please log in again",
"role_admin": "admin",
"role_member": "member"
},
"user_profile": {
"title": "Profile",
"your_profile": "Your Profile",
"profile_updated": "Profile updated successfully",
"save_profile": "Save Profile",
"name_label": "Display Name",
"name_placeholder": "Your name or preferred name",
"callsign_label": "Callsign",
"callsign_placeholder": "Amateur radio callsign (e.g., W1ABC)",
"description_label": "Description",
"description_placeholder": "A short bio or description",
"url_label": "Website / Profile Link",
"url_placeholder": "https://...",
"adopted_nodes": "Adopted Nodes",
"no_adopted_nodes": "No adopted nodes",
"login_to_view": "Log in to view your profile",
"view_profile": "View Profile",
"edit_profile": "Edit Profile",
"role_operator": "Operator",
"role_member": "Member",
"member_since": "Member since {{date}}"
},
"members_page": {
"operators": "Operators",
"members": "Members",
"node_count": "{{count}} nodes",
"empty_state": "No members yet",
"empty_description": "Members will appear here once users log in and adopt nodes."
},
"footer": {
"powered_by": "Powered by"
},
"errors": {
"go_home": "Go Home",
"not_found": "Page not found",
"server_error": "Internal server error",
"unexpected": "An unexpected error occurred"
"entities": {
"home": "Home",
"dashboard": "Dashboard",
"nodes": "Nodes",
"node": "Node",
"node_detail": "Node Detail",
"advertisements": "Adverts",
"advertisement": "Advert",
"messages": "Messages",
"packets": "Packets",
"message": "Message",
"packet": "Packet",
"map": "Map",
"members": "Members",
"member": "Member",
"tags": "Tags",
"tag": "Tag",
"channel": "Channel",
"channels": "Channels"
},
"common": {
"filter": "Filter",
"filters": "Filters",
"clear": "Clear",
"clear_filters": "Clear Filters",
"search": "Search",
"cancel": "Cancel",
"delete": "Delete",
"edit": "Edit",
"move": "Move",
"save": "Save",
"save_changes": "Save Changes",
"add": "Add",
"add_entity": "Add {{entity}}",
"add_new_entity": "Add New {{entity}}",
"edit_entity": "Edit {{entity}}",
"delete_entity": "Delete {{entity}}",
"delete_all_entity": "Delete All {{entity}}",
"move_entity": "Move {{entity}}",
"move_entity_to_another_node": "Move {{entity}} to Another Node",
"copy_entity": "Copy {{entity}}",
"copy_all_entity_to_another_node": "Copy All {{entity}} to Another Node",
"view_entity": "View {{entity}}",
"recent_entity": "Recent {{entity}}",
"total_entity": "Total {{entity}}",
"all_entity": "All {{entity}}",
"no_entity_found": "No {{entity}} found",
"no_entity_recorded": "No {{entity}} recorded",
"no_entity_defined": "No {{entity}} defined",
"no_entity_in_database": "No {{entity}} in database",
"no_entity_configured": "No {{entity}} configured",
"no_entity_yet": "No {{entity}} yet",
"entity_not_found_details": "{{entity}} not found: {{details}}",
"page_not_found": "Page not found",
"delete_entity_confirm": "Are you sure you want to delete {{entity}} <strong>{{name}}</strong>?",
"delete_all_entity_confirm": "Are you sure you want to delete all {{count}} {{entity}} from <strong>{{name}}</strong>?",
"cannot_be_undone": "This action cannot be undone.",
"entity_added_success": "{{entity}} added successfully",
"entity_updated_success": "{{entity}} updated successfully",
"entity_deleted_success": "{{entity}} deleted successfully",
"entity_moved_success": "{{entity}} moved successfully",
"all_entity_deleted_success": "All {{entity}} deleted successfully",
"copy_all_entity_description": "Copy all {{count}} {{entity}} from <strong>{{name}}</strong> to another node.",
"previous": "Previous",
"next": "Next",
"go_home": "Go Home",
"loading": "Loading...",
"error": "Error",
"failed_to_load_page": "Failed to load page",
"total": "{{count}} total",
"shown": "{{count}} shown",
"count_entity": "{{count}} {{entity}}",
"type": "Type",
"name": "Name",
"key": "Key",
"value": "Value",
"time": "Time",
"actions": "Actions",
"updated": "Updated",
"sign_in": "Sign In",
"sign_out": "Sign Out",
"view_details": "View Details",
"all_types": "All Types",
"all_channels": "All Channels",
"all_members": "All Members",
"all_operators": "All Operators",
"filter_member_label": "Member",
"filter_operator_label": "Operator",
"filter_observer_label": "Observer",
"node_type": "Node Type",
"show": "Show",
"search_placeholder": "Search by name, ID, or public key...",
"contact": "Contact",
"description": "Description",
"callsign": "Callsign",
"tags": "Tags",
"last_seen": "Last Seen",
"first_seen_label": "First seen:",
"last_seen_label": "Last seen:",
"location": "Location",
"public_key": "Public Key",
"received": "Received",
"received_by": "Received By",
"observers": "Observers",
"snr_db": "SNR (dB)",
"hops": "Hops",
"from": "From",
"close": "close",
"unnamed": "Unnamed",
"unnamed_node": "Unnamed Node",
"validation_invalid_number": "Value must be a valid number",
"validation_invalid_boolean": "Value must be true, false, yes, no, 1, or 0",
"sort_by": "Sort by"
},
"links": {
"website": "Website",
"github": "GitHub",
"discord": "Discord",
"youtube": "YouTube",
"profile": "Profile"
},
"auto_refresh": {
"pause": "Pause auto-refresh",
"resume": "Resume auto-refresh"
},
"time": {
"days_ago": "{{count}}d ago",
"hours_ago": "{{count}}h ago",
"minutes_ago": "{{count}}m ago",
"less_than_minute": "<1m ago",
"last_7_days": "Last 7 days",
"per_day_last_7_days": "Per day (last 7 days)",
"over_time_last_7_days": "Over time (last 7 days)",
"activity_per_day_last_7_days": "Activity per day (last 7 days)"
},
"node_types": {
"chat": "Chat",
"repeater": "Repeater",
"companion": "Companion",
"room": "Room Server",
"unknown": "Unknown"
},
"home": {
"welcome_default": "Welcome to the {{network_name}} mesh network dashboard. Monitor network activity, view connected nodes, and explore message history.",
"all_discovered_nodes": "All discovered nodes",
"network_info": "Network Info",
"network_activity": "Network Activity",
"frequency": "Frequency",
"bandwidth": "Bandwidth",
"spreading_factor": "Spreading Factor",
"coding_rate": "Coding Rate",
"tx_power": "TX Power"
},
"dashboard": {
"all_discovered_nodes": "All discovered nodes",
"recent_channel_messages": "Recent Channel Messages",
"channel": "Channel {{number}}"
},
"nodes": {
"scan_to_add": "Scan to add as contact",
"ownership": "Ownership",
"adopt": "Adopt",
"release": "Release",
"adopted_by": "Adopted by {{name}}",
"adopted_by_prefix": "Adopted by",
"not_adopted": "This node has not been adopted by any operator.",
"adopt_success": "Node adopted successfully",
"release_success": "Node released successfully",
"release_confirm": "Are you sure you want to release this node?",
"sort": {
"last_seen_newest": "Last Seen (newest)",
"last_seen_oldest": "Last Seen (oldest)",
"name_az": "Name (AZ)",
"name_za": "Name (ZA)",
"key_asc": "Public Key (ascending)",
"key_desc": "Public Key (descending)"
}
},
"advertisements": {
"filter_route_type_label": "Advert Type",
"route_type_all": "All",
"route_type_flood": "Flood & Relay",
"route_type_direct": "Zero-hop only",
"route_type_unknown": "Unknown",
"col_route_type": "Type",
"sort": {
"newest": "Time (newest)",
"oldest": "Time (oldest)",
"node_az": "Node (AZ)",
"node_za": "Node (ZA)",
"key_asc": "Public Key (ascending)",
"key_desc": "Public Key (descending)"
}
},
"messages": {
"type_direct": "Direct",
"type_channel": "Channel",
"type_contact": "Contact",
"type_public": "Public",
"sort": {
"newest": "Time (newest)",
"oldest": "Time (oldest)",
"type_az": "Type (AZ)",
"type_za": "Type (ZA)",
"from_az": "From (AZ)",
"from_za": "From (ZA)",
"message_az": "Message (AZ)",
"message_za": "Message (ZA)"
}
},
"packets": {
"filter_event_type": "Event Type",
"filter_min_snr": "Min SNR",
"filter_max_snr": "Max SNR",
"redacted": "Restricted",
"redacted_title": "Hidden by channel visibility",
"redacted_notice": "This packet was observed on a channel above your access level. The payload is hidden.",
"detail_title": "Packet Detail",
"decoded": "Decoded",
"copy_raw": "Copy raw hex",
"view_raw": "View raw packets",
"packet_hash": "Packet Hash",
"packet_type": "Packet Type",
"payload_type": "Payload Type",
"col_event_type": "Event",
"col_source": "Source",
"col_route_type": "Route Type",
"col_raw": "Raw",
"sort": {
"newest": "Time (newest)",
"oldest": "Time (oldest)",
"event_az": "Event (AZ)",
"snr_high": "SNR (highest)",
"path_high": "Hops (most)"
}
},
"map": {
"show_labels": "Show Labels",
"infrastructure_only": "Infrastructure Only",
"legend": "Legend:",
"infrastructure": "Infrastructure",
"public": "Public",
"nodes_on_map": "{{count}} nodes on map",
"nodes_none_have_coordinates": "{{count}} nodes (none have coordinates)",
"gps_description": "Nodes are placed on the map based on GPS coordinates from node reports or manual tags.",
"owner": "Owner:",
"role": "Role:",
"select_destination_node": "-- Select destination node --"
},
"members": {
"empty_state_description": "No members yet.",
"empty_description": "Members will appear here once users log in and adopt nodes."
},
"channels": {
"title": "Channels",
"add_channel": "Add Channel",
"edit_channel": "Edit Channel",
"delete_channel": "Delete Channel",
"delete_confirm": "Are you sure you want to delete channel {{name}}?",
"name_label": "Channel Name",
"key_label": "Channel Key (hex)",
"visibility_label": "Visibility",
"visibility_community": "Community",
"visibility_member": "Member",
"visibility_operator": "Operator",
"visibility_admin": "Admin",
"enabled_label": "Enabled",
"channel_hash_label": "Hash",
"disabled": "Disabled",
"optgroup_standard": "Standard",
"optgroup_custom": "Custom"
},
"not_found": {
"description": "The page you're looking for doesn't exist or has been moved."
},
"custom_page": {
"failed_to_load": "Failed to load page"
},
"auth": {
"login": "Login",
"logout": "Logout",
"login_required": "Login required",
"admin_required": "Admin access required",
"login_hint": "Log in to access admin features",
"logged_in_as": "Logged in as {{name}}",
"session_expired": "Session expired, please log in again",
"role_admin": "admin",
"role_member": "member"
},
"user_profile": {
"title": "Profile",
"your_profile": "Your Profile",
"profile_updated": "Profile updated successfully",
"save_profile": "Save Profile",
"name_label": "Display Name",
"name_placeholder": "Your name or preferred name",
"callsign_label": "Callsign",
"callsign_placeholder": "Amateur radio callsign (e.g., W1ABC)",
"description_label": "Description",
"description_placeholder": "A short bio or description",
"url_label": "Website / Profile Link",
"url_placeholder": "https://...",
"adopted_nodes": "Adopted Nodes",
"no_adopted_nodes": "No adopted nodes",
"login_to_view": "Log in to view your profile",
"view_profile": "View Profile",
"edit_profile": "Edit Profile",
"role_operator": "Operator",
"role_member": "Member",
"member_since": "Member since {{date}}"
},
"members_page": {
"operators": "Operators",
"members": "Members",
"node_count": "{{count}} nodes",
"empty_state": "No members yet",
"empty_description": "Members will appear here once users log in and adopt nodes."
},
"footer": {
"powered_by": "Powered by"
},
"errors": {
"go_home": "Go Home",
"not_found": "Page not found",
"server_error": "Internal server error",
"unexpected": "An unexpected error occurred"
}
}
+244 -216
View File
@@ -1,219 +1,247 @@
{
"entities": {
"home": "Startpagina",
"dashboard": "Dashboard",
"nodes": "Knooppunten",
"node": "Knooppunt",
"node_detail": "Knooppuntdetails",
"advertisements": "Advertenties",
"advertisement": "Advertentie",
"messages": "Berichten",
"message": "Bericht",
"map": "Kaart",
"members": "Leden",
"member": "Lid",
"admin": "Beheer",
"tags": "Labels",
"tag": "Label"
},
"common": {
"filter": "Filter",
"clear": "Wissen",
"clear_filters": "Filters wissen",
"search": "Zoeken",
"cancel": "Annuleren",
"delete": "Verwijderen",
"edit": "Bewerken",
"move": "Verplaatsen",
"save": "Opslaan",
"save_changes": "Wijzigingen opslaan",
"add": "Toevoegen",
"add_entity": "{{entity}} toevoegen",
"add_new_entity": "Nieuwe {{entity}} toevoegen",
"edit_entity": "{{entity}} bewerken",
"delete_entity": "{{entity}} verwijderen",
"delete_all_entity": "Alle {{entity}} verwijderen",
"move_entity": "{{entity}} verplaatsen",
"move_entity_to_another_node": "{{entity}} naar ander knooppunt verplaatsen",
"copy_entity": "{{entity}} kopiëren",
"copy_all_entity_to_another_node": "Alle {{entity}} naar ander knooppunt kopiëren",
"view_entity": "{{entity}} bekijken",
"recent_entity": "Recente {{entity}}",
"total_entity": "Totaal {{entity}}",
"all_entity": "Alle {{entity}}",
"no_entity_found": "Geen {{entity}} gevonden",
"no_entity_recorded": "Geen {{entity}} geregistreerd",
"no_entity_defined": "Geen {{entity}} gedefinieerd",
"no_entity_in_database": "Geen {{entity}} in database",
"no_entity_configured": "Geen {{entity}} geconfigureerd",
"no_entity_yet": "Nog geen {{entity}}",
"entity_not_found_details": "{{entity}} niet gevonden: {{details}}",
"page_not_found": "Pagina niet gevonden",
"delete_entity_confirm": "Weet u zeker dat u {{entity}} <strong>{{name}}</strong> wilt verwijderen?",
"delete_all_entity_confirm": "Weet u zeker dat u alle {{count}} {{entity}} van <strong>{{name}}</strong> wilt verwijderen?",
"cannot_be_undone": "Deze actie kan niet ongedaan worden gemaakt.",
"entity_added_success": "{{entity}} succesvol toegevoegd",
"entity_updated_success": "{{entity}} succesvol bijgewerkt",
"entity_deleted_success": "{{entity}} succesvol verwijderd",
"entity_moved_success": "{{entity}} succesvol verplaatst",
"all_entity_deleted_success": "Alle {{entity}} succesvol verwijderd",
"copy_all_entity_description": "Kopieer alle {{count}} {{entity}} van <strong>{{name}}</strong> naar een ander knooppunt.",
"previous": "Vorige",
"next": "Volgende",
"go_home": "Naar startpagina",
"loading": "Laden...",
"error": "Fout",
"failed_to_load_page": "Pagina laden mislukt",
"total": "{{count}} totaal",
"shown": "{{count}} weergegeven",
"count_entity": "{{count}} {{entity}}",
"type": "Type",
"name": "Naam",
"key": "Sleutel",
"value": "Waarde",
"time": "Tijd",
"actions": "Acties",
"updated": "Bijgewerkt",
"sign_in": "Inloggen",
"sign_out": "Uitloggen",
"view_details": "Details bekijken",
"all_types": "Alle types",
"node_type": "Knooppunttype",
"show": "Toon",
"search_placeholder": "Zoek op naam, ID of publieke sleutel...",
"contact": "Contact",
"description": "Beschrijving",
"callsign": "Roepnaam",
"tags": "Labels",
"last_seen": "Laatst gezien",
"first_seen_label": "Eerst gezien:",
"last_seen_label": "Laatst gezien:",
"location": "Locatie",
"public_key": "Publieke sleutel",
"received": "Ontvangen",
"received_by": "Ontvangen door",
"receivers": "Ontvangers",
"from": "Van",
"close": "sluiten",
"unnamed": "Naamloos",
"unnamed_node": "Naamloos knooppunt",
"all_operators": "Alle Operators",
"filter_operator_label": "Operator"
},
"links": {
"website": "Website",
"github": "GitHub",
"discord": "Discord",
"youtube": "YouTube",
"profile": "Profiel"
},
"auto_refresh": {
"pause": "Pauzeer verversen",
"resume": "Hervat verversen"
},
"time": {
"days_ago": "{{count}}d geleden",
"hours_ago": "{{count}}u geleden",
"minutes_ago": "{{count}}m geleden",
"less_than_minute": "<1m geleden",
"last_7_days": "Laatste 7 dagen",
"per_day_last_7_days": "Per dag (laatste 7 dagen)",
"over_time_last_7_days": "In de tijd (laatste 7 dagen)",
"activity_per_day_last_7_days": "Activiteit per dag (laatste 7 dagen)"
},
"node_types": {
"chat": "Chat",
"repeater": "Repeater",
"room": "Ruimte",
"unknown": "Onbekend"
},
"home": {
"welcome_default": "Welkom bij het {{network_name}} mesh-netwerk dashboard. Monitor netwerkactiviteit, bekijk verbonden knooppunten en verken berichtgeschiedenis.",
"all_discovered_nodes": "Alle ontdekte knooppunten",
"network_info": "Netwerkinfo",
"network_activity": "Netwerkactiviteit",
"frequency": "Frequentie",
"bandwidth": "Bandbreedte",
"spreading_factor": "Spreading Factor",
"coding_rate": "Coderingssnelheid",
"tx_power": "TX Vermogen"
},
"dashboard": {
"all_discovered_nodes": "Alle ontdekte knooppunten",
"recent_channel_messages": "Recente kanaalberichten",
"channel": "Kanaal {{number}}"
},
"nodes": {
"scan_to_add": "Scan om als contact toe te voegen"
},
"advertisements": {},
"messages": {
"type_direct": "Direct",
"type_channel": "Kanaal",
"type_contact": "Contact",
"type_public": "Publiek"
},
"map": {
"show_labels": "Toon labels",
"infrastructure_only": "Alleen infrastructuur",
"legend": "Legenda:",
"infrastructure": "Infrastructuur",
"public": "Publiek",
"nodes_on_map": "{{count}} knooppunten op kaart",
"nodes_none_have_coordinates": "{{count}} knooppunten (geen met coördinaten)",
"gps_description": "Knooppunten worden op de kaart geplaatst op basis van GPS-coördinaten uit knooppuntrapporten of handmatige labels.",
"owner": "Eigenaar:",
"role": "Rol:",
"select_destination_node": "-- Selecteer bestemmingsknooppunt --"
},
"members": {
"empty_state_description": "Nog geen leden.",
"empty_description": "Leden verschijnen hier zodra gebruikers inloggen en knooppunten adopteren."
},
"channels": {
"visibility_community": "Community",
"visibility_member": "Lid",
"visibility_operator": "Operator",
"visibility_admin": "Beheerder"
},
"not_found": {
"description": "De pagina die u zoekt bestaat niet of is verplaatst."
},
"custom_page": {
"failed_to_load": "Pagina laden mislukt"
},
"admin": {
"access_denied": "Toegang geweigerd",
"admin_not_enabled": "De beheerinterface is niet ingeschakeld.",
"admin_enable_hint": "Stel <code>WEB_ADMIN_ENABLED=true</code> in om beheerfuncties in te schakelen.",
"welcome": "Welkom bij het beheerpaneel.",
"members_description": "Beheer netwerkleden en operators.",
"tags_description": "Beheer aangepaste labels en metadata voor netwerkknooppunten."
},
"admin_members": {
"network_members": "Netwerkleden ({{count}})",
"member_id": "Lid-ID",
"member_id_hint": "Unieke identificatie (letters, cijfers, underscore)",
"empty_state_hint": "Klik op \"Lid toevoegen\" om het eerste lid aan te maken."
},
"admin_node_tags": {
"select_node": "Selecteer knooppunt",
"select_node_placeholder": "-- Selecteer een knooppunt --",
"load_tags": "Labels laden",
"move_warning": "Dit verplaatst het label van het huidige knooppunt naar het bestemmingsknooppunt.",
"copy_all": "Alles kopiëren",
"copy_all_info": "Labels die al bestaan op het bestemmingsknooppunt worden overgeslagen. Originele labels blijven op dit knooppunt.",
"delete_all": "Alles verwijderen",
"delete_all_warning": "Alle labels worden permanent verwijderd.",
"destination_node": "Bestemmingsknooppunt",
"tag_key": "Label sleutel",
"for_this_node": "voor dit knooppunt",
"empty_state_hint": "Voeg hieronder een nieuw label toe.",
"select_a_node": "Selecteer een knooppunt",
"select_a_node_description": "Kies een knooppunt uit de vervolgkeuzelijst hierboven om de labels te bekijken en beheren.",
"copied_entities": "{{copied}} label(s) gekopieerd, {{skipped}} overgeslagen"
},
"footer": {
"powered_by": "Mogelijk gemaakt door"
"entities": {
"home": "Startpagina",
"dashboard": "Dashboard",
"nodes": "Knooppunten",
"node": "Knooppunt",
"node_detail": "Knooppuntdetails",
"advertisements": "Advertenties",
"advertisement": "Advertentie",
"messages": "Berichten",
"packets": "Pakketten",
"message": "Bericht",
"packet": "Pakket",
"map": "Kaart",
"members": "Leden",
"member": "Lid",
"admin": "Beheer",
"tags": "Labels",
"tag": "Label"
},
"common": {
"filter": "Filter",
"clear": "Wissen",
"clear_filters": "Filters wissen",
"search": "Zoeken",
"cancel": "Annuleren",
"delete": "Verwijderen",
"edit": "Bewerken",
"move": "Verplaatsen",
"save": "Opslaan",
"save_changes": "Wijzigingen opslaan",
"add": "Toevoegen",
"add_entity": "{{entity}} toevoegen",
"add_new_entity": "Nieuwe {{entity}} toevoegen",
"edit_entity": "{{entity}} bewerken",
"delete_entity": "{{entity}} verwijderen",
"delete_all_entity": "Alle {{entity}} verwijderen",
"move_entity": "{{entity}} verplaatsen",
"move_entity_to_another_node": "{{entity}} naar ander knooppunt verplaatsen",
"copy_entity": "{{entity}} kopiëren",
"copy_all_entity_to_another_node": "Alle {{entity}} naar ander knooppunt kopiëren",
"view_entity": "{{entity}} bekijken",
"recent_entity": "Recente {{entity}}",
"total_entity": "Totaal {{entity}}",
"all_entity": "Alle {{entity}}",
"no_entity_found": "Geen {{entity}} gevonden",
"no_entity_recorded": "Geen {{entity}} geregistreerd",
"no_entity_defined": "Geen {{entity}} gedefinieerd",
"no_entity_in_database": "Geen {{entity}} in database",
"no_entity_configured": "Geen {{entity}} geconfigureerd",
"no_entity_yet": "Nog geen {{entity}}",
"entity_not_found_details": "{{entity}} niet gevonden: {{details}}",
"page_not_found": "Pagina niet gevonden",
"delete_entity_confirm": "Weet u zeker dat u {{entity}} <strong>{{name}}</strong> wilt verwijderen?",
"delete_all_entity_confirm": "Weet u zeker dat u alle {{count}} {{entity}} van <strong>{{name}}</strong> wilt verwijderen?",
"cannot_be_undone": "Deze actie kan niet ongedaan worden gemaakt.",
"entity_added_success": "{{entity}} succesvol toegevoegd",
"entity_updated_success": "{{entity}} succesvol bijgewerkt",
"entity_deleted_success": "{{entity}} succesvol verwijderd",
"entity_moved_success": "{{entity}} succesvol verplaatst",
"all_entity_deleted_success": "Alle {{entity}} succesvol verwijderd",
"copy_all_entity_description": "Kopieer alle {{count}} {{entity}} van <strong>{{name}}</strong> naar een ander knooppunt.",
"previous": "Vorige",
"next": "Volgende",
"go_home": "Naar startpagina",
"loading": "Laden...",
"error": "Fout",
"failed_to_load_page": "Pagina laden mislukt",
"total": "{{count}} totaal",
"shown": "{{count}} weergegeven",
"count_entity": "{{count}} {{entity}}",
"type": "Type",
"name": "Naam",
"key": "Sleutel",
"value": "Waarde",
"time": "Tijd",
"actions": "Acties",
"updated": "Bijgewerkt",
"sign_in": "Inloggen",
"sign_out": "Uitloggen",
"view_details": "Details bekijken",
"all_types": "Alle types",
"node_type": "Knooppunttype",
"show": "Toon",
"search_placeholder": "Zoek op naam, ID of publieke sleutel...",
"contact": "Contact",
"description": "Beschrijving",
"callsign": "Roepnaam",
"tags": "Labels",
"last_seen": "Laatst gezien",
"first_seen_label": "Eerst gezien:",
"last_seen_label": "Laatst gezien:",
"location": "Locatie",
"public_key": "Publieke sleutel",
"received": "Ontvangen",
"received_by": "Ontvangen door",
"receivers": "Ontvangers",
"from": "Van",
"close": "sluiten",
"unnamed": "Naamloos",
"unnamed_node": "Naamloos knooppunt",
"all_operators": "Alle Operators",
"filter_operator_label": "Operator"
},
"links": {
"website": "Website",
"github": "GitHub",
"discord": "Discord",
"youtube": "YouTube",
"profile": "Profiel"
},
"auto_refresh": {
"pause": "Pauzeer verversen",
"resume": "Hervat verversen"
},
"time": {
"days_ago": "{{count}}d geleden",
"hours_ago": "{{count}}u geleden",
"minutes_ago": "{{count}}m geleden",
"less_than_minute": "<1m geleden",
"last_7_days": "Laatste 7 dagen",
"per_day_last_7_days": "Per dag (laatste 7 dagen)",
"over_time_last_7_days": "In de tijd (laatste 7 dagen)",
"activity_per_day_last_7_days": "Activiteit per dag (laatste 7 dagen)"
},
"node_types": {
"chat": "Chat",
"repeater": "Repeater",
"room": "Ruimte",
"unknown": "Onbekend"
},
"home": {
"welcome_default": "Welkom bij het {{network_name}} mesh-netwerk dashboard. Monitor netwerkactiviteit, bekijk verbonden knooppunten en verken berichtgeschiedenis.",
"all_discovered_nodes": "Alle ontdekte knooppunten",
"network_info": "Netwerkinfo",
"network_activity": "Netwerkactiviteit",
"frequency": "Frequentie",
"bandwidth": "Bandbreedte",
"spreading_factor": "Spreading Factor",
"coding_rate": "Coderingssnelheid",
"tx_power": "TX Vermogen"
},
"dashboard": {
"all_discovered_nodes": "Alle ontdekte knooppunten",
"recent_channel_messages": "Recente kanaalberichten",
"channel": "Kanaal {{number}}"
},
"nodes": {
"scan_to_add": "Scan om als contact toe te voegen"
},
"advertisements": {},
"messages": {
"type_direct": "Direct",
"type_channel": "Kanaal",
"type_contact": "Contact",
"type_public": "Publiek"
},
"packets": {
"filter_event_type": "Gebeurtenistype",
"filter_min_snr": "Min. SNR",
"filter_max_snr": "Max. SNR",
"redacted": "Beperkt",
"redacted_title": "Verborgen door kanaalzichtbaarheid",
"redacted_notice": "Dit pakket is waargenomen op een kanaal boven uw toegangsniveau. De inhoud is verborgen.",
"detail_title": "Pakketdetail",
"decoded": "Gedecodeerd",
"copy_raw": "Ruwe hex kopiëren",
"view_raw": "Ruwe pakketten bekijken",
"packet_hash": "Pakkethash",
"packet_type": "Pakkettype",
"payload_type": "Payloadtype",
"col_event_type": "Gebeurtenis",
"col_source": "Bron",
"col_route_type": "Routetype",
"col_raw": "Ruw",
"sort": {
"newest": "Tijd (nieuwste)",
"oldest": "Tijd (oudste)",
"event_az": "Gebeurtenis (AZ)",
"snr_high": "SNR (hoogste)",
"path_high": "Hops (meeste)"
}
},
"map": {
"show_labels": "Toon labels",
"infrastructure_only": "Alleen infrastructuur",
"legend": "Legenda:",
"infrastructure": "Infrastructuur",
"public": "Publiek",
"nodes_on_map": "{{count}} knooppunten op kaart",
"nodes_none_have_coordinates": "{{count}} knooppunten (geen met coördinaten)",
"gps_description": "Knooppunten worden op de kaart geplaatst op basis van GPS-coördinaten uit knooppuntrapporten of handmatige labels.",
"owner": "Eigenaar:",
"role": "Rol:",
"select_destination_node": "-- Selecteer bestemmingsknooppunt --"
},
"members": {
"empty_state_description": "Nog geen leden.",
"empty_description": "Leden verschijnen hier zodra gebruikers inloggen en knooppunten adopteren."
},
"channels": {
"visibility_community": "Community",
"visibility_member": "Lid",
"visibility_operator": "Operator",
"visibility_admin": "Beheerder"
},
"not_found": {
"description": "De pagina die u zoekt bestaat niet of is verplaatst."
},
"custom_page": {
"failed_to_load": "Pagina laden mislukt"
},
"admin": {
"access_denied": "Toegang geweigerd",
"admin_not_enabled": "De beheerinterface is niet ingeschakeld.",
"admin_enable_hint": "Stel <code>WEB_ADMIN_ENABLED=true</code> in om beheerfuncties in te schakelen.",
"welcome": "Welkom bij het beheerpaneel.",
"members_description": "Beheer netwerkleden en operators.",
"tags_description": "Beheer aangepaste labels en metadata voor netwerkknooppunten."
},
"admin_members": {
"network_members": "Netwerkleden ({{count}})",
"member_id": "Lid-ID",
"member_id_hint": "Unieke identificatie (letters, cijfers, underscore)",
"empty_state_hint": "Klik op \"Lid toevoegen\" om het eerste lid aan te maken."
},
"admin_node_tags": {
"select_node": "Selecteer knooppunt",
"select_node_placeholder": "-- Selecteer een knooppunt --",
"load_tags": "Labels laden",
"move_warning": "Dit verplaatst het label van het huidige knooppunt naar het bestemmingsknooppunt.",
"copy_all": "Alles kopiëren",
"copy_all_info": "Labels die al bestaan op het bestemmingsknooppunt worden overgeslagen. Originele labels blijven op dit knooppunt.",
"delete_all": "Alles verwijderen",
"delete_all_warning": "Alle labels worden permanent verwijderd.",
"destination_node": "Bestemmingsknooppunt",
"tag_key": "Label sleutel",
"for_this_node": "voor dit knooppunt",
"empty_state_hint": "Voeg hieronder een nieuw label toe.",
"select_a_node": "Selecteer een knooppunt",
"select_a_node_description": "Kies een knooppunt uit de vervolgkeuzelijst hierboven om de labels te bekijken en beheren.",
"copied_entities": "{{copied}} label(s) gekopieerd, {{skipped}} overgeslagen"
},
"footer": {
"powered_by": "Mogelijk gemaakt door"
}
}
+3
View File
@@ -73,6 +73,9 @@
{% if features.messages %}
<li><a href="/messages" data-nav-link><svg xmlns="http://www.w3.org/2000/svg" class="h-4 w-4 nav-icon-messages" fill="none" viewBox="0 0 24 24" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 10h.01M12 10h.01M16 10h.01M9 16H5a2 2 0 01-2-2V6a2 2 0 012-2h14a2 2 0 012 2v8a2 2 0 01-2 2h-5l-5 5v-5z" /></svg> {{ t('entities.messages') }}</a></li>
{% endif %}
{% if features.packets %}
<li><a href="/packets" data-nav-link><svg xmlns="http://www.w3.org/2000/svg" class="h-4 w-4 nav-icon-packets" fill="none" viewBox="0 0 24 24" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M20 7l-8-4-8 4m16 0l-8 4m8-4v10l-8 4m0-10L4 7m8 4v10M4 7v10l8 4" /></svg> {{ t('entities.packets') }}</a></li>
{% endif %}
{% if features.map %}
<li><a href="/map" data-nav-link><svg xmlns="http://www.w3.org/2000/svg" class="h-4 w-4 nav-icon-map" fill="none" viewBox="0 0 24 24" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 20l-5.447-2.724A1 1 0 013 16.382V5.618a1 1 0 011.447-.894L9 7m0 13l6-3m-6 3V7m6 10l4.553 2.276A1 1 0 0021 18.382V7.618a1 1 0 00-.553-.894L15 4m0 13V4m0 0L9 7" /></svg> {{ t('entities.map') }}</a></li>
{% endif %}
+228
View File
@@ -0,0 +1,228 @@
"""Tests for raw packet API routes."""
from datetime import datetime, timezone
import pytest
from meshcore_hub.common.models import Channel, RawPacket
class TestListRawPackets:
"""Tests for GET /packets endpoint."""
def test_list_empty(self, client_no_auth):
response = client_no_auth.get("/api/v1/packets")
assert response.status_code == 200
data = response.json()
assert data["items"] == []
assert data["total"] == 0
def test_list_with_data(self, client_no_auth, api_db_session):
api_db_session.add(
RawPacket(
raw_hex="0011",
packet_hash="h1",
event_type="letsmesh_packet",
received_at=datetime.now(timezone.utc),
)
)
api_db_session.commit()
response = client_no_auth.get("/api/v1/packets")
assert response.status_code == 200
data = response.json()
assert data["total"] == 1
item = data["items"][0]
assert item["raw_hex"] == "0011"
assert item["redacted"] is False
def test_filter_event_type(self, client_no_auth, api_db_session):
now = datetime.now(timezone.utc)
api_db_session.add_all(
[
RawPacket(raw_hex="00", event_type="advertisement", received_at=now),
RawPacket(raw_hex="11", event_type="channel_msg_recv", received_at=now),
]
)
api_db_session.commit()
response = client_no_auth.get("/api/v1/packets?event_type=advertisement")
assert response.status_code == 200
data = response.json()
assert data["total"] == 1
assert data["items"][0]["event_type"] == "advertisement"
def test_filter_source_prefix_and_snr(self, client_no_auth, api_db_session):
now = datetime.now(timezone.utc)
api_db_session.add_all(
[
RawPacket(
raw_hex="00",
source_pubkey_prefix="AABBCCDDEEFF",
snr=10.0,
received_at=now,
),
RawPacket(
raw_hex="11",
source_pubkey_prefix="112233445566",
snr=2.0,
received_at=now,
),
]
)
api_db_session.commit()
response = client_no_auth.get(
"/api/v1/packets?pubkey_prefix=AABBCCDDEEFF&min_snr=5"
)
assert response.status_code == 200
data = response.json()
assert data["total"] == 1
assert data["items"][0]["source_pubkey_prefix"] == "AABBCCDDEEFF"
def test_pagination_params_echoed(self, client_no_auth):
response = client_no_auth.get("/api/v1/packets?limit=25&offset=5")
assert response.status_code == 200
data = response.json()
assert data["limit"] == 25
assert data["offset"] == 5
def test_filter_by_exact_packet_hash(self, client_no_auth, api_db_session):
now = datetime.now(timezone.utc)
api_db_session.add_all(
[
RawPacket(raw_hex="00", packet_hash="HASH_AAA", received_at=now),
RawPacket(raw_hex="11", packet_hash="HASH_AAA", received_at=now),
RawPacket(raw_hex="22", packet_hash="HASH_BBB", received_at=now),
]
)
api_db_session.commit()
response = client_no_auth.get("/api/v1/packets?packet_hash=HASH_AAA")
assert response.status_code == 200
data = response.json()
assert data["total"] == 2
assert {i["packet_hash"] for i in data["items"]} == {"HASH_AAA"}
class TestRawPacketRedaction:
"""Tests for channel-visibility redaction of raw packets."""
@pytest.fixture
def packets_with_visibility(self, api_db_session):
"""Community-channel, admin-channel, and non-channel raw packets."""
pub_key = "AABBCCDDEEFF00112233445566778899"
adm_key = "FFEEDDCCBBAA99887766554433221100"
pub_idx = int(Channel.compute_channel_hash(pub_key), 16)
adm_idx = int(Channel.compute_channel_hash(adm_key), 16)
api_db_session.add_all(
[
Channel(
name="CommunityCh",
key_hex=pub_key,
channel_hash=Channel.compute_channel_hash(pub_key),
visibility="community",
enabled=True,
),
Channel(
name="AdminCh",
key_hex=adm_key,
channel_hash=Channel.compute_channel_hash(adm_key),
visibility="admin",
enabled=True,
),
]
)
now = datetime.now(timezone.utc)
pub_pkt = RawPacket(
raw_hex="C0FFEE",
event_type="channel_msg_recv",
channel_idx=pub_idx,
decoded={"payload": {"decoded": {"channelHash": "x"}}},
received_at=now,
)
adm_pkt = RawPacket(
raw_hex="ADADAD",
event_type="channel_msg_recv",
channel_idx=adm_idx,
decoded={"payload": {"decoded": {"channelHash": "y"}}},
received_at=now,
)
plain_pkt = RawPacket(
raw_hex="DEADBE",
event_type="advertisement",
channel_idx=None,
received_at=now,
)
api_db_session.add_all([pub_pkt, adm_pkt, plain_pkt])
api_db_session.commit()
return pub_pkt, adm_pkt, plain_pkt
def test_anonymous_redacts_admin_channel(
self, client_no_auth, packets_with_visibility
):
"""Anonymous sees the admin-channel packet metadata-only (redacted)."""
response = client_no_auth.get("/api/v1/packets")
assert response.status_code == 200
data = response.json()
# All three rows are returned (count stable), but the admin one is redacted.
assert data["total"] == 3
adm = next(i for i in data["items"] if i["raw_hex"] is None)
assert adm["redacted"] is True
assert adm["decoded"] is None
# Non-channel and community packets are not redacted.
assert any(
i["raw_hex"] == "DEADBE" and not i["redacted"] for i in data["items"]
)
assert any(
i["raw_hex"] == "C0FFEE" and not i["redacted"] for i in data["items"]
)
def test_admin_sees_full(self, client_no_auth, packets_with_visibility):
"""Admin role sees the admin-channel packet in full."""
response = client_no_auth.get(
"/api/v1/packets", headers={"X-User-Roles": "admin"}
)
assert response.status_code == 200
data = response.json()
assert data["total"] == 3
assert all(not i["redacted"] for i in data["items"])
assert any(i["raw_hex"] == "ADADAD" for i in data["items"])
def test_redacted_filter(self, client_no_auth, packets_with_visibility):
"""The redacted filter narrows to (only|excluding) redacted rows."""
only = client_no_auth.get("/api/v1/packets?redacted=true").json()
assert only["total"] == 1
assert only["items"][0]["redacted"] is True
without = client_no_auth.get("/api/v1/packets?redacted=false").json()
assert without["total"] == 2
assert all(not i["redacted"] for i in without["items"])
def test_count_stable_across_roles(self, client_no_auth, packets_with_visibility):
"""Pagination count is the same regardless of role (rows redacted not hidden)."""
anon = client_no_auth.get("/api/v1/packets").json()
admin = client_no_auth.get(
"/api/v1/packets", headers={"X-User-Roles": "admin"}
).json()
assert anon["total"] == admin["total"] == 3
def test_get_single_redacted(self, client_no_auth, packets_with_visibility):
"""GET /packets/{id} redacts an above-role channel packet (not 404)."""
_, adm_pkt, _ = packets_with_visibility
response = client_no_auth.get(f"/api/v1/packets/{adm_pkt.id}")
assert response.status_code == 200
data = response.json()
assert data["redacted"] is True
assert data["raw_hex"] is None
def test_get_single_visible_to_admin(self, client_no_auth, packets_with_visibility):
_, adm_pkt, _ = packets_with_visibility
response = client_no_auth.get(
f"/api/v1/packets/{adm_pkt.id}", headers={"X-User-Roles": "admin"}
)
assert response.status_code == 200
data = response.json()
assert data["redacted"] is False
assert data["raw_hex"] == "ADADAD"
+82
View File
@@ -18,6 +18,7 @@ from meshcore_hub.common.models import (
Message,
Node,
NodeTag,
RawPacket,
Telemetry,
TracePath,
)
@@ -466,3 +467,84 @@ async def test_cleanup_clears_stale_observer_flags(
)
assert stale_flag is False
assert active_flag is True
@pytest.mark.asyncio
async def test_raw_packets_use_dedicated_retention(
async_db_session: AsyncSession,
) -> None:
"""Raw packets older than RAW_PACKET_RETENTION_DAYS are deleted."""
old_date = datetime.now(timezone.utc) - timedelta(days=20)
recent_date = datetime.now(timezone.utc) - timedelta(days=3)
async_db_session.add(
RawPacket(raw_hex="00", created_at=old_date, updated_at=old_date)
)
async_db_session.add(
RawPacket(raw_hex="11", created_at=recent_date, updated_at=recent_date)
)
await async_db_session.commit()
# Global retention 30 days would keep both; raw retention of 7 prunes the old one.
stats = await cleanup_old_data(
async_db_session,
retention_days=30,
dry_run=False,
raw_packet_retention_days=7,
)
assert stats.raw_packets_deleted == 1
remaining = await async_db_session.scalar(
select(func.count()).select_from(RawPacket)
)
assert remaining == 1
@pytest.mark.asyncio
async def test_raw_packet_retention_defaults_to_global(
async_db_session: AsyncSession,
) -> None:
"""When raw retention is None, the global retention value is used."""
old_date = datetime.now(timezone.utc) - timedelta(days=40)
async_db_session.add(
RawPacket(raw_hex="00", created_at=old_date, updated_at=old_date)
)
await async_db_session.commit()
stats = await cleanup_old_data(
async_db_session,
retention_days=30,
dry_run=False,
raw_packet_retention_days=None,
)
assert stats.raw_packets_deleted == 1
@pytest.mark.asyncio
async def test_raw_packet_only_observer_keeps_flag(
async_db_session: AsyncSession,
) -> None:
"""A node observing only (live) raw packets keeps is_observer=True."""
node = Node(public_key="d" * 64, is_observer=True)
async_db_session.add(node)
await async_db_session.flush()
recent_date = datetime.now(timezone.utc) - timedelta(days=2)
async_db_session.add(
RawPacket(
raw_hex="00",
observer_node_id=node.id,
created_at=recent_date,
updated_at=recent_date,
)
)
await async_db_session.commit()
stats = await cleanup_old_data(async_db_session, retention_days=30, dry_run=False)
assert stats.observers_cleared == 0
await async_db_session.rollback()
flag = await async_db_session.scalar(
select(Node.is_observer).where(Node.id == node.id)
)
assert flag is True
@@ -276,3 +276,19 @@ class TestHandleAdvertisement:
ads = db_session.execute(select(Advertisement)).scalars().all()
assert len(ads) == 1
class TestAdvertisementPacketHash:
"""packet_hash links an advertisement to its raw_packets rows."""
def test_stores_packet_hash(self, db_manager, db_session):
payload = {
"public_key": "a" * 64,
"name": "TestNode",
"adv_type": "chat",
"packet_hash": "FEEDFACE9999",
}
handle_advertisement("b" * 64, "advertisement", payload, db_manager)
ad = db_session.execute(select(Advertisement)).scalar_one()
assert ad.packet_hash == "FEEDFACE9999"
@@ -174,3 +174,29 @@ class TestHandleChannelMessage:
assert observer is not None
assert observer.path_len == 5
assert observer.snr == 10.0
class TestMessagePacketHash:
"""packet_hash links a message to its raw_packets rows."""
def test_stores_packet_hash_from_payload(self, db_manager, db_session):
payload = {
"pubkey_prefix": "01ab2186c4d5",
"text": "Hello!",
"packet_hash": "DEADBEEF1234",
}
handle_contact_message("a" * 64, "contact_msg_recv", payload, db_manager)
msg = db_session.execute(select(Message)).scalar_one()
assert msg.packet_hash == "DEADBEEF1234"
def test_falls_back_to_hash_field(self, db_manager, db_session):
payload = {
"pubkey_prefix": "01ab2186c4d5",
"text": "Hello again!",
"hash": "CAFEBABE5678",
}
handle_contact_message("a" * 64, "contact_msg_recv", payload, db_manager)
msg = db_session.execute(select(Message)).scalar_one()
assert msg.packet_hash == "CAFEBABE5678"
@@ -0,0 +1,98 @@
"""Tests for the raw packet capture handler."""
from sqlalchemy import select
from meshcore_hub.collector.handlers.raw_packet import store_raw_packet
from meshcore_hub.common.models import Node, RawPacket
def _channel_decoded() -> dict:
"""Decoder output resembling a channel-message packet on channel 0x2A."""
return {
"payloadType": 4,
"routeType": 1,
"pathLength": 2,
"payload": {
"decoded": {
"channelHash": "2A",
"sourceHash": "01AB2186C4D5EE",
}
},
}
class TestStoreRawPacket:
"""Tests for store_raw_packet."""
def test_captures_one_row(self, db_manager, db_session):
"""A single packet writes exactly one raw_packets row."""
payload = {"raw": "0011223344", "hash": "deadbeef", "SNR": 9.5, "path": "0102"}
store_raw_packet(
"a" * 64, payload, _channel_decoded(), "channel_msg_recv", db_manager
)
rows = db_session.execute(select(RawPacket)).scalars().all()
assert len(rows) == 1
rp = rows[0]
assert rp.raw_hex == "0011223344"
assert rp.packet_hash == "deadbeef"
assert rp.event_type == "channel_msg_recv"
assert rp.snr == 9.5
def test_derives_channel_idx_and_source_prefix(self, db_manager, db_session):
"""channel_idx from channelHash; source prefix from sourceHash."""
payload = {"raw": "00", "hash": "h1"}
store_raw_packet(
"a" * 64, payload, _channel_decoded(), "channel_msg_recv", db_manager
)
rp = db_session.execute(select(RawPacket)).scalar_one()
assert rp.channel_idx == 0x2A # 42
assert rp.source_pubkey_prefix == "01AB2186C4D5"
assert rp.route_type == "flood"
assert rp.path_len == 2
assert rp.payload_type == 4
def test_no_dedup_one_row_per_observer(self, db_manager, db_session):
"""The same packet hash from two observers stores two rows."""
payload = {"raw": "00", "hash": "shared"}
store_raw_packet(
"a" * 64, payload, _channel_decoded(), "channel_msg_recv", db_manager
)
store_raw_packet(
"b" * 64, payload, _channel_decoded(), "channel_msg_recv", db_manager
)
rows = db_session.execute(select(RawPacket)).scalars().all()
assert len(rows) == 2
assert {r.packet_hash for r in rows} == {"shared"}
def test_marks_observer_node(self, db_manager, db_session):
"""The observing node is created and flagged is_observer."""
payload = {"raw": "00", "hash": "h1"}
store_raw_packet(
"c" * 64, payload, _channel_decoded(), "channel_msg_recv", db_manager
)
node = db_session.execute(
select(Node).where(Node.public_key == "c" * 64)
).scalar_one()
assert node.is_observer is True
rp = db_session.execute(select(RawPacket)).scalar_one()
assert rp.observer_node_id == node.id
def test_non_channel_packet_has_null_channel_idx(self, db_manager, db_session):
"""A packet with no channelHash stores channel_idx = None."""
payload = {"raw": "00", "hash": "h1"}
decoded = {"payloadType": 1, "payload": {"decoded": {"sourceHash": "AABBCCDD"}}}
store_raw_packet("a" * 64, payload, decoded, "letsmesh_packet", db_manager)
rp = db_session.execute(select(RawPacket)).scalar_one()
assert rp.channel_idx is None
assert rp.source_pubkey_prefix == "AABBCCDD"[:12]
@@ -212,7 +212,7 @@ class TestGroupTextPacket:
result = norm._normalize_letsmesh_event("t", {"raw": raw})
assert result is not None
_, event_type, pl = result
assert event_type == "letsmesh_packet"
assert event_type == "encrypted_channel"
assert pl.get("decoded_payload_type") == 5
@@ -333,7 +333,7 @@ class TestTracePacket:
result = norm._normalize_letsmesh_event("t", {"raw": raw})
assert result is not None
_, event_type, pl = result
assert event_type == "letsmesh_packet"
assert event_type == "trace"
class TestControlPacket:
@@ -547,7 +547,7 @@ class TestResponsePacket:
class TestAckPacket:
def test_ack_falls_through_to_letsmesh_packet(self) -> None:
def test_ack_classified_as_ack(self) -> None:
raw = "ack1"
decoded = {
"payloadType": 3,
@@ -569,13 +569,13 @@ class TestAckPacket:
result = norm._normalize_letsmesh_event("t", {"raw": raw})
assert result is not None
_, event_type, pl = result
assert event_type == "letsmesh_packet"
assert event_type == "ack"
assert pl["decoded_payload_type"] == 3
assert pl["decoded_packet"]["payloadType"] == 3
class TestRequestPacket:
def test_request_without_decrypted_content_falls_through(self) -> None:
def test_request_without_decrypted_content_classified_as_req(self) -> None:
raw = "req1"
decoded = {
"payloadType": 0,
@@ -602,7 +602,7 @@ class TestRequestPacket:
result = norm._normalize_letsmesh_event("t", {"raw": raw})
assert result is not None
_, event_type, pl = result
assert event_type == "letsmesh_packet"
assert event_type == "req"
assert pl["decoded_payload_type"] == 0
@@ -625,3 +625,40 @@ class TestUnrecognizedFeed:
result = norm._normalize_letsmesh_event("garbage", {})
assert result is None
class TestFallbackClassification:
"""The fallback classifier maps payload types to specific event types."""
def test_payload_type_to_event_type_map(self) -> None:
cases = {
0: "req",
1: "response",
2: "encrypted_direct",
3: "ack",
4: "advert",
5: "encrypted_channel",
6: "grp_data",
7: "anon_req",
8: "path",
9: "trace",
10: "multipart",
11: "control",
15: "raw_custom",
}
for packet_type, expected in cases.items():
event_type = LetsMeshNormalizer._classify_fallback_event_type(
{"packet_type": packet_type}, None
)
assert event_type == expected, f"type {packet_type}"
def test_unknown_type_keeps_letsmesh_packet(self) -> None:
# Unresolvable type falls back to the generic safety-net label.
assert (
LetsMeshNormalizer._classify_fallback_event_type({}, None)
== "letsmesh_packet"
)
assert (
LetsMeshNormalizer._classify_fallback_event_type({"packet_type": 99}, None)
== "letsmesh_packet"
)
+109 -7
View File
@@ -83,6 +83,105 @@ class TestSubscriber:
handler.assert_called_once()
def test_raw_capture_disabled_writes_no_rows(self, mock_mqtt_client, db_manager):
"""With capture disabled, no raw_packets rows are written but the
structured handler still runs."""
from sqlalchemy import select
from meshcore_hub.common.models import RawPacket
mock_mqtt_client.topic_builder.parse_letsmesh_upload_topic.return_value = (
"a" * 64,
"packets",
)
subscriber = Subscriber(
mock_mqtt_client, db_manager, raw_packet_capture_enabled=False
)
handler = MagicMock()
subscriber.register_handler("channel_msg_recv", handler)
subscriber._normalize_letsmesh_event = MagicMock( # type: ignore[method-assign]
return_value=("a" * 64, "channel_msg_recv", {"text": "hi"})
)
subscriber._handle_mqtt_message(
topic="meshcore/STN/abc/packets",
pattern="meshcore/+/+/packets",
payload={"raw": "00", "hash": "h1"},
)
handler.assert_called_once()
session = db_manager.get_session()
try:
rows = session.execute(select(RawPacket)).scalars().all()
assert len(rows) == 0
finally:
session.close()
def test_raw_capture_enabled_writes_row(self, mock_mqtt_client, db_manager):
"""With capture enabled, a packets-feed message writes one raw row."""
from sqlalchemy import select
from meshcore_hub.common.models import RawPacket
mock_mqtt_client.topic_builder.parse_letsmesh_upload_topic.return_value = (
"a" * 64,
"packets",
)
subscriber = Subscriber(
mock_mqtt_client, db_manager, raw_packet_capture_enabled=True
)
subscriber._normalize_letsmesh_event = MagicMock( # type: ignore[method-assign]
return_value=("a" * 64, "channel_msg_recv", {"text": "hi"})
)
subscriber._letsmesh_decoder.decode_payload = MagicMock( # type: ignore[method-assign]
return_value={"payloadType": 4, "payload": {"decoded": {}}}
)
subscriber._handle_mqtt_message(
topic="meshcore/STN/abc/packets",
pattern="meshcore/+/+/packets",
payload={"raw": "0011", "hash": "h1"},
)
session = db_manager.get_session()
try:
rows = session.execute(select(RawPacket)).scalars().all()
assert len(rows) == 1
assert rows[0].raw_hex == "0011"
assert rows[0].event_type == "channel_msg_recv"
finally:
session.close()
def test_raw_capture_skips_status_feed(self, mock_mqtt_client, db_manager):
"""Capture is packets-feed only; status feed writes no raw rows."""
from sqlalchemy import select
from meshcore_hub.common.models import RawPacket
mock_mqtt_client.topic_builder.parse_letsmesh_upload_topic.return_value = (
"a" * 64,
"status",
)
subscriber = Subscriber(
mock_mqtt_client, db_manager, raw_packet_capture_enabled=True
)
subscriber._normalize_letsmesh_event = MagicMock( # type: ignore[method-assign]
return_value=("a" * 64, "letsmesh_status", {})
)
subscriber._handle_mqtt_message(
topic="meshcore/STN/abc/status",
pattern="meshcore/+/+/status",
payload={"some": "status"},
)
session = db_manager.get_session()
try:
rows = session.execute(select(RawPacket)).scalars().all()
assert len(rows) == 0
finally:
session.close()
def test_start_subscribes_to_letsmesh_topics(self, mock_mqtt_client, db_manager):
"""LetsMesh ingest mode subscribes to packets/status/internal feeds."""
subscriber = Subscriber(
@@ -246,7 +345,7 @@ class TestSubscriber:
def test_letsmesh_packet_without_decrypted_text_is_not_shown_as_message(
self, mock_mqtt_client, db_manager
) -> None:
"""Undecodable LetsMesh packets are kept as informational events, not messages."""
"""Undecodable channel packets are kept as informational events, not messages."""
mock_mqtt_client.topic_builder.parse_letsmesh_upload_topic.return_value = (
"a" * 64,
"packets",
@@ -255,9 +354,11 @@ class TestSubscriber:
mock_mqtt_client,
db_manager,
)
letsmesh_packet_handler = MagicMock()
# A GRP_TXT (type 5) that does not decrypt is classified as
# encrypted_channel, not routed to the channel message handler.
encrypted_handler = MagicMock()
channel_handler = MagicMock()
subscriber.register_handler("letsmesh_packet", letsmesh_packet_handler)
subscriber.register_handler("encrypted_channel", encrypted_handler)
subscriber.register_handler("channel_msg_recv", channel_handler)
subscriber.start()
@@ -276,7 +377,7 @@ class TestSubscriber:
},
)
letsmesh_packet_handler.assert_called_once()
encrypted_handler.assert_called_once()
channel_handler.assert_not_called()
def test_letsmesh_packet_uses_decoder_text_when_available(
@@ -760,7 +861,7 @@ class TestSubscriber:
def test_letsmesh_packet_fallback_logs_decoded_payload(
self, mock_mqtt_client, db_manager
) -> None:
"""Unmapped packets include decoder output in letsmesh_packet payload."""
"""Unmapped packets are classified by payload type and keep decoder output."""
mock_mqtt_client.topic_builder.parse_letsmesh_upload_topic.return_value = (
"a" * 64,
"packets",
@@ -769,8 +870,9 @@ class TestSubscriber:
mock_mqtt_client,
db_manager,
)
# MULTIPART (type 10) has no structured handler -> classified as multipart.
packet_handler = MagicMock()
subscriber.register_handler("letsmesh_packet", packet_handler)
subscriber.register_handler("multipart", packet_handler)
subscriber.start()
decoded_packet = {
@@ -799,7 +901,7 @@ class TestSubscriber:
packet_handler.assert_called_once()
_public_key, event_type, payload, _db = packet_handler.call_args.args
assert event_type == "letsmesh_packet"
assert event_type == "multipart"
assert payload["decoded_payload_type"] == 10
assert payload["decoded_packet"] == decoded_packet
+61
View File
@@ -14,6 +14,7 @@ from meshcore_hub.common.models import (
Telemetry,
EventLog,
EventObserver,
RawPacket,
add_event_observer,
)
@@ -67,6 +68,66 @@ class TestNodeModel:
assert node.tags[0].key == "altitude"
class TestRawPacketModel:
"""Tests for RawPacket model."""
def test_create_raw_packet(self, db_session) -> None:
"""Test creating a raw packet with defaults."""
packet = RawPacket(
packet_hash="abc123",
raw_hex="0011223344",
packet_type=5,
payload_type=4,
event_type="channel_msg_recv",
channel_idx=42,
source_pubkey_prefix="01AB2186C4D5",
route_type="flood",
path_len=2,
snr=12.5,
decoded={"payloadType": 4},
)
db_session.add(packet)
db_session.commit()
assert packet.id is not None
assert packet.received_at is not None
assert packet.created_at is not None
assert packet.updated_at is not None
assert packet.event_type == "channel_msg_recv"
assert packet.channel_idx == 42
assert packet.decoded == {"payloadType": 4}
def test_raw_packet_nullable_columns(self, db_session) -> None:
"""Test that the optional columns accept None."""
packet = RawPacket()
db_session.add(packet)
db_session.commit()
assert packet.id is not None
assert packet.observer_node_id is None
assert packet.packet_hash is None
assert packet.raw_hex is None
assert packet.channel_idx is None
assert packet.source_pubkey_prefix is None
assert packet.decoded is None
def test_raw_packet_indexes(self) -> None:
"""Test that the expected indexes are declared on the table."""
index_names = {idx.name for idx in RawPacket.__table__.indexes} # type: ignore[attr-defined]
for expected in (
"ix_raw_packets_received_at",
"ix_raw_packets_event_type",
"ix_raw_packets_packet_hash",
"ix_raw_packets_channel_idx",
"ix_raw_packets_source_pubkey_prefix",
"ix_raw_packets_observer_node_id",
"ix_raw_packets_event_type_received_at",
"ix_raw_packets_channel_idx_received_at",
"ix_raw_packets_source_pubkey_prefix_received_at",
):
assert expected in index_names
class TestMessageModel:
"""Tests for Message model."""
+42 -1
View File
@@ -6,7 +6,7 @@ import pytest
from fastapi.testclient import TestClient
from meshcore_hub.web.app import create_app
from tests.test_web.conftest import MockHttpClient
from tests.test_web.conftest import ALL_FEATURES_ENABLED, MockHttpClient
class TestFeatureFlagsConfig:
@@ -193,6 +193,47 @@ class TestFeatureFlagsSEO:
)
class TestPacketsFeatureFlag:
"""Test the packets feature flag (off by default)."""
def _make_app(self, mock_http_client: MockHttpClient, packets: bool):
features = dict(ALL_FEATURES_ENABLED)
features["packets"] = packets
app = create_app(
api_url="http://localhost:8000",
api_key="test-api-key",
network_name="Test Network",
features=features,
)
app.state.http_client = mock_http_client
return TestClient(app, raise_server_exceptions=True)
def test_packets_nav_hidden_when_disabled(
self, mock_http_client: MockHttpClient
) -> None:
"""The packets nav link is absent when the feature is off."""
client = self._make_app(mock_http_client, packets=False)
html = client.get("/").text
assert 'href="/packets"' not in html
# Messages still shows (ordering sanity)
assert 'href="/messages"' in html
def test_packets_nav_shown_when_enabled(
self, mock_http_client: MockHttpClient
) -> None:
"""The packets nav link appears when the feature is on."""
client = self._make_app(mock_http_client, packets=True)
html = client.get("/").text
assert 'href="/packets"' in html
def test_packets_disabled_by_default_in_settings(self) -> None:
"""The declared default for feature_packets is False (env-independent)."""
from meshcore_hub.common.config import WebSettings
# Check the field default directly so a local .env cannot mask it.
assert WebSettings.model_fields["feature_packets"].default is False
class TestFeatureFlagsIndividual:
"""Test individual feature flags."""