feat: raw packet capture, browse, and classification (v0.13.0)

Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.

Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
  composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
  reuses the decoder's per-hex cache (no second decode), one row per observer
  reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
  cleanup runs regardless of capture so disabling drains the table. Raw-packet
  observers retained in the is_observer recompute union.

API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
  cache key, and channel-visibility redaction (restricted-channel packets are
  returned metadata-only, not hidden, so pagination counts stay stable).

Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
  cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
  precedes Members; new packets icon + colour.

Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
  types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
  control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
  safety net.

Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
  exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
  Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
  on and the row has a stored hash.

Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Louis King
2026-06-12 22:40:31 +01:00
parent deb5af508c
commit 76f3dfa7eb
50 changed files with 3637 additions and 539 deletions
+17
View File
@@ -238,6 +238,20 @@ DATA_RETENTION_DAYS=30
# Applies to both event data and node cleanup
DATA_RETENTION_INTERVAL_HOURS=24
# -------------------
# Raw Packet Capture
# -------------------
# Capture every inbound packets-feed packet into the raw_packets table.
# Off by default. In Docker Compose this is derived from FEATURE_PACKETS, so
# setting FEATURE_PACKETS=true enables both capture and the Packets page.
# RAW_PACKET_CAPTURE_ENABLED=false
# Days to retain raw packets. Defaults to DATA_RETENTION_DAYS when unset.
# The raw_packets table grows fastest of all; lower this on busy meshes or
# constrained storage. Retention runs regardless of capture being enabled, so
# disabling capture lets existing rows drain.
# RAW_PACKET_RETENTION_DAYS=30
# -------------------
# Node Cleanup Settings
# -------------------
@@ -497,6 +511,9 @@ NETWORK_ANNOUNCEMENT=
# FEATURE_PAGES=true
# FEATURE_CHANNELS=true
# FEATURE_RADIO_CONFIG=true
# Packets page is OFF by default. Enabling it also drives raw-packet capture
# on the collector via Compose (RAW_PACKET_CAPTURE_ENABLED=${FEATURE_PACKETS}).
# FEATURE_PACKETS=false
# -------------------
# Contact Information