mirror of
https://github.com/ipnet-mesh/meshcore-hub.git
synced 2026-06-29 14:31:51 +02:00
Louis King
76f3dfa7eb
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
62 lines
4.1 KiB
Markdown
62 lines
4.1 KiB
Markdown
# LetsMesh Packet Decoding
|
|
|
|
The collector subscribes to packets published by [meshcore-packet-capture](https://github.com/agessaman/meshcore-packet-capture):
|
|
|
|
- `<prefix>/+/+/packets`
|
|
- `<prefix>/+/+/status`
|
|
- `<prefix>/+/+/internal`
|
|
|
|
## Normalization Behavior
|
|
|
|
- `status` packets are stored as informational `letsmesh_status` events and are not mapped to `advertisement` rows.
|
|
- Decoder payload type `4` is mapped to `advertisement` when node identity metadata is present.
|
|
- Decoder payload type `11` (control discover response) is mapped to `contact`.
|
|
- Decoder payload type `9` is mapped to `trace_data`.
|
|
- Decoder payload type `8` is mapped to informational `path_updated` events.
|
|
- Decoder payload type `1` can map to native response events (`telemetry_response`, `battery`, `path_updated`, `status_response`) when decrypted structured content is available.
|
|
- `packet_type=5` packets are mapped to `channel_msg_recv`.
|
|
- `packet_type=1`, `2`, and `7` packets are mapped to `contact_msg_recv` when decryptable text is available.
|
|
- Packets that no structured handler claims are no longer all labelled `letsmesh_packet`. They are classified by their MeshCore payload type so the `event_type` is specific:
|
|
|
|
| Payload type | `event_type` |
|
|
|---|---|
|
|
| `0x00 REQ` | `req` |
|
|
| `0x01 RESPONSE` | `response` |
|
|
| `0x02 TXT_MSG` (undecryptable) | `encrypted_direct` |
|
|
| `0x03 ACK` | `ack` |
|
|
| `0x04 ADVERT` (no identity) | `advert` |
|
|
| `0x05 GRP_TXT` (unknown key) | `encrypted_channel` |
|
|
| `0x06 GRP_DATA` | `grp_data` |
|
|
| `0x07 ANON_REQ` | `anon_req` |
|
|
| `0x08 PATH` | `path` |
|
|
| `0x09 TRACE` | `trace` |
|
|
| `0x0A MULTIPART` | `multipart` |
|
|
| `0x0B CONTROL` | `control` |
|
|
| `0x0F RAW_CUSTOM` | `raw_custom` |
|
|
|
|
`letsmesh_packet` is retained only as a safety net for packets whose payload type cannot be resolved. Reaching these fallbacks for `TXT_MSG`/`GRP_TXT` means the payload did not decrypt, hence the `encrypted_*` labels (decryptable ones become `contact_msg_recv` / `channel_msg_recv`).
|
|
|
|
## Raw Packet Capture
|
|
|
|
- When `RAW_PACKET_CAPTURE_ENABLED` is set (Compose derives it from `FEATURE_PACKETS`), every packet on the `packets` feed is also stored verbatim in the `raw_packets` table — one row per observer reception, independent of structured classification. The `status` and `internal` feeds carry no on-air `raw` hex and are **not** captured as raw packets.
|
|
- Capture reuses the single decode the normalizer already performs (the decoder caches per raw hex), so it adds only an insert plus an observer upsert to the ingest path.
|
|
- The `/packets` API and Packets page apply channel-visibility rules: channel-message packets on a channel above the viewer's role are returned **metadata-only with the payload redacted** (`redacted=true`, `raw_hex`/`decoded` nulled), not hidden. Non-channel and visible-channel packets are returned in full.
|
|
|
|
## Channel Keys
|
|
|
|
- For channel packets, if a channel key is available, a channel label is attached (for example `Public` or `#test`) for UI display.
|
|
- In the messages feed and dashboard channel sections, known channel indexes are preferred for labels (`17 -> Public`, `217 -> #test`) to avoid stale channel-name mismatches.
|
|
- Additional channel names are loaded from the `channels` database table (managed via CLI, API, or seed YAML).
|
|
- The collector keeps built-in keys for `Public` and `#test`, and merges any additional keys from enabled database channel rows.
|
|
|
|
## Location and Messages
|
|
|
|
- Decoder-advertisement packets with location metadata update node GPS (`lat/lon`) for map display.
|
|
- This keeps advertisement listings focused on node advert traffic only, not observer status telemetry.
|
|
- Packets without decryptable message text are kept as informational `letsmesh_packet` events and are not shown in the messages feed; when decode succeeds the decoded JSON is attached to those packet log events.
|
|
- When decoder output includes a human sender (`payload.decoded.decrypted.sender`), message text is normalized to `Name: Message` before storage; receiver/observer names are never used as sender fallback.
|
|
|
|
## Decoder Runtime
|
|
|
|
- Docker runtime uses the native Python `meshcoredecoder` library (no external Node.js dependency).
|