Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4.1 KiB
LetsMesh Packet Decoding
The collector subscribes to packets published by meshcore-packet-capture:
<prefix>/+/+/packets<prefix>/+/+/status<prefix>/+/+/internal
Normalization Behavior
-
statuspackets are stored as informationalletsmesh_statusevents and are not mapped toadvertisementrows. -
Decoder payload type
4is mapped toadvertisementwhen node identity metadata is present. -
Decoder payload type
11(control discover response) is mapped tocontact. -
Decoder payload type
9is mapped totrace_data. -
Decoder payload type
8is mapped to informationalpath_updatedevents. -
Decoder payload type
1can map to native response events (telemetry_response,battery,path_updated,status_response) when decrypted structured content is available. -
packet_type=5packets are mapped tochannel_msg_recv. -
packet_type=1,2, and7packets are mapped tocontact_msg_recvwhen decryptable text is available. -
Packets that no structured handler claims are no longer all labelled
letsmesh_packet. They are classified by their MeshCore payload type so theevent_typeis specific:Payload type event_type0x00 REQreq0x01 RESPONSEresponse0x02 TXT_MSG(undecryptable)encrypted_direct0x03 ACKack0x04 ADVERT(no identity)advert0x05 GRP_TXT(unknown key)encrypted_channel0x06 GRP_DATAgrp_data0x07 ANON_REQanon_req0x08 PATHpath0x09 TRACEtrace0x0A MULTIPARTmultipart0x0B CONTROLcontrol0x0F RAW_CUSTOMraw_customletsmesh_packetis retained only as a safety net for packets whose payload type cannot be resolved. Reaching these fallbacks forTXT_MSG/GRP_TXTmeans the payload did not decrypt, hence theencrypted_*labels (decryptable ones becomecontact_msg_recv/channel_msg_recv).
Raw Packet Capture
- When
RAW_PACKET_CAPTURE_ENABLEDis set (Compose derives it fromFEATURE_PACKETS), every packet on thepacketsfeed is also stored verbatim in theraw_packetstable — one row per observer reception, independent of structured classification. Thestatusandinternalfeeds carry no on-airrawhex and are not captured as raw packets. - Capture reuses the single decode the normalizer already performs (the decoder caches per raw hex), so it adds only an insert plus an observer upsert to the ingest path.
- The
/packetsAPI and Packets page apply channel-visibility rules: channel-message packets on a channel above the viewer's role are returned metadata-only with the payload redacted (redacted=true,raw_hex/decodednulled), not hidden. Non-channel and visible-channel packets are returned in full.
Channel Keys
- For channel packets, if a channel key is available, a channel label is attached (for example
Publicor#test) for UI display. - In the messages feed and dashboard channel sections, known channel indexes are preferred for labels (
17 -> Public,217 -> #test) to avoid stale channel-name mismatches. - Additional channel names are loaded from the
channelsdatabase table (managed via CLI, API, or seed YAML). - The collector keeps built-in keys for
Publicand#test, and merges any additional keys from enabled database channel rows.
Location and Messages
- Decoder-advertisement packets with location metadata update node GPS (
lat/lon) for map display. - This keeps advertisement listings focused on node advert traffic only, not observer status telemetry.
- Packets without decryptable message text are kept as informational
letsmesh_packetevents and are not shown in the messages feed; when decode succeeds the decoded JSON is attached to those packet log events. - When decoder output includes a human sender (
payload.decoded.decrypted.sender), message text is normalized toName: Messagebefore storage; receiver/observer names are never used as sender fallback.
Decoder Runtime
- Docker runtime uses the native Python
meshcoredecoderlibrary (no external Node.js dependency).