meshcore-hub/SCHEMAS.md

# MeshCore Event Schemas

This document defines the complete JSON payload schemas for all MeshCore events supported by the API.

## Event Categories

Events are categorized by how they're handled:

- **Persisted Events**: Stored in database tables and available via REST API
- **Webhook Events**: Trigger HTTP POST notifications when configured
- **Informational Events**: Logged but not persisted to separate tables

## Table of Contents

- [Persisted & Webhook Events](#persisted--webhook-events)
  - [ADVERTISEMENT / NEW_ADVERT](#advertisement--new_advert)
  - [CONTACT_MSG_RECV](#contact_msg_recv)
  - [CHANNEL_MSG_RECV](#channel_msg_recv)
- [Persisted Events (Non-Webhook)](#persisted-events-non-webhook)
  - [TRACE_DATA](#trace_data)
  - [TELEMETRY_RESPONSE](#telemetry_response)
  - [CONTACTS](#contacts)
- [Informational Events](#informational-events)
  - [SEND_CONFIRMED](#send_confirmed)
  - [STATUS_RESPONSE](#status_response)
  - [BATTERY](#battery)
  - [PATH_UPDATED](#path_updated)
- [Webhook Payload Format](#webhook-payload-format)

---

## Persisted & Webhook Events

These events are both stored in the database and trigger webhooks when configured.

### ADVERTISEMENT / NEW_ADVERT

Node advertisements announcing presence and metadata.

**Database Table**: `advertisements`

**Payload Schema**:
```json
{
  "public_key": "string (64 hex chars)",
  "name": "string (optional)",
  "adv_type": "string (optional)",
  "flags": "integer (optional)",
  "lat": "number (optional)",
  "lon": "number (optional)",
  "route_type": "string (optional)",
  "advert_timestamp": "integer (optional)"
}
```

**Field Descriptions**:
- `public_key`: Node's full 64-character hexadecimal public key (required)
- `name`: Node name/alias (e.g., "Gateway-01", "Alice")
- `adv_type`: Node type - common values: `"chat"`, `"repeater"`, `"room"`, `"companion"` (other values may appear from upstream feeds and are normalized by the collector when possible)
- `flags`: Node capability/status flags (bitmask)
- `lat`: GPS latitude when provided by decoder metadata
- `lon`: GPS longitude when provided by decoder metadata
- `route_type`: Route type of the advertisement packet — `"flood"` (original flood), `"transport_flood"` (relayed flood), `"direct"` (zero-hop local), `"transport_direct"` (relayed direct). Only present for LetsMesh-decoded adverts; native mode adverts have `route_type=NULL`.
- `advert_timestamp`: Node's own Unix timestamp (uint32) from the decoded advert payload. Used for deduplication when within ±4 hours of `received_at`. May be `NULL` for native mode adverts or when the decoder does not provide a timestamp.

**Example**:
```json
{
  "public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
  "name": "Gateway-01",
  "adv_type": "repeater",
  "flags": 218,
  "lat": 42.470001,
  "lon": -71.330001,
  "route_type": "flood",
  "advert_timestamp": 1747300000
}
```

**Webhook Trigger**: Yes
**REST API**: `GET /api/v1/advertisements`

---

### CONTACT_MSG_RECV

Direct/private messages between two nodes.

**Database Table**: `messages`

**Payload Schema**:
```json
{
  "pubkey_prefix": "string (12 chars)",
  "path_len": "integer (optional)",
  "txt_type": "integer (optional)",
  "signature": "string (optional)",
  "text": "string",
  "SNR": "number (optional)",
  "sender_timestamp": "integer (optional)"
}
```

**Field Descriptions**:
- `pubkey_prefix`: First 12 characters of the source public key prefix, used for message identification (or source hash prefix in compatibility ingest modes)
- `path_len`: Number of hops message traveled
- `txt_type`: Message type indicator (0=plain, 2=signed, etc.)
- `signature`: Message signature (8 hex chars) when `txt_type=2`
- `text`: Message content (required)
- `SNR`: Signal-to-Noise Ratio in dB
- `sender_timestamp`: Unix timestamp when message was sent

**Example**:
```json
{
  "pubkey_prefix": "01ab2186c4d5",
  "path_len": 3,
  "txt_type": 0,
  "signature": null,
  "text": "Hello Bob!",
  "SNR": 15.5,
  "sender_timestamp": 1732820498
}
```

**Webhook Trigger**: Yes
**REST API**: `GET /api/v1/messages`
**Webhook JSONPath Examples**:
- Send only text: `$.data.text`
- Send text + SNR: `$.data.[text,SNR]`

---

### CHANNEL_MSG_RECV

Group/broadcast messages on specific channels.

**Database Table**: `messages`

**Payload Schema**:
```json
{
  "channel_idx": "integer (optional)",
  "channel_name": "string (optional)",
  "pubkey_prefix": "string (12 chars, optional)",
  "path_len": "integer (optional)",
  "txt_type": "integer (optional)",
  "signature": "string (optional)",
  "text": "string",
  "SNR": "number (optional)",
  "sender_timestamp": "integer (optional)"
}
```

**Field Descriptions**:
- `channel_idx`: Channel number (0-255) when available
- `channel_name`: Channel display label (e.g., `"Public"`, `"Community"`, `"#test"`) when available
- `pubkey_prefix`: First 12 characters of the source public key prefix, used for message identification when available
- `path_len`: Number of hops message traveled
- `txt_type`: Message type indicator (0=plain, 2=signed, etc.)
- `signature`: Message signature (8 hex chars) when `txt_type=2`
- `text`: Message content (required)
- `SNR`: Signal-to-Noise Ratio in dB
- `sender_timestamp`: Unix timestamp when message was sent

**Example**:
```json
{
  "channel_idx": 4,
  "path_len": 10,
  "txt_type": 0,
  "signature": null,
  "text": "Hello from the mesh!",
  "SNR": 8.5,
  "sender_timestamp": 1732820498
}
```

**Webhook Trigger**: Yes
**REST API**: `GET /api/v1/messages`
**Webhook JSONPath Examples**:
- Send only text: `$.data.text`
- Send channel + text: `$.data.[channel_idx,text]`

**Compatibility ingest note**:
- In LetsMesh upload compatibility mode, packet type `5` is normalized to `CHANNEL_MSG_RECV` and packet types `1`, `2`, and `7` are normalized to `CONTACT_MSG_RECV` when decryptable text is available.
- LetsMesh packets without decryptable message text are treated as informational `letsmesh_packet` events instead of message events.
- For UI labels, known channel indexes are mapped (`17 -> Public`, `217 -> #test`) and preferred over ambiguous/stale channel-name hints.
- Additional channel labels are loaded from the `channels` database table via the collector's periodic refresh.
- When decoder output includes a human sender (`payload.decoded.decrypted.sender`), message text is normalized to `Name: Message`; sender identity remains unknown when only hash/prefix metadata is available.

**Spam scoring columns** (stored on `messages`, populated only when `SPAM_DETECTION_ENABLED` is set; otherwise null):
- `path_prefix` (`string(48)`): first `SPAM_PATH_HOPS` origin-side hop hashes joined (e.g. `"16,69,23"`), derived from the decoded `path`. Stored null when `path_len` is below `SPAM_MIN_PATH_HOPS` so short local-mesh paths don't pollute path counts. Indexed with `received_at`.
- `sender_normalized` (`string(255)`): the self-declared channel sender name (the part before the first `": "` in the body) lower-cased with the trailing digit/space suffix stripped (`bob17` → `bob`). Indexed with `received_at`.
- `spam_score` (`float`): likely-spam score `0.0`–`1.0` computed at ingest from windowed joint `(path_prefix, sender_normalized)` and `sender_normalized` frequency counts, refined by a background re-scoring sweep. Messages at/above `SPAM_SCORE_THRESHOLD` are hidden from `GET /api/v1/messages` unless `include_spam=true`.

**Compatibility ingest note (advertisements)**:
- In LetsMesh upload compatibility mode, `status` feed payloads are persisted as informational `letsmesh_status` events and are not normalized to `ADVERTISEMENT`.
- In LetsMesh upload compatibility mode, decoded payload type `4` is normalized to `ADVERTISEMENT` when node identity metadata is present.
- Payload type `4` location metadata (`appData.location.latitude/longitude`) is mapped to node `lat/lon` for map rendering.
- This keeps advertisement persistence aligned with meshcore-packet-capture expectations (advertisement traffic only).

**Compatibility ingest note (envelope fields)**:
- The LetsMesh upload envelope carries `SNR` and `path` fields alongside the decoded packet payload. These are available on all packet types (messages, advertisements, traces, telemetry).
- The normalizer extracts `SNR` (normalized to lowercase `snr`) and `path` (converted to `path_len` via hop count) from the envelope and includes them in the normalized payload.
- Per-observer `snr` and `path_len` are stored in the `event_observers` junction table, allowing each observer to record its own signal strength and hop count.

**Compatibility ingest note (non-message structured events)**:
- Decoded payload type `9` is normalized to `TRACE_DATA` (`traceTag`, flags, auth, path hashes, and SNR values).
- Decoded payload type `11` (`Control/NodeDiscoverResp`) is normalized to `contact` events for node upsert parity.
- Decoded payload type `8` is normalized to informational `PATH_UPDATED` events (`hop_count` + path hashes).
- Decoded payload type `1` can be normalized to `TELEMETRY_RESPONSE`, `BATTERY`, `PATH_UPDATED`, or `STATUS_RESPONSE` when decrypted response content is structured and parseable.

---

## Persisted Events (Non-Webhook)

These events are stored in the database but do not trigger webhooks.

### TRACE_DATA

Network trace path results showing route and signal strength.

**Database Table**: `trace_paths`

**Payload Schema**:
```json
{
  "initiator_tag": "integer",
  "path_len": "integer (optional)",
  "flags": "integer (optional)",
  "auth": "integer (optional)",
  "path_hashes": "array of strings",
  "snr_values": "array of numbers",
  "hop_count": "integer (optional)"
}
```

**Field Descriptions**:
- `initiator_tag`: Unique trace identifier (0-4294967295)
- `path_len`: Length of the path
- `flags`: Trace flags/options
- `auth`: Authentication/validation data
- `path_hashes`: Array of hex-encoded node hash identifiers, variable length (e.g., `"4a"` for single-byte, `"b3fa"` for multibyte), ordered by hops
- `snr_values`: Array of SNR values corresponding to each hop
- `hop_count`: Total number of hops

**Example**:
```json
{
  "initiator_tag": 123456789,
  "path_len": 3,
  "flags": 0,
  "auth": 1,
  "path_hashes": ["4a", "b3fa", "02"],
  "snr_values": [25.3, 18.7, 12.4],
  "hop_count": 3
}
```

**Note**: MeshCore firmware v1.14+ supports multibyte path hashes. Older nodes use single-byte (2-character) hashes. Mixed-length hash arrays are expected in heterogeneous networks where nodes run different firmware versions.

**Webhook Trigger**: No
**REST API**: `GET /api/v1/trace-paths`

---

### TELEMETRY_RESPONSE

Sensor data from network nodes (temperature, humidity, battery, etc.).

**Database Table**: `telemetry`

**Payload Schema**:
```json
{
  "node_public_key": "string (64 hex chars)",
  "lpp_data": "bytes (optional)",
  "parsed_data": "object (optional)"
}
```

**Field Descriptions**:
- `node_public_key`: Full public key of the reporting node
- `lpp_data`: Raw LPP-encoded sensor data (CayenneLPP format)
- `parsed_data`: Decoded sensor readings as key-value pairs

**Example**:
```json
{
  "node_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
  "lpp_data": null,
  "parsed_data": {
    "temperature": 22.5,
    "humidity": 65,
    "battery": 3.8,
    "pressure": 1013.25
  }
}
```

**Webhook Trigger**: No
**REST API**: `GET /api/v1/telemetry`

---

### CONTACTS

Contact sync event containing all known nodes.

**Database Table**: Updates `nodes` table

**Payload Schema**:
```json
{
  "contacts": [
    {
      "public_key": "string (64 hex chars)",
      "name": "string (optional)",
      "node_type": "string (optional)"
    }
  ]
}
```

**Field Descriptions**:
- `contacts`: Array of contact objects
- `public_key`: Node's full public key
- `name`: Node name/alias
- `node_type`: One of: `"chat"`, `"repeater"`, `"room"`, `"none"`

**Example**:
```json
{
  "contacts": [
    {
      "public_key": "01ab2186c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1",
      "name": "Alice",
      "node_type": "chat"
    },
    {
      "public_key": "b3f4e5d6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4",
      "name": "Bob",
      "node_type": "chat"
    }
  ]
}
```

**Webhook Trigger**: No
**REST API**: `GET /api/v1/nodes`

---

## Informational Events

These events are logged to `events_log` table but not persisted to separate tables.

### SEND_CONFIRMED

Confirmation that a sent message was delivered.

**Payload Schema**:
```json
{
  "destination_public_key": "string (64 hex chars)",
  "round_trip_ms": "integer"
}
```

**Field Descriptions**:
- `destination_public_key`: Recipient's full public key
- `round_trip_ms`: Round-trip time in milliseconds

**Example**:
```json
{
  "destination_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
  "round_trip_ms": 2500
}
```

---

### STATUS_RESPONSE

Device status information.

**Payload Schema**:
```json
{
  "node_public_key": "string (64 hex chars)",
  "status": "string (optional)",
  "uptime": "integer (optional)",
  "message_count": "integer (optional)"
}
```

**Field Descriptions**:
- `node_public_key`: Node's full public key
- `status`: Status description
- `uptime`: Uptime in seconds
- `message_count`: Total messages processed

**Example**:
```json
{
  "node_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
  "status": "operational",
  "uptime": 86400,
  "message_count": 1523
}
```

---

### BATTERY

Battery status information.

**Payload Schema**:
```json
{
  "battery_voltage": "number",
  "battery_percentage": "integer"
}
```

**Field Descriptions**:
- `battery_voltage`: Battery voltage (e.g., 3.7V)
- `battery_percentage`: Battery level 0-100%

**Example**:
```json
{
  "battery_voltage": 3.8,
  "battery_percentage": 75
}
```

---

### PATH_UPDATED

Notification that routing path to a node has changed.

**Payload Schema**:
```json
{
  "node_public_key": "string (64 hex chars)",
  "hop_count": "integer"
}
```

**Field Descriptions**:
- `node_public_key`: Target node's full public key
- `hop_count`: Number of hops in new path

**Example**:
```json
{
  "node_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
  "hop_count": 3
}
```

---

### Per-Payload-Type Classifications

Packets on the LetsMesh `packets` feed that no structured handler claims are classified by their MeshCore payload type rather than the generic `letsmesh_packet`. These are informational (logged to `events_log`, captured to `raw_packets`), with no dedicated table:

| Payload type | `event_type` | Notes |
|---|---|---|
| `0x00 REQ` | `req` | Encrypted request, metadata only |
| `0x01 RESPONSE` | `response` | Response with no structured content |
| `0x02 TXT_MSG` | `encrypted_direct` | Direct text that did not decrypt |
| `0x03 ACK` | `ack` | Acknowledgment |
| `0x04 ADVERT` | `advert` | Advert lacking identity metadata |
| `0x05 GRP_TXT` | `encrypted_channel` | Channel text with unknown key |
| `0x06 GRP_DATA` | `grp_data` | Group datagram |
| `0x07 ANON_REQ` | `anon_req` | Anonymous request |
| `0x08 PATH` | `path` | Returned path (unparsed) |
| `0x09 TRACE` | `trace` | Trace (unparsed) |
| `0x0A MULTIPART` | `multipart` | Fragment of a sequence (no reassembly) |
| `0x0B CONTROL` | `control` | Control packet not matched to contact/status |
| `0x0F RAW_CUSTOM` | `raw_custom` | Custom-encrypted, opaque |

`letsmesh_packet` remains as the safety-net label when the payload type cannot be resolved.

### Other Informational Events

The following events are logged but have varying or device-specific payloads:

- **STATISTICS**: Network statistics (varies by implementation)
- **DEVICE_INFO**: Device hardware/firmware information
- **BINARY_RESPONSE**: Binary data responses
- **CONTROL_DATA**: Control/command responses
- **RAW_DATA**: Raw protocol data
- **NEXT_CONTACT**: Contact enumeration progress
- **ERROR**: Error messages with description
- **MESSAGES_WAITING**: Notification of queued messages
- **NO_MORE_MSGS**: End of message queue
- **RX_LOG_DATA**: Receive log data

---

## Raw Packets

When `RAW_PACKET_CAPTURE_ENABLED` is set (Compose derives it from `FEATURE_PACKETS`), the collector stores **every** inbound packet from the LetsMesh `packets` feed exactly as received, independent of how it is classified. One row is written per observer reception (no deduplication), reusing the decode the normalizer already performs.

**Database Table**: `raw_packets`

| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key |
| `observer_node_id` | FK `nodes.id` (SET NULL) | Receiving interface, from the MQTT topic public key |
| `packet_hash` | string(32) | LetsMesh packet hash; links to structured records, groups multi-observer receptions |
| `raw_hex` | text | On-air bytes from `payload["raw"]` |
| `packet_type` | integer | Wire packet type |
| `payload_type` | integer | Decoder payload type |
| `event_type` | string(50) | How the collector classified the packet (`advertisement`, `channel_msg_recv`, `letsmesh_packet`, …) |
| `channel_idx` | integer | `int(channelHash, 16)` for channel-message packets, else NULL; drives visibility redaction |
| `source_pubkey_prefix` | string(12) | Sender prefix from decoder `sourceHash` / `senderPublicKey` |
| `route_type` | string(20) | `flood`, `transport_flood`, `direct`, `transport_direct` |
| `path_len` | integer | Hop count |
| `snr` | float | Signal-to-noise ratio reported by the observer |
| `decoded` | JSON | Decoder summary (so detail views need no re-decode) |
| `received_at` | datetime | When received by the observing interface |

**Linking from structured tables**: `advertisements` and `messages` carry a nullable `packet_hash` (the LetsMesh wire hash) populated at ingest. Because the structured tables are deduplicated across observers, one row links to *all* the per-observer `raw_packets` sharing that hash — reachable via `GET /api/v1/packets?packet_hash=<hash>`. Only rows ingested while capture was enabled have a populated `packet_hash` (no backfill).

**REST API**: `GET /api/v1/packets`, `GET /api/v1/packets/{id}`

**`RawPacketRead` shape** mirrors the columns above plus a `redacted: bool` field and observer details (`observed_by`, `observer_name`, `observer_tag_name`). Channel-message packets on a channel above the requesting role's visibility level are returned with `redacted=true` and `raw_hex`/`decoded`/`source_pubkey_prefix` nulled; non-channel and visible-channel packets are returned in full. Responses are cached in Redis with a role-aware key.

Filters: `search` (hash/raw_hex/observer, bounded to a recent window when no `since` given), `event_type`, `packet_type`, `channel_idx`, `route_type`, `public_key`/`pubkey_prefix`, `observed_by`, `decryptable`, `min_snr`/`max_snr`, `min_path_len`/`max_path_len`, `redacted`, `since`/`until`, `sort`/`order`, `limit`/`offset`.

---

## Webhook Payload Format

All webhook events are wrapped in a standard envelope before being sent:

```json
{
  "event_type": "string",
  "timestamp": "string (ISO 8601)",
  "data": {
    // Event-specific payload (see schemas above)
  }
}
```

**Example (Channel Message)**:
```json
{
  "event_type": "CHANNEL_MSG_RECV",
  "timestamp": "2025-11-28T19:41:38.748379Z",
  "data": {
    "channel_idx": 4,
    "path_len": 10,
    "txt_type": 0,
    "text": "Hello from the mesh!",
    "SNR": 8.5,
    "sender_timestamp": 1732820498
  }
}
```

### JSONPath Filtering

You can filter webhook payloads using JSONPath expressions:

| JSONPath | Result |
|----------|--------|
| `$` | Full payload (default) |
| `$.data` | Event data only |
| `$.data.text` | Message text only |
| `$.data.[text,SNR]` | Multiple fields as array |
| `$.event_type` | Event type string |
| `$.timestamp` | Timestamp string |

See [AGENTS.md](AGENTS.md) for webhook configuration details.

---

## API Response: Observer Info

Events that support multi-observer tracking (messages, advertisements, trace paths, telemetry) include an `observers` array in API responses. Each observer entry contains:

| Field | Type | Description |
|-------|------|-------------|
| `node_id` | string (UUID) | Observer node UUID |
| `public_key` | string (64 hex chars) | Observer node public key |
| `name` | string or null | Observer node advertised name |
| `tag_name` | string or null | Observer name from node tags |
| `snr` | number or null | Signal-to-noise ratio at this observer (dB) |
| `path_len` | integer or null | Hop count at this observer |
| `observed_at` | string (ISO 8601) | When this observer captured the event |

---

## Event Flow

1. **Hardware/Mock MeshCore** → Generates raw events
2. **Event Handler** → Processes and persists to database
3. **Webhook Handler** → Sends HTTP POST to configured URLs (if enabled)
4. **REST API** → Query historical data from database

Most events are logged to the `events_log` table with full payloads for debugging and audit purposes. Some high-frequency informational events (e.g., `NEXT_CONTACT`) are intentionally excluded from logging to reduce database size.