mirror of
https://github.com/ipnet-mesh/meshcore-hub.git
synced 2026-08-01 22:42:29 +02:00
docs(next-gen): drop LetsMesh brand terminology from new-repo vocabulary
LetsMesh (a now-inactive platform whose MQTT observer pattern + topic format this project adopted for compatibility) is not carried into the new repo. Genericize every brand-term-in-prose reference: - 'LetsMesh on-air hash' / 'LetsMesh wire hash' -> 'on-air wire hash' (the wire_hash field) across data-model, ingest, migration, infrastructure, review-findings, code-warts, overview. - 'MQTT LetsMesh feeds' -> 'MQTT upload feeds' (overview). - 'powers the official LetsMesh analyzer' -> 'powers the reference analyzer' (D22). Adds a 'Legacy brand terms' clause to the naming convention (phasing.md): on-air wire hash / MQTT upload feeds / descriptive fallback event types (unhandled_packet, not letsmesh_packet) / no letsmesh_* modules or classes in the new repo. Code-font citations of real legacy files/classes (letsmesh_normalizer.py, LetsMeshNormalizer, _normalize_letsmesh_event) remain in the current-system analysis as factual references to the old codebase.
This commit is contained in:
@@ -55,7 +55,7 @@
|
||||
**Rewrite:** [data-model.md](components/data-model.md) — fresh schema, each index deliberate, unique constraints enforced.
|
||||
|
||||
### DM6 — Dual-hash identity confusion
|
||||
**Where:** `event_hash` (MD5 content dedup) and `packet_hash` (LetsMesh wire hash) both appear on `Message`, `Advertisement`, `Telemetry`, `TracePath`, `RawPacket`, and denormalized onto `PacketPathHop`. The evaluator "prefers `event_hash` over `packet_hash`" with fallback logic. (`raw_packet.py:25-33`, `packet_path_hop.py:30-35`)
|
||||
**Where:** `event_hash` (MD5 content dedup) and `packet_hash` (the on-air wire hash) both appear on `Message`, `Advertisement`, `Telemetry`, `TracePath`, `RawPacket`, and denormalized onto `PacketPathHop`. The evaluator "prefers `event_hash` over `packet_hash`" with fallback logic. (`raw_packet.py:25-33`, `packet_path_hop.py:30-35`)
|
||||
**Cost:** Two nullable columns with fallback logic spread across 6 tables = real complexity and storage duplication.
|
||||
**Rewrite:** [data-model.md](components/data-model.md) §1.2 — `event_hash` (SHA-256) on dedup'd event tables; `wire_hash` only on `raw_receptions`.
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ These are the *structural* changes — column renames and feature additions are
|
||||
Standardize on **one** content identity column per event table:
|
||||
|
||||
- `event_hash` = content dedup key (stays).
|
||||
- `wire_hash` = the LetsMesh on-air hash, stored **only on `raw_receptions`** (the history store) and joined when needed — not denormalized onto every structured row.
|
||||
- `wire_hash` = the on-air wire hash, stored **only on `raw_receptions`** (the history store) and joined when needed — not denormalized onto every structured row.
|
||||
|
||||
### 1.3 Restructure the high-volume tables
|
||||
|
||||
@@ -352,7 +352,7 @@ CREATE TABLE raw_receptions (
|
||||
received_at timestamptz NOT NULL,
|
||||
id bigint GENERATED ALWAYS AS IDENTITY, -- cheap, hypertable-friendly
|
||||
observer_node_id uuid, -- loose ref to nodes.id; NO FK (see note below)
|
||||
wire_hash char(32), -- LetsMesh on-air hash; Nats-Msg-Id source
|
||||
wire_hash char(32), -- on-air wire hash; Nats-Msg-Id source
|
||||
event_hash bytea, -- backlink to dedup'd event, filled post-dispatch
|
||||
instance_id uuid NOT NULL REFERENCES instances(id),
|
||||
-- D8: raw_hex stays for now (TimescaleDB compresses it); object_key nullable for a later move
|
||||
|
||||
@@ -511,4 +511,4 @@ The end-to-end "stand up the new stack" order. This is a **greenfield** deployme
|
||||
8. **Cut over** DNS / MQTT exclusivity to the new stack.
|
||||
9. **Decommission** the old stack after the grace period.
|
||||
|
||||
Steps 6–9 are where D14 (5-day parallel-stack window) is exercised. The diff harness compares per-hour event counts and `wire_hash` coverage between the old API and the new API — it matches on the LetsMesh on-air `wire_hash` (identical in both stacks), **not** `event_hash`, which differs because the old stack hashes with MD5 and the new with SHA-256 (see migration.md → diff harness). Any divergence blocks cutover.
|
||||
Steps 6–9 are where D14 (5-day parallel-stack window) is exercised. The diff harness compares per-hour event counts and `wire_hash` coverage between the old API and the new API — it matches on the on-air `wire_hash` (identical in both stacks), **not** `event_hash`, which differs because the old stack hashes with MD5 and the new with SHA-256 (see migration.md → diff harness). Any divergence blocks cutover.
|
||||
|
||||
@@ -214,7 +214,7 @@ JSON for v1 (debuggable; the decode is the expensive part, not serialization). S
|
||||
"feed": "packets", // "packets" | "status" | "internal"
|
||||
"is_observer": true // IATA code matches a known observer; drives nodes.is_observer via touchNode
|
||||
},
|
||||
"wire_hash": "<32 hex>", // LetsMesh on-air hash; becomes Nats-Msg-Id
|
||||
"wire_hash": "<32 hex>", // on-air wire hash; becomes Nats-Msg-Id
|
||||
"mqtt": {
|
||||
"topic": "meshcore/IPT/<pubkey>/packets",
|
||||
"qos": 1,
|
||||
|
||||
@@ -127,7 +127,7 @@ meshcore-hub admin diff-stacks \
|
||||
> **Match on `wire_hash`, not `event_hash`.** The two stacks compute the content dedup hash with
|
||||
> different algorithms (old = MD5, new = SHA-256 truncated), so the *same* event has a different
|
||||
> `event_hash` in each stack — a coverage check keyed on `event_hash` would always report 0%. The
|
||||
> LetsMesh on-air `wire_hash` is identical in both stacks, so it is the correct join key for verifying the
|
||||
> The on-air `wire_hash` is identical in both stacks, so it is the correct join key for verifying the
|
||||
> new pipeline decoded the same packets.
|
||||
|
||||
**What it compares (per hour bucket, per event type):**
|
||||
|
||||
@@ -9,7 +9,7 @@ The rewrite plan (iterations 1–6) assumed Python (FastAPI + SQLAlchemy + Alemb
|
||||
|
||||
Two findings from the iteration-7 ecosystem research challenge this:
|
||||
|
||||
1. **The MeshCore packet decoder is TypeScript-primary.** `@michaelhart/meshcore-decoder` (npm, 50 stars, powers the official LetsMesh analyzer) is the original implementation. `meshcoredecoder` (PyPI, 8 stars) is explicitly a port that lags the TS original. The current hub carries ~100 LOC of workaround code (`_enrich_payload_decoded`, `_flatten_control_parsed`) patching fields the Python port omits. Every new MeshCore protocol feature lands in the TS decoder first.
|
||||
1. **The MeshCore packet decoder is TypeScript-primary.** `@michaelhart/meshcore-decoder` (npm, 50 stars, powers the reference analyzer) is the original implementation. `meshcoredecoder` (PyPI, 8 stars) is explicitly a port that lags the TS original. The current hub carries ~100 LOC of workaround code (`_enrich_payload_decoded`, `_flatten_control_parsed`) patching fields the Python port omits. Every new MeshCore protocol feature lands in the TS decoder first.
|
||||
|
||||
2. **The NATS JetStream Node client is first-party.** `@nats-io/nats-core` + `@nats-io/jetstream` (v3.4.0) is maintained by Synadia with typed consumer APIs, `Nats-Msg-Id` as a standard publish option, and active development. `nats-py` (v2.15.0) is community-maintained with sparse docs and slower feature parity. The ingest pipeline (D4) is the architectural centerpiece — the queue client quality matters.
|
||||
|
||||
|
||||
@@ -67,7 +67,7 @@ flowchart LR
|
||||
|
||||
| Domain | Feature | Notes |
|
||||
|---|---|---|
|
||||
| **Ingest** | MQTT LetsMesh feeds (`packets`, `status`, `internal`) | 3 wildcard topics; only ingest path |
|
||||
| **Ingest** | MQTT upload feeds (`packets`, `status`, `internal`) | 3 wildcard topics; only ingest path |
|
||||
| | Packet decode (meshcoredecoder lib) + 2048-entry cache | per-hex cache, FIFO eviction |
|
||||
| | Payload-type classification (0x00–0x0F) | normalizer cascade |
|
||||
| | Multi-observer aggregation | `event_observers` junction + dialect-aware upsert |
|
||||
@@ -165,7 +165,7 @@ These are grouped by the architectural lever that addresses them. The tags — *
|
||||
| W1 | **Single-threaded ingest.** `_on_message` runs on paho's network thread; one DB session per message; no batching, no backpressure. A slow DB stalls all topics. | `subscriber.py` dispatch; AGENTS.md notes "the collector is the only writer." |
|
||||
| W2 | **`raw_packets` write amplification.** 16 columns (incl. `raw_hex` Text + `decoded` JSON = duplicate payload storage), **9 indexes** (5 composite), one row per observer reception, no dedup. Retention capped at 2 days because of cost. | `raw_packet.py:121-138` |
|
||||
| W3 | **`packet_path_hops` write amplification.** One row per (reception × hop); 6-hop packet × 4 observers = 24 rows. Denormalizes 4 columns from `raw_packets`. Required a Postgres covering-index rebuild (HEAD migration) and a `window_hours` clamp as a perf band-aid. | `packet_path_hop.py`; `a59611449e2a` |
|
||||
| W4 | **Dual-hash identity confusion.** `event_hash` (MD5 dedup) and `packet_hash` (LetsMesh wire hash) both appear on 6 tables with fallback logic. The evaluator "prefers event_hash" for dedup. | `raw_packet.py:25-33` |
|
||||
| W4 | **Dual-hash identity confusion.** `event_hash` (MD5 dedup) and `packet_hash` (the on-air wire hash) both appear on 6 tables with fallback logic. The evaluator "prefers event_hash" for dedup. | `raw_packet.py:25-33` |
|
||||
| W5 | **MD5 for dedup keys.** Not a security issue but cheaply-collidable; SHA-256 truncated costs nothing. | `hash_utils.py` |
|
||||
| W6 | **Route health = hand-rolled materialized views.** 7 tables, 2 background cadences, full-scan candidate loads, in-Python per-day partitioning. `route_results` is 1:1 with `routes` (a cache). | `routes.py` (1,364 LOC); `route_evaluator.py` |
|
||||
| W7 | **`events_log` is an unbounded audit sink** that roughly doubles per-event storage. | `event_log.py` |
|
||||
|
||||
@@ -37,6 +37,18 @@ Internally, code stays **brand-agnostic** so a future rename never cascades thro
|
||||
- **Class & identifier names:** descriptive — `MqttIngester`, `IngestWorker`, `DerivedStateWorker`,
|
||||
`WebhookWorker`, `AuthMiddleware` — never brand-prefixed (`MeshloomIngester` etc.).
|
||||
|
||||
**Legacy brand terms — not carried forward.** LetsMesh (a now-inactive platform whose MQTT observer
|
||||
pattern + topic format this project adopted for compatibility) is dropped from the new repo's
|
||||
vocabulary. Use generic terms:
|
||||
|
||||
- "the on-air wire hash" (the `wire_hash` field) — not "LetsMesh hash".
|
||||
- "the MQTT upload feeds" (`packets` / `status` / `internal`) — not "LetsMesh feeds".
|
||||
- Descriptive fallback event types (e.g. `unhandled_packet`) — not `letsmesh_packet` / `letsmesh_status`.
|
||||
- No `letsmesh_*` module or class names in the new repo.
|
||||
|
||||
Code-font citations of legacy files (`letsmesh_normalizer.py`, `LetsMeshNormalizer`) remain in the
|
||||
current-system analysis (overview.md, code-warts.md) as factual references to the old codebase.
|
||||
|
||||
## Phase 0 — Foundations (no behavior change)
|
||||
|
||||
- **Run the D5 fold-vs-separate benchmark first** (synthetic data, throwaway TimescaleDB — testing.md → [D5 plan](testing.md#d5-benchmark-plan-fold-vs-separate)). Its outcome decides the `raw_receptions.path_hashes` shape, so it must land **before** the DDL is authored. This was previously (mis)scheduled in Phase 2, which created a circular dependency — the schema was "frozen" in Phase 0 but the benchmark that shapes it ran two phases later (F5).
|
||||
|
||||
@@ -105,7 +105,7 @@ Worker subscribe corrected to `meshcore.ingest.>`. Files: `components/ingest.md`
|
||||
The harness compared `event_hash` between stacks, but the old stack uses MD5 and the new uses SHA-256,
|
||||
so the same event has different hashes and coverage is always 0%.
|
||||
|
||||
**Resolution:** the hash-coverage check keys on `wire_hash` (the LetsMesh on-air hash, identical in both
|
||||
**Resolution:** the hash-coverage check keys on `wire_hash` (the on-air wire hash, identical in both
|
||||
stacks). Files: `components/migration.md`, `testing.md`.
|
||||
|
||||
### F10 — Spam-rescore sweep evaluated the scoring function twice per row
|
||||
|
||||
Reference in New Issue
Block a user