From fd51e7298ff2835a8974a02e170e7ebaa9873672 Mon Sep 17 00:00:00 2001 From: Louis King Date: Sun, 26 Jul 2026 14:48:30 +0100 Subject: [PATCH] docs(next-gen): drop LetsMesh brand terminology from new-repo vocabulary LetsMesh (a now-inactive platform whose MQTT observer pattern + topic format this project adopted for compatibility) is not carried into the new repo. Genericize every brand-term-in-prose reference: - 'LetsMesh on-air hash' / 'LetsMesh wire hash' -> 'on-air wire hash' (the wire_hash field) across data-model, ingest, migration, infrastructure, review-findings, code-warts, overview. - 'MQTT LetsMesh feeds' -> 'MQTT upload feeds' (overview). - 'powers the official LetsMesh analyzer' -> 'powers the reference analyzer' (D22). Adds a 'Legacy brand terms' clause to the naming convention (phasing.md): on-air wire hash / MQTT upload feeds / descriptive fallback event types (unhandled_packet, not letsmesh_packet) / no letsmesh_* modules or classes in the new repo. Code-font citations of real legacy files/classes (letsmesh_normalizer.py, LetsMeshNormalizer, _normalize_letsmesh_event) remain in the current-system analysis as factual references to the old codebase. --- docs/plans/next-gen/code-warts.md | 2 +- docs/plans/next-gen/components/data-model.md | 4 ++-- docs/plans/next-gen/components/infrastructure.md | 2 +- docs/plans/next-gen/components/ingest.md | 2 +- docs/plans/next-gen/components/migration.md | 2 +- .../decisions/D22-node-typescript-backend.md | 2 +- docs/plans/next-gen/overview.md | 4 ++-- docs/plans/next-gen/phasing.md | 12 ++++++++++++ docs/plans/next-gen/review-findings.md | 2 +- 9 files changed, 22 insertions(+), 10 deletions(-) diff --git a/docs/plans/next-gen/code-warts.md b/docs/plans/next-gen/code-warts.md index 12b2d14..3436323 100644 --- a/docs/plans/next-gen/code-warts.md +++ b/docs/plans/next-gen/code-warts.md @@ -55,7 +55,7 @@ **Rewrite:** [data-model.md](components/data-model.md) — fresh schema, each index deliberate, unique constraints enforced. ### DM6 — Dual-hash identity confusion -**Where:** `event_hash` (MD5 content dedup) and `packet_hash` (LetsMesh wire hash) both appear on `Message`, `Advertisement`, `Telemetry`, `TracePath`, `RawPacket`, and denormalized onto `PacketPathHop`. The evaluator "prefers `event_hash` over `packet_hash`" with fallback logic. (`raw_packet.py:25-33`, `packet_path_hop.py:30-35`) +**Where:** `event_hash` (MD5 content dedup) and `packet_hash` (the on-air wire hash) both appear on `Message`, `Advertisement`, `Telemetry`, `TracePath`, `RawPacket`, and denormalized onto `PacketPathHop`. The evaluator "prefers `event_hash` over `packet_hash`" with fallback logic. (`raw_packet.py:25-33`, `packet_path_hop.py:30-35`) **Cost:** Two nullable columns with fallback logic spread across 6 tables = real complexity and storage duplication. **Rewrite:** [data-model.md](components/data-model.md) §1.2 — `event_hash` (SHA-256) on dedup'd event tables; `wire_hash` only on `raw_receptions`. diff --git a/docs/plans/next-gen/components/data-model.md b/docs/plans/next-gen/components/data-model.md index fcd1015..ce5d617 100644 --- a/docs/plans/next-gen/components/data-model.md +++ b/docs/plans/next-gen/components/data-model.md @@ -24,7 +24,7 @@ These are the *structural* changes — column renames and feature additions are Standardize on **one** content identity column per event table: - `event_hash` = content dedup key (stays). -- `wire_hash` = the LetsMesh on-air hash, stored **only on `raw_receptions`** (the history store) and joined when needed — not denormalized onto every structured row. +- `wire_hash` = the on-air wire hash, stored **only on `raw_receptions`** (the history store) and joined when needed — not denormalized onto every structured row. ### 1.3 Restructure the high-volume tables @@ -352,7 +352,7 @@ CREATE TABLE raw_receptions ( received_at timestamptz NOT NULL, id bigint GENERATED ALWAYS AS IDENTITY, -- cheap, hypertable-friendly observer_node_id uuid, -- loose ref to nodes.id; NO FK (see note below) - wire_hash char(32), -- LetsMesh on-air hash; Nats-Msg-Id source + wire_hash char(32), -- on-air wire hash; Nats-Msg-Id source event_hash bytea, -- backlink to dedup'd event, filled post-dispatch instance_id uuid NOT NULL REFERENCES instances(id), -- D8: raw_hex stays for now (TimescaleDB compresses it); object_key nullable for a later move diff --git a/docs/plans/next-gen/components/infrastructure.md b/docs/plans/next-gen/components/infrastructure.md index e3e78e5..e1ed72f 100644 --- a/docs/plans/next-gen/components/infrastructure.md +++ b/docs/plans/next-gen/components/infrastructure.md @@ -511,4 +511,4 @@ The end-to-end "stand up the new stack" order. This is a **greenfield** deployme 8. **Cut over** DNS / MQTT exclusivity to the new stack. 9. **Decommission** the old stack after the grace period. -Steps 6–9 are where D14 (5-day parallel-stack window) is exercised. The diff harness compares per-hour event counts and `wire_hash` coverage between the old API and the new API — it matches on the LetsMesh on-air `wire_hash` (identical in both stacks), **not** `event_hash`, which differs because the old stack hashes with MD5 and the new with SHA-256 (see migration.md → diff harness). Any divergence blocks cutover. +Steps 6–9 are where D14 (5-day parallel-stack window) is exercised. The diff harness compares per-hour event counts and `wire_hash` coverage between the old API and the new API — it matches on the on-air `wire_hash` (identical in both stacks), **not** `event_hash`, which differs because the old stack hashes with MD5 and the new with SHA-256 (see migration.md → diff harness). Any divergence blocks cutover. diff --git a/docs/plans/next-gen/components/ingest.md b/docs/plans/next-gen/components/ingest.md index 25559e1..09608f1 100644 --- a/docs/plans/next-gen/components/ingest.md +++ b/docs/plans/next-gen/components/ingest.md @@ -214,7 +214,7 @@ JSON for v1 (debuggable; the decode is the expensive part, not serialization). S "feed": "packets", // "packets" | "status" | "internal" "is_observer": true // IATA code matches a known observer; drives nodes.is_observer via touchNode }, - "wire_hash": "<32 hex>", // LetsMesh on-air hash; becomes Nats-Msg-Id + "wire_hash": "<32 hex>", // on-air wire hash; becomes Nats-Msg-Id "mqtt": { "topic": "meshcore/IPT//packets", "qos": 1, diff --git a/docs/plans/next-gen/components/migration.md b/docs/plans/next-gen/components/migration.md index d1eee0b..f66869d 100644 --- a/docs/plans/next-gen/components/migration.md +++ b/docs/plans/next-gen/components/migration.md @@ -127,7 +127,7 @@ meshcore-hub admin diff-stacks \ > **Match on `wire_hash`, not `event_hash`.** The two stacks compute the content dedup hash with > different algorithms (old = MD5, new = SHA-256 truncated), so the *same* event has a different > `event_hash` in each stack — a coverage check keyed on `event_hash` would always report 0%. The -> LetsMesh on-air `wire_hash` is identical in both stacks, so it is the correct join key for verifying the +> The on-air `wire_hash` is identical in both stacks, so it is the correct join key for verifying the > new pipeline decoded the same packets. **What it compares (per hour bucket, per event type):** diff --git a/docs/plans/next-gen/decisions/D22-node-typescript-backend.md b/docs/plans/next-gen/decisions/D22-node-typescript-backend.md index d6dd7fe..d63a877 100644 --- a/docs/plans/next-gen/decisions/D22-node-typescript-backend.md +++ b/docs/plans/next-gen/decisions/D22-node-typescript-backend.md @@ -9,7 +9,7 @@ The rewrite plan (iterations 1–6) assumed Python (FastAPI + SQLAlchemy + Alemb Two findings from the iteration-7 ecosystem research challenge this: -1. **The MeshCore packet decoder is TypeScript-primary.** `@michaelhart/meshcore-decoder` (npm, 50 stars, powers the official LetsMesh analyzer) is the original implementation. `meshcoredecoder` (PyPI, 8 stars) is explicitly a port that lags the TS original. The current hub carries ~100 LOC of workaround code (`_enrich_payload_decoded`, `_flatten_control_parsed`) patching fields the Python port omits. Every new MeshCore protocol feature lands in the TS decoder first. +1. **The MeshCore packet decoder is TypeScript-primary.** `@michaelhart/meshcore-decoder` (npm, 50 stars, powers the reference analyzer) is the original implementation. `meshcoredecoder` (PyPI, 8 stars) is explicitly a port that lags the TS original. The current hub carries ~100 LOC of workaround code (`_enrich_payload_decoded`, `_flatten_control_parsed`) patching fields the Python port omits. Every new MeshCore protocol feature lands in the TS decoder first. 2. **The NATS JetStream Node client is first-party.** `@nats-io/nats-core` + `@nats-io/jetstream` (v3.4.0) is maintained by Synadia with typed consumer APIs, `Nats-Msg-Id` as a standard publish option, and active development. `nats-py` (v2.15.0) is community-maintained with sparse docs and slower feature parity. The ingest pipeline (D4) is the architectural centerpiece — the queue client quality matters. diff --git a/docs/plans/next-gen/overview.md b/docs/plans/next-gen/overview.md index 4f6f43b..f29995d 100644 --- a/docs/plans/next-gen/overview.md +++ b/docs/plans/next-gen/overview.md @@ -67,7 +67,7 @@ flowchart LR | Domain | Feature | Notes | |---|---|---| -| **Ingest** | MQTT LetsMesh feeds (`packets`, `status`, `internal`) | 3 wildcard topics; only ingest path | +| **Ingest** | MQTT upload feeds (`packets`, `status`, `internal`) | 3 wildcard topics; only ingest path | | | Packet decode (meshcoredecoder lib) + 2048-entry cache | per-hex cache, FIFO eviction | | | Payload-type classification (0x00–0x0F) | normalizer cascade | | | Multi-observer aggregation | `event_observers` junction + dialect-aware upsert | @@ -165,7 +165,7 @@ These are grouped by the architectural lever that addresses them. The tags — * | W1 | **Single-threaded ingest.** `_on_message` runs on paho's network thread; one DB session per message; no batching, no backpressure. A slow DB stalls all topics. | `subscriber.py` dispatch; AGENTS.md notes "the collector is the only writer." | | W2 | **`raw_packets` write amplification.** 16 columns (incl. `raw_hex` Text + `decoded` JSON = duplicate payload storage), **9 indexes** (5 composite), one row per observer reception, no dedup. Retention capped at 2 days because of cost. | `raw_packet.py:121-138` | | W3 | **`packet_path_hops` write amplification.** One row per (reception × hop); 6-hop packet × 4 observers = 24 rows. Denormalizes 4 columns from `raw_packets`. Required a Postgres covering-index rebuild (HEAD migration) and a `window_hours` clamp as a perf band-aid. | `packet_path_hop.py`; `a59611449e2a` | -| W4 | **Dual-hash identity confusion.** `event_hash` (MD5 dedup) and `packet_hash` (LetsMesh wire hash) both appear on 6 tables with fallback logic. The evaluator "prefers event_hash" for dedup. | `raw_packet.py:25-33` | +| W4 | **Dual-hash identity confusion.** `event_hash` (MD5 dedup) and `packet_hash` (the on-air wire hash) both appear on 6 tables with fallback logic. The evaluator "prefers event_hash" for dedup. | `raw_packet.py:25-33` | | W5 | **MD5 for dedup keys.** Not a security issue but cheaply-collidable; SHA-256 truncated costs nothing. | `hash_utils.py` | | W6 | **Route health = hand-rolled materialized views.** 7 tables, 2 background cadences, full-scan candidate loads, in-Python per-day partitioning. `route_results` is 1:1 with `routes` (a cache). | `routes.py` (1,364 LOC); `route_evaluator.py` | | W7 | **`events_log` is an unbounded audit sink** that roughly doubles per-event storage. | `event_log.py` | diff --git a/docs/plans/next-gen/phasing.md b/docs/plans/next-gen/phasing.md index 1d0d091..f6cf8a4 100644 --- a/docs/plans/next-gen/phasing.md +++ b/docs/plans/next-gen/phasing.md @@ -37,6 +37,18 @@ Internally, code stays **brand-agnostic** so a future rename never cascades thro - **Class & identifier names:** descriptive — `MqttIngester`, `IngestWorker`, `DerivedStateWorker`, `WebhookWorker`, `AuthMiddleware` — never brand-prefixed (`MeshloomIngester` etc.). +**Legacy brand terms — not carried forward.** LetsMesh (a now-inactive platform whose MQTT observer +pattern + topic format this project adopted for compatibility) is dropped from the new repo's +vocabulary. Use generic terms: + +- "the on-air wire hash" (the `wire_hash` field) — not "LetsMesh hash". +- "the MQTT upload feeds" (`packets` / `status` / `internal`) — not "LetsMesh feeds". +- Descriptive fallback event types (e.g. `unhandled_packet`) — not `letsmesh_packet` / `letsmesh_status`. +- No `letsmesh_*` module or class names in the new repo. + +Code-font citations of legacy files (`letsmesh_normalizer.py`, `LetsMeshNormalizer`) remain in the +current-system analysis (overview.md, code-warts.md) as factual references to the old codebase. + ## Phase 0 — Foundations (no behavior change) - **Run the D5 fold-vs-separate benchmark first** (synthetic data, throwaway TimescaleDB — testing.md → [D5 plan](testing.md#d5-benchmark-plan-fold-vs-separate)). Its outcome decides the `raw_receptions.path_hashes` shape, so it must land **before** the DDL is authored. This was previously (mis)scheduled in Phase 2, which created a circular dependency — the schema was "frozen" in Phase 0 but the benchmark that shapes it ran two phases later (F5). diff --git a/docs/plans/next-gen/review-findings.md b/docs/plans/next-gen/review-findings.md index e9b172f..3cc08f7 100644 --- a/docs/plans/next-gen/review-findings.md +++ b/docs/plans/next-gen/review-findings.md @@ -105,7 +105,7 @@ Worker subscribe corrected to `meshcore.ingest.>`. Files: `components/ingest.md` The harness compared `event_hash` between stacks, but the old stack uses MD5 and the new uses SHA-256, so the same event has different hashes and coverage is always 0%. -**Resolution:** the hash-coverage check keys on `wire_hash` (the LetsMesh on-air hash, identical in both +**Resolution:** the hash-coverage check keys on `wire_hash` (the on-air wire hash, identical in both stacks). Files: `components/migration.md`, `testing.md`. ### F10 — Spam-rescore sweep evaluated the scoring function twice per row