web: fix meshcore message duplication regression (#825)

This commit is contained in:
l5y
2026-06-24 22:22:35 +02:00
committed by GitHub
parent 1804a8a75e
commit 554fa38f50
3 changed files with 78 additions and 9 deletions
+40
View File
@@ -1608,3 +1608,43 @@ remain green:**
- **B1** (all suites). The only contract change is the additive `nodes` publish on
message ingest (a new SSE event, documented in `CONTRACTS.md`); no POST/GET shape
changes, so **C2** and the Python suite are unaffected.
---
## Bugfix: MeshCore cross-ingestor dedup keys on the stable channel name
A single physical MeshCore channel message heard by two ingestors that store the
same logical channel at **different local channel-slot indices** was stored twice.
The per-receiver `channel` index is not stable across ingestors (e.g. `#bot` sits
at slot 4 on one device and slot 6 on another), yet it fed both the ingestor
fingerprint discriminator (`c<N>` → two different `messages.id` values) and the
#756 web content-dedup SELECT (`AND channel = ?` → no match), so neither dedup
layer collapsed the duplicate. Fix (web-only, no wire change): the content-dedup
matches on the sender-stable `channel_name` (NULL-safe) instead of the local
`channel` index, so the safety net collapses the duplicate at the system of record
regardless of differing ids/slots. Strengthens **C5**.
### MD-A1 — Same message on different local channel slots collapses to one row
```bash
( cd web && bundle exec rspec spec/data_processing_spec.rb -e "meshcore content dedup" )
```
**Expected:** pass, including "collapses the same meshcore channel message heard on
different local channel indices": two meshcore messages with identical `from_id` /
`to_id` / `text` / in-window `rx_time` and the **same `channel_name`** ("#bot") but
**different `channel` indices** (4 vs 6) and different ids collapse to a **single**
stored row. Companion examples still hold: messages with a **different
`channel_name`** are kept separate (the legitimate distinct-channel case), and
different `text` / `to_id` / beyond-window `rx_time` stay separate.
### MD-R1 — Regression: prior acceptance still holds
```bash
( cd web && bundle exec rspec ) && ( cd web && npm test )
( . .venv/bin/activate && pytest -q tests/ )
```
**Expected:** all green. At risk and required to remain green: **C5** (cross-ingestor
dedup by id — now strengthened), the other #756 content-dedup examples (window
inclusivity, different text/recipient), and **B1**. The pre-existing "does not
collapse two meshcore messages on different channels" example is **updated** to use
different channel *names* (the stable identifier) rather than different local
indices — it is updated, not removed. No POST/GET/event contract change and no
ingestor change, so **C2**, `CONTRACTS.md`, and the Python suite are unaffected.
@@ -225,23 +225,28 @@ module PotatoMesh
# ``db.transaction(:immediate)`` is a future tightening if the race
# is ever observed in production.
if protocol == "meshcore" && from_id && channel_index && text && !text.to_s.empty?
# ``channel = ?`` matches the ``channel_index`` bind cleanly
# because the guard above rejects nil; ``to_id`` may legitimately
# be nil (rare meshcore fallback), so it keeps ``IS ?`` for a
# NULL-safe compare.
# Match on the sender-stable ``channel_name`` (NULL-safe ``IS ?``)
# rather than the per-receiver ``channel`` slot index. Two
# ingestors store the same logical channel at different local
# indices (e.g. ``#bot`` at slot 4 on one device, 6 on another), so
# keying the dedup on the index lets the same physical transmission
# through twice — the reported duplication. The channel *name* is
# carried in the message text/contact roster identically across
# receivers, so it is the stable discriminator. ``to_id`` is also
# ``IS ?`` (rare meshcore nil fallback).
duplicate_id = db.get_first_value(
<<~SQL,
SELECT id FROM messages
WHERE protocol = 'meshcore'
AND from_id = ?
AND to_id IS ?
AND channel = ?
AND channel_name IS ?
AND text = ?
AND rx_time BETWEEN ? AND ?
AND id != ?
LIMIT 1
SQL
[from_id, to_id, channel_index, text,
[from_id, to_id, channel_name, text,
rx_time - MESHCORE_CONTENT_DEDUP_WINDOW_SECONDS,
rx_time + MESHCORE_CONTENT_DEDUP_WINDOW_SECONDS, msg_id],
)
+27 -3
View File
@@ -1356,15 +1356,39 @@ RSpec.describe PotatoMesh::App::DataProcessing do
db&.close
end
it "does not collapse two meshcore messages on different channels" do
it "does not collapse two meshcore messages on different named channels" do
# Genuinely distinct channels are now distinguished by the stable
# channel *name*, not the per-receiver local slot index.
db = open_db
meshcore_harness.insert_message(db, base_message.merge("id" => 1_000_005, "channel" => 5))
meshcore_harness.insert_message(db, base_message.merge("id" => 1_000_006, "channel" => 6))
meshcore_harness.insert_message(
db, base_message.merge("id" => 1_000_005, "channel" => 5, "channel_name" => "#alpha"),
)
meshcore_harness.insert_message(
db, base_message.merge("id" => 1_000_006, "channel" => 6, "channel_name" => "#beta"),
)
expect(message_count(db)).to eq(2)
ensure
db&.close
end
it "collapses the same meshcore channel message heard on different local channel indices" do
# One physical #bot transmission heard by two ingestors that store it at
# different LOCAL channel slots (4 vs 6) — so each computes a different
# fingerprint id. The channel *name* ("#bot") is identical across
# receivers, so the content-dedup must collapse it to a single row.
# Regression for the cross-ingestor duplication in the bug report.
db = open_db
meshcore_harness.insert_message(
db, base_message.merge("id" => 1_000_201, "channel" => 4, "channel_name" => "#bot"),
)
meshcore_harness.insert_message(
db, base_message.merge("id" => 1_000_202, "channel" => 6, "channel_name" => "#bot"),
)
expect(message_count(db)).to eq(1)
ensure
db&.close
end
it "does not collapse two meshcore messages with different text" do
db = open_db
meshcore_harness.insert_message(db, base_message.merge("id" => 1_000_007, "text" => "first"))