CMD_SEND_RAW_PACKET bypasses sendFlood()'s _tables->hasSeen() self-mark.
The firmware seen-table is 160 entries in RAM and rolls over in minutes
on a busy mesh, so a resend issued after the original hash got evicted
comes back via repeater echo, fails hasSeen (entry gone), and is
delivered as RESP_CODE_CHANNEL_MSG_RECV. Result: the user's own resend
appears as an "incoming" message from themselves a few minutes later.
Detection: in _on_channel_message compute the expected pkt_payload from
the event's (channel_secret, sender_timestamp, txt_type, raw_text) and
ask the DB if any own row already has that exact hash. If yes, treat as
self-echo and return — no DB insert, no SocketIO emit. Index idx_cm_pkt
keeps the lookup cheap. Guarded with try/except so any detection failure
falls through to normal handling — we never want a check bug to drop
real inbound messages.
Documented as a known limitation in reference_meshcore_raw_packet.md;
this lifts that caveat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PR #2 of 5. Builds the full GRP_TXT wire bytes (header + transport_codes if
scoped + path_len + encrypted payload) from the ts+0 pkt_payload guess and
stores it in channel_messages.raw_packet right after the send. When echo
correlation later identifies the actual pkt_payload (potentially using a
different ±dt candidate due to host/firmware clock drift), the raw_packet is
rebuilt from the actual one so a future resend matches the original packet
hash and dedupes at the repeaters.
Transport-scope codes are computed in Python via HMAC-SHA256(scope_key,
payload_type||payload)[:2], mirroring TransportKey::calcTransportCode in
MeshCore Core (including the 0x0000/0xFFFF reservations).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stores the full hex packet snapshot (header+transport_codes+path_len+payload)
captured at send time, enabling future "Resend (same packet hash)" feature
that lets repeaters deduplicate via Packet::calculatePacketHash while still
forwarding to nodes that didn't hear the original.
NULL for received and pre-migration rows (resend button stays disabled there).
PR #1 of 5; subsequent PRs wire up send-time capture, backend endpoint, echo
list merge, and UI.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SQLite DELETE marks pages free but doesn't shrink the file, so the
new retention job would keep DBs at their bloated size forever without
a follow-up VACUUM. Add db.vacuum() that runs PRAGMA-free VACUUM and
reports size_before/size_after/elapsed so callers can surface results.
The retention job now calls vacuum() automatically when it deleted at
least 1000 rows. Threshold avoids the multi-second VACUUM cost on quiet
days. Failure is logged, not raised — a missed VACUUM never crashes
the scheduler.
Power-user override: new "Optimize now" button in the Database Backup
modal triggers VACUUM on demand via POST /api/db/vacuum, alongside a
GET /api/db/size that drives the live "Current size" label. This way
users don't have to wait until 03:30 to reclaim space after the first
big retention pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The retention/cleanup scheduler was throwing "Working outside of application
context" on every boot because init_*_schedule() and the APScheduler jobs
themselves call api.get_*_settings(), which reads current_app.db. Pass the
Flask app into archiver.manager via set_flask_app() and decorate every
scheduled job with @_with_app_context so the context is active when the
worker thread runs. Wrap the init calls in main.py in app.app_context() too.
Extend cleanup_old_messages() to also delete from echoes, paths and acks
(the diagnostic tables — 191k echo rows account for the bulk of the 85 MB
DB). Each diagnostic table has its own retention window, defaulting to
min(days, 30) so debug data is purged on a tighter schedule than
user-visible messages.
Switch RETENTION_DEFAULTS to enabled=True with 90/90/60/30 days for
channel_messages/DMs/adverts/diagnostics; user opted in explicitly. First
run scheduled for 03:30 local. SQLite DELETE only marks pages free —
file size won't shrink until a manual VACUUM is run separately.
Archive job stopped working the day the device was renamed to include an
emoji ("MarWoj 💡"): the meshcore library strips non-ASCII when writing
the .msgs file, but our archiver still built /data/{device_name}.msgs and
hit ENOENT every night at 00:00. Add a tolerant fallback that globs the
data dir for a single non-archive .msgs file when the expected path is
missing — mirroring how migrate_v1 already handles this.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a Settings > Analyzer tab letting users CRUD custom MeshCore Analyzer
services with a star-toggle default and inline disabled switch. The chart
icon under each group-chat message now resolves at click time: built-in
Letsmesh when no enabled customs, the default when set, or a chooser
modal otherwise. Backend stops shipping the prebuilt analyzer_url and
emits packet_hash instead — the frontend substitutes {packetHash} in the
chosen URL template.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live dm_delivered_info already carried the correct hash_size, but the
DB row only kept delivery_path. After a reload the API filled in
path_hash_size from the incoming path_len column (NULL for outgoing
DMs → default 1), so 2-byte routes were re-rendered as single-byte
hops.
Added a delivery_path_hash_size column (auto-migrated, defaults to 1)
that update_dm_delivery_info now stores alongside the delivery path,
populated from the same hash_size already known by each delivery path
(retry ctx, PATH event, delayed contact backfill). /api/dm/messages
returns the new field; dm.js prefers it over path_hash_size when
rendering the Route line, falling back to the old field for legacy
rows.
Channels in the sidebar and mobile dropdown now sort by most recent
message first, with favorited channels pinned above non-favorites.
Reordering is push-driven via the existing new_message socket event:
the affected item is moved to the top of its tier in the DOM, no full
re-render. Favorites are toggled via a star icon in Manage Channels
and persisted in read_status.is_favorite for cross-device sync.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduces the SQLite-backed region registry and channel->region mapping
that will drive the per-channel flood-scope feature. No UI or device
wiring yet; those land in subsequent PRs.
- schema.sql: new `regions` and `channel_scopes` tables + partial index
on the default flag.
- database.py: CRUD helpers for regions (create/list/get/delete/default)
and channel_scopes (set/get/bulk-load) with ON DELETE CASCADE.
- app/meshcore/regions.py: pure helpers for SHA256('#'+name)[:16] key
derivation and firmware-compatible name validation (mirrors the
`RegionMap::is_name_char` rule `c in {-,$,#} or c>='0' or c>='A'`).
- tests/test_regions.py: known SHA256 vectors, validator coverage
(incl. the firmware quirk that `_` and other 0x5B-0x60 chars are
admitted), and CRUD + cascade integration tests.
hard_delete_contact() failed with FOREIGN KEY constraint when the
contact had a row in ignored_contacts or blocked_contacts, since those
FKs lacked ON DELETE CASCADE. Delete dependent rows first in the same
transaction; also update schema for new deployments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stage 1 of path_hash_mode support. The critical bug in _on_rx_log_data
treated the raw path_len byte as a direct byte count, which breaks with
mode>0 (e.g. mode=1, 0 hops → path_len=0x40=64, reading 64 bytes of
non-existent path data). Now properly decodes the encoded path_len byte
into hop_count, hash_size, and path_byte_len.
Changes:
- Add decode_path_len() utility for MeshCore v1.14+ path_len encoding
- Fix _on_rx_log_data binary parsing to use decoded path length
- Pass hash_size through _process_echo → DB insert → SocketIO emission
- Add hash_size column to echoes table (schema + migration)
- Update insert_echo() to store hash_size (default 1 for backward compat)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When both ACK and PATH_UPDATE fire for FLOOD delivery, _on_ack may
store empty path before PATH_UPDATE can provide the discovered route.
Now _on_path_update also checks for recently-delivered DMs with empty
delivery_path and backfills with the discovered path from the event.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Show retry progress in DM message bubble via WebSocket:
- "attempt X/Y" counter updates in real-time during retries
- Failed icon (✗) when all retries exhausted
- Delivery info persisted in DB (attempt number, path used)
Backend: emit dm_retry_status/dm_retry_failed socket events,
store delivery_attempt/delivery_path in direct_messages table.
Frontend: socket listeners update status icon and counter,
delivered tooltip shows attempt info and path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backup filenames now derive from the active DB stem (e.g. mc_9cebbd27.2026-03-24.db).
Listing and cleanup glob *.db so existing mc-webui.* backups remain visible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace file-based .read_status.json with DB-backed read_status table.
One-time migration imports existing data at startup. The read_status.py
module keeps the same public API so route handlers need no changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DB filename changes from {device_name}.db to mc_{pubkey[:8]}.db,
making it stable across device renames and preparing for multi-device support.
Existing databases are auto-migrated at startup by probing the device table.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New `contact_paths` table for storing multiple user-configured paths per contact
- New `no_auto_flood` column on contacts to prevent automatic DIRECT→FLOOD reset
- Path rotation during DM retry: cycles through configured paths before optional flood fallback
- REST API for path CRUD, reorder, reset-to-flood, repeater listing
- Path management UI in Contact Info modal: add/delete/reorder paths, repeater picker with uniqueness warnings, hash size selector (1B/2B/3B)
- "No Flood" per-contact toggle in modal footer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All settings (protected_contacts, cleanup_settings, retention_settings,
manual_add_contacts) moved from .webui_settings.json file to SQLite database.
Startup migration auto-imports existing file and renames it to .json.bak.
Added safeguard in _on_new_contact: if firmware fires NEW_CONTACT for a
contact already on the device, skip pending and log a warning. Also added
diagnostic logging showing previous DB state (source, protected) when
contacts reappear as pending.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Delete button now sends public_key instead of name to avoid matching
wrong contacts when multiple share similar names.
2. _on_advertisement adds cache-only contacts to mc.pending_contacts when
manual approval is enabled, so they appear in the pending list after
advertising (even if meshcore fires ADVERTISEMENT instead of NEW_CONTACT).
3. Added Delete button for cache-only contacts with dedicated
/api/contacts/cached/delete endpoint and hard_delete_contact DB method.
4. approve_contact/reject_contact now handle DB-only pending contacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of reloading the entire message list when echo data arrives,
now updates only the affected message elements in the DOM:
- Add data-msg-id attribute to message wrappers for targeted lookup
- Add GET /api/messages/<id>/meta endpoint returning metadata for a
single message (computes pkt_payload, looks up echoes, analyzer URL)
- Replace loadMessages() echo handler with refreshMessagesMeta() that
finds messages missing metadata and updates them individually
- Fix path_len=0 treated as falsy (use ?? instead of ||)
Flow: message appears instantly via WebSocket (with SNR + hops), then
~2s later echo data triggers targeted meta fetch → route info and
analyzer button appear smoothly without any chat window reload.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New blocked_names table for blocking bots without known public_key
- get_blocked_contact_names() returns union of pubkey-blocked + name-blocked
- POST /api/contacts/block-name endpoint for name-based blocking
- GET /api/contacts/blocked-names-list for management UI
- Block button always visible in chat (falls back to name-based block)
- Blocked Names section shown in Existing Contacts Blocked filter
- CSS breakpoint for icon-only buttons: 768px → 428px (iPhone-sized)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _on_new_contact() in manual mode: upsert to DB as cache (source='advert')
so contacts appear in @mentions and Cache filter before approval
- _on_advertisement(): check mc.pending_contacts for name/metadata fallback
- get_pending_contacts(): include last_advert in response
- /api/contacts/cached: return numeric last_advert timestamp
- contacts.js: fix adv_lat/adv_lon field names (was c.lat/c.lon),
use last_advert timestamp instead of last_seen datetime string
- upsert_contact: source priority — never downgrade 'device' to 'advert'
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- get_contacts_with_last_seen() reads from mc.contacts (device firmware)
instead of DB, so /api/contacts/detailed returns only device contacts
- _sync_contacts_to_db() now bidirectional: downgrades stale 'device'
contacts to 'advert' (cache-only) when not on device anymore
- delete_contact() sets source='advert' (cache) instead of 'deleted',
keeping contacts visible in @mentions and cache filter
- get_contacts() returns all contacts (no 'deleted' filter needed)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- delete_contact() now sets source='deleted' instead of SQL DELETE
- get_contacts() filters out deleted contacts (hidden from UI)
- upsert_contact() on re-add overwrites source, auto-undeleting
- DM FK references stay intact, no more orphaned messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Stop deleting contacts from DB on device removal (preserves DM history)
- Filter NULL contact_pubkey from DM conversations list
- Match outgoing DMs by contact name in raw_json during relinking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for DM sending after contact delete/re-add:
1. approve_contact() now calls ensure_contacts() to refresh mc.contacts
so send_dm can find newly added contacts immediately
2. cli.send_dm() falls back to DB name lookup when mc.contacts misses,
preventing the contact name from being passed as a pubkey string
3. approve_contact() re-links orphaned DMs (NULL contact_pubkey from
ON DELETE SET NULL) back to the re-added contact
New DB methods: get_contact_by_name(), relink_orphaned_dms()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a DM arrives with only pubkey_prefix (short hex) and the sender
is not in mc.contacts, fall back to DB prefix lookup to get the full
64-char public key. Prevents ghost contact entries and "Unknown" DM
conversations.
Also adds get_contact_by_prefix() database helper.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix ACK handler bug: read 'code' field instead of 'expected_ack'
- Add DM retry (up to 3 attempts) with same timestamp for receiver dedup
- Add receiver-side dedup in _on_dm_received() (sender_timestamp or time-window)
- Add PATH_UPDATE as backup delivery signal for flood DMs
- Track pending acks with dm_id for proper ACK→DM linkage
- Return dm_id and expected_ack from POST /dm/messages API
- Add find_dm_duplicate() and get_dm_by_id() database helpers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Subscribe to RX_LOG_DATA events to capture repeated radio packets
- Parse GRP_TXT (0x05) payload to extract pkt_payload and path
- Classify echoes as sent (pending echo correlation) or incoming
- Register pending echo when sending channel messages for pkt_payload capture
- Add update_message_pkt_payload() DB method for sent message correlation
- Return echo_paths/echo_snrs for ALL messages (not just own) in GET /messages
- Frontend: build paths from echo_paths for incoming message route display
- Emit SocketIO 'echo' event for real-time badge updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add daily retention job that deletes old channel messages, DMs, and
advertisements based on configurable age threshold
- Add GET/POST /api/retention-settings endpoints
- Extend cleanup_old_messages() to optionally include DMs and adverts
- Wire up APScheduler in create_app() (also enables existing archiving
and contact cleanup schedulers that were never started in v2)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add GET /api/advertisements with optional pubkey filter and limit.
Enriches results with contact name lookup from cache.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Convert bytes to hex string for expected_ack and pkt_payload via _to_str()
- Support pubkey prefix matching in get_dm_messages() (LIKE for short keys)
- Fixes "Object of type bytes is not JSON serializable" error on DM view
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration now imports all archive files (oldest first) in addition to the
live .msgs file, with deduplication. Archives endpoint and message history
now query SQLite by date instead of reading .msgs files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add _detect_serial_port() to DeviceManager — resolves 'auto' to
actual device via /dev/serial/by-id with common path fallbacks
- Make channel_idx optional in get_channel_messages() so status and
channel-updates endpoints can query across all channels
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pkt_payload column to direct_messages table for stable packet
hash generation and Analyzer URL linking
- Update insert_direct_message() and DeviceManager to store pkt_payload
- Add test for DM pkt_payload storage (43 tests pass)
- Update watchdog to monitor only mc-webui (meshcore-bridge removed)
- USB reset trigger now fires for mc-webui container failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
MarekWo