Add a "Grayscale map" option to the map settings popover that desaturates
the base tiles so colored route lines and node markers stand out. The CSS
filter is scoped to Leaflet's tile pane only, so overlays stay in color,
and it works with any selected tileset. The setting persists in
mapLayerSettings (localStorage).
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Derive more neighbor edges: hash-size-aware paths, anchors, confidence tiers
Rework the meshcore_all_neighbor_edges materialized view (migration 008)
so the map's "show all neighbors" layer surfaces many more, and
higher-quality, edges:
- Parse routing paths at the packet's real hash_size (1/2/3 bytes per hop)
instead of assuming 1-byte hops, so extended-hash packets are no longer
mis-sliced, and wide (2-/3-byte) prefixes resolve uniquely.
- Add anchored edges: resolve a hop against a fully-known endpoint (the
advert originator or the uploading gateway) when exactly one in-region
repeater of the right prefix sits within a plausible LoRa-hop distance.
- Tag every edge with a derivation `method` and a `confidence` score, and
aggregate undirected edges across observations.
Frontend: expose method/confidence on the all-neighbors API and color edges
by method / fade by confidence. The old "only MQTT neighbors" checkbox
becomes the top notch of a single confidence selector (MQTT-direct only ->
high -> standard -> all).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Tighten neighbor edges: 100km anchor cap + region-unique anchor binds
Two precision fixes after reviewing cross-metro over-linking:
- Cap anchored edges at 100 km (was 206 km). Anchors bind a hop to the
single nearby repeater of a prefix, so a wrong bind is indistinguishable
from a real hop -> a loose cap admitted ~200 km false links. path-uniq
and direct edges (observed consecutive hops / zero-hop adverts) keep the
206 km backstop.
- Require a region-unique prefix for anchor binds. Without it, a 1-byte hop
whose true (unlocated) forwarder is out of set mis-binds to a coincidental
same-prefix repeater that merely happens to be the lone nearby candidate
(~22% of 1-byte anchors). Region-uniqueness removes the proximity guess;
the 100 km check becomes a sanity confirm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Demote anchored edges to substandard confidence
Anchored edges are geographically inferred (resolve a hop to the region-unique
repeater near a known endpoint), not observed consecutive hops, so they are less
trustworthy than any path-uniq edge. Re-rank confidence to:
direct 1.0 > path-uniq-3b 0.8 > path-uniq-2b 0.6 > anchor 0.45 > path-uniq-1b 0.4.
Anchors now fall below the 0.5 "Standard" default (hidden by default, just above
the noisy 1-byte tier). Add an "Include anchored (lower confidence)" UI tier at
0.45 so they can be opted into without the 1-byte noise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Unify neighbor inference: global confidence setting, hover + node page
- Move the neighbor-confidence preference from per-map-layer settings to global
Settings (ConfigContext); the map all-neighbors layer, map hover, and node
page all read it. Default "Standard" (0.5).
- Map hover and the node detail page now derive neighbors from the unified graph
(meshcore_all_neighbor_edges) via a new getMeshcoreNodeAllEdges + a mode=all
option on the node neighbors route/hook, so they match "show all neighbors".
- Node page shows direct and inferred neighbors in one list; inferred cards carry
an "Inferred · <method>" + confidence indicator instead of direction arrows.
- Color map hover lines by derivation method (purple/blue/teal), fade by confidence.
- Remove the "Manage Channel Keys" button from the Settings popover (still reachable
via the + on the messages page).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Keep neighbor lines when either endpoint is in view
Both the server bbox filter (getAllNodeNeighbors) and the client filter
(AllNeighborLines) required BOTH endpoints inside the viewport, so any line
with one end off-screen vanished when zooming in. Switch both to "either
endpoint in view": the server returns an edge if source OR target falls in the
bbox, and the client keeps it if either endpoint is a visible node. Each edge
carries both endpoints' coordinates, so the off-screen end still draws.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node search now scans/merges meshcore_adverts_latest once and fans each row
out to matching query branches via arrayJoin + a window function, instead of
one UNION ALL branch per query each re-merging the full AggregatingMergeTree.
Chat message queries push the region filter onto the raw table's stored
`region` column before the GROUP BY (instead of a post-aggregation arrayExists
over origin_path_info), eliminating the multi-second full-table scans.
Grafana: rewrite the nodes-over-time panel to drop the rolling self-join, and
fix the messages-by-channel-hash panel which referenced a nonexistent
`regions` column (ClickHouse error code 47).
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operators can opt a node out of the public-facing surfaces by putting an
opt-out emoji in the node name. Hidden nodes are removed from the map,
neighbor edges/lists, search, and their own detail page (server-side
ClickHouse filters), and chat messages from a hidden sender are dropped
client-side after decryption. Matching keys on the base codepoint so the
variation-selector form (⛔️) is caught too.
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ingest: batch ClickHouse inserts to stop MQTT flapping & packet loss
The meshcore handler did a synchronous per-message ClickHouse insert on
paho's single inbound goroutine. At ~86ms/insert (single-row inserts +
async_insert wait + materialized views) the goroutine couldn't keep up
with the high-volume letsmesh feed, so it stalled past PingTimeout and
paho declared "pingresp not received" and reconnected — ~847 cycles in
19.5h, ~45% downtime, ~50% of letsmesh packets lost. The low-volume
davekeogh broker never saturated the goroutine and was unaffected.
Decouple receipt from insertion: the handler now enqueues decoded rows
onto a buffered channel and a single background writer flushes them to
meshcore_packets in batched native inserts (every MESHCORE_BATCH_FLUSH_
SECONDS or MESHCORE_BATCH_MAX_ROWS rows). The inbound goroutine never
blocks, so PINGRESP is always processed in time.
- New batch writer with env-configurable flush interval / max rows /
buffer size (MESHCORE_BATCH_* ), wired in docker-compose.
- Drop server-side async_insert (redundant once we batch app-side).
- Bump PingTimeout 10s -> 20s (env MQTT_PING_TIMEOUT_SECONDS) for margin
against Cloudflare WebSocket buffering jitter.
- Enqueue is non-blocking; rows are dropped+counted only if the buffer
fills (ClickHouse unavailable). A failed batch is dropped and retried
by the next flush (native blocks commit atomically).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ingest: make MQTT KeepAlive configurable (MQTT_KEEPALIVE_SECONDS)
As a near-silent subscriber, paho emits a PINGREQ roughly every KeepAlive
seconds; lowering it sends client->server frames more often to keep the
Cloudflare-proxied WebSocket path warm in both directions, a lever for the
residual mid-stream "pingresp not received" stalls on the letsmesh broker.
Default unchanged (30s); wired through docker-compose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ingest: add configurable MQTT write timeout (MQTT_WRITE_TIMEOUT_SECONDS)
Bounds PINGREQ/SUBSCRIBE writes so a stalled write through the Cloudflare
WebSocket proxy can't hang the client. Default 0 (paho's existing no-timeout
behavior); wired through docker-compose. Recommended ~20s when behind a
buffering reverse proxy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the Location History coordinate table with an embedded Leaflet
map (new LocationHistoryMap component): one circle marker per position
with timestamp/coord popups, the most-recent point highlighted, and a
subtle polyline connecting points chronologically, auto-fit to bounds.
Cap Recent Adverts at the 5 most recent with a "Show more"/"Show less"
toggle and a count in the subtitle.
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The map page slowed the browser at high node counts (~5k) because every
marker eagerly rendered its popover to an HTML string at creation, and the
visual-update effects re-rendered every marker's icon and popup on each
selection change. Hovering a meshcore node (the default type) re-rendered
all markers.
Bind popups lazily so PopupContent is only rendered when a popup actually
opens, drop the now-unnecessary popup setContent calls, and re-skin only the
markers whose selected state changed instead of the whole set.
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The letsmesh broker was migrated behind Cloudflare and changed its topic
layout on 2026-06-02, which left prod's MQTT client in a zombie state:
connected per paho's IsConnected() (so the 30s monitor never rebuilt it) but
receiving zero messages, because the subscription was established only once
after the initial connect and never re-applied on paho auto-reconnects. Result:
12 days of silently missing letsmesh ingestion while davekeogh masked the loss.
Make reconnection robust instead of relying on broker-side session persistence:
- Subscribe inside the OnConnect handler so every (re)connect — including paho
auto-reconnects — restores delivery. Use CleanSession(true)+ResumeSubs(false)
so we never depend on the broker remembering our session.
- Add a per-broker data-staleness watchdog: a broker that reports connected but
delivers no messages for MQTT_STALE_AFTER_SECONDS (default 300) is treated as a
zombie and force-rebuilt (disconnect + fresh connect/subscribe). This catches
exactly the failure IsConnected() misses.
- Reduce the external monitor to that watchdog role; transient drops are left to
paho auto-reconnect rather than racing it with a brand-new client.
- Stable per-broker client IDs (by index) and pre-sized MQTTClients slice so
indices stay aligned when an earlier broker fails; guard BrokerStatus/lastActivity
with a mutex; promote connect/subscribe logs to Info for visibility.
Adds unit tests for the watchdog and env parsing; documents the new env var.
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
