* Derive more neighbor edges: hash-size-aware paths, anchors, confidence tiers
Rework the meshcore_all_neighbor_edges materialized view (migration 008)
so the map's "show all neighbors" layer surfaces many more, and
higher-quality, edges:
- Parse routing paths at the packet's real hash_size (1/2/3 bytes per hop)
instead of assuming 1-byte hops, so extended-hash packets are no longer
mis-sliced, and wide (2-/3-byte) prefixes resolve uniquely.
- Add anchored edges: resolve a hop against a fully-known endpoint (the
advert originator or the uploading gateway) when exactly one in-region
repeater of the right prefix sits within a plausible LoRa-hop distance.
- Tag every edge with a derivation `method` and a `confidence` score, and
aggregate undirected edges across observations.
Frontend: expose method/confidence on the all-neighbors API and color edges
by method / fade by confidence. The old "only MQTT neighbors" checkbox
becomes the top notch of a single confidence selector (MQTT-direct only ->
high -> standard -> all).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Tighten neighbor edges: 100km anchor cap + region-unique anchor binds
Two precision fixes after reviewing cross-metro over-linking:
- Cap anchored edges at 100 km (was 206 km). Anchors bind a hop to the
single nearby repeater of a prefix, so a wrong bind is indistinguishable
from a real hop -> a loose cap admitted ~200 km false links. path-uniq
and direct edges (observed consecutive hops / zero-hop adverts) keep the
206 km backstop.
- Require a region-unique prefix for anchor binds. Without it, a 1-byte hop
whose true (unlocated) forwarder is out of set mis-binds to a coincidental
same-prefix repeater that merely happens to be the lone nearby candidate
(~22% of 1-byte anchors). Region-uniqueness removes the proximity guess;
the 100 km check becomes a sanity confirm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Demote anchored edges to substandard confidence
Anchored edges are geographically inferred (resolve a hop to the region-unique
repeater near a known endpoint), not observed consecutive hops, so they are less
trustworthy than any path-uniq edge. Re-rank confidence to:
direct 1.0 > path-uniq-3b 0.8 > path-uniq-2b 0.6 > anchor 0.45 > path-uniq-1b 0.4.
Anchors now fall below the 0.5 "Standard" default (hidden by default, just above
the noisy 1-byte tier). Add an "Include anchored (lower confidence)" UI tier at
0.45 so they can be opted into without the 1-byte noise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Unify neighbor inference: global confidence setting, hover + node page
- Move the neighbor-confidence preference from per-map-layer settings to global
Settings (ConfigContext); the map all-neighbors layer, map hover, and node
page all read it. Default "Standard" (0.5).
- Map hover and the node detail page now derive neighbors from the unified graph
(meshcore_all_neighbor_edges) via a new getMeshcoreNodeAllEdges + a mode=all
option on the node neighbors route/hook, so they match "show all neighbors".
- Node page shows direct and inferred neighbors in one list; inferred cards carry
an "Inferred · <method>" + confidence indicator instead of direction arrows.
- Color map hover lines by derivation method (purple/blue/teal), fade by confidence.
- Remove the "Manage Channel Keys" button from the Settings popover (still reachable
via the + on the messages page).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Keep neighbor lines when either endpoint is in view
Both the server bbox filter (getAllNodeNeighbors) and the client filter
(AllNeighborLines) required BOTH endpoints inside the viewport, so any line
with one end off-screen vanished when zooming in. Switch both to "either
endpoint in view": the server returns an edge if source OR target falls in the
bbox, and the client keeps it if either endpoint is a visible node. Each edge
carries both endpoints' coordinates, so the off-screen end still draws.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MeshCore path_length byte encodes hop_count in its low 6 bits and a
hash_size_code in its high 2 bits (0/1/2 -> 1/2/3 bytes per hop), and
transport route types (TRANSPORT_FLOOD/TRANSPORT_DIRECT) insert a 4-byte
transport_codes field between the header and path_length. The packet
decoder previously assumed every hop was a single byte and that
path_length always sat at byte 2, so it only handled 1-byte-hash,
non-transport packets; anything else decoded to an over-long path and an
empty payload.
Migration 007 reworks the meshcore_packets read-time aliases to honor the
transport_codes offset and compute the path as hop_count * hash_size
bytes, and exposes hop_count / hash_size_code / hash_size (bytes per hop)
as columns. payload, path and packet_hash now decode correctly for every
route type and hash size; the adverts and public-channel derived tables
are rebuilt from the corrected decode (invalid hash_size_code 3 packets
are skipped per spec).
hash_size is carried through the chat and advert APIs so the path
visualization splits a path into hops of the correct width
(pathUtils/PathVisualization), instead of always slicing one byte per hop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-request argMax/GROUP-BY views with insert-triggered
(incremental) materialized views so map node positions, node search, and
public-channel chat read pre-aggregated state instead of re-scanning all of
meshcore_packets on every query.
- 005: meshcore_adverts_latest_state (AggregatingMergeTree of argMaxState/
min/maxState) + incremental MV + backfill; meshcore_adverts_latest becomes a
-Merge view with the identical column contract. Node search reads it directly;
map (unified_latest_nodeinfo) is unchanged.
- 006: meshcore_public_channel_messages_raw, a decoded payload_type=5 MergeTree
keyed (channel_hash, ingest_timestamp); chat dedups by message_id at read time
over a timestamp-bounded scan. Streaming/pagination push channel+cursor onto
the primary key.
- Neighbor-edge MVs stay hourly REFRESH (they read the preserved view).
Verified against full prod data (14.5M rows): exact parity (0 mismatches) and
5-9x faster reads with no regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the hand-curated region groups with data-driven ones and make the
region_groups ClickHouse table the single source of truth.
- scripts/generate-region-groups.ts: offline generator — clusters regions by
cross-region packet co-occurrence (min-share single-linkage) at two levels
(broad "region" + tight "metro"), names clusters via `claude -p`, reconciles
codes by member overlap so permalinks stay stable, and emits the region_groups
seed. Migration 004 reseeded with the resulting 39 groups.
- Groups are DB-sourced: getRegionGroups() (cached) feeds /api/regions and the
dropdown/labels; filtering resolves a selector in SQL to a region
(region = 'X') or a group (region IN / hasAny ... SELECT region_code FROM
region_groups WHERE group_code = ...). No hardcoded membership in TS;
resolveSelector removed.
- Drop the TS<->SQL parity script (no membership left to sync); regionSql and
the migration ALIAS are kept in sync by hand.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace hardcoded (broker, topic) region slugs with uppercase IATA codes
derived from the meshcore/{IATA} base topic, discovered dynamically from
data (adding a region needs no code change). Adds region groups, Grafana
region/group filtering, and fixes the neighbor graph.
- regions.ts: single source of truth — regionFromTopic / normalizeRegion /
regionSql / resolveSelector / selectorLabel. Legacy slugs (seattle->SEA)
and bare meshcore + meshcore/salish -> SEA still resolve.
- regionGroups.ts + seeded region_groups table: PNW/CAL/DEU/POL.
- migration 004: region ALIAS column on meshcore_packets; 001 views expose
region / regions[]; reworked neighbor MV (region-scoped, no cross-region
edges, drops implausible >150km and (0,0) edges); scheduled meshcore_regions MV.
- API/streaming/actions resolve selectors; stream routes drop the hardcoded
region allow-lists; map node query excludes (0,0) sentinel nodes.
- Dynamic region/group dropdowns (useRegions/RegionSelect); /api/regions.
- Grafana: cascading $region / $region_group template vars + panel filters.
- region-parity.ts (npm run check:regions) guards TS<->SQL drift.
- nix dev shell (flake.nix, Node 24).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Point the seattle region at the letsmesh broker (wss://mqtt-us-v1.letsmesh.net:443,
topic meshcore/SEA) where Seattle traffic now lives.
- Fix a pre-existing bug in the path-edge extraction: `path` is a hex string of
1-byte hop prefixes, so use substring(path, 2*i-1, 2) instead of
hex(substring(path, i, 1)) (which re-hexed a single hex char and never matched
the 2-char repeater prefixes -> path edges were always empty). Seattle now yields
path edges again.
Verified on a full prod snapshot: the MV-backed "show all neighbors" query drops
from ~1.6s / 145M rows / 11.8 GiB to ~1ms / 108 rows / 3.8 KiB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two slow neighbor queries are converted to read precomputed tables that an
hourly REFRESH EVERY 1 HOUR materialized view maintains, instead of re-aggregating
meshcore_packets per request:
- meshcore_all_neighbor_edges: the global per-region edge graph (direct path_len=0
adverts + repeater-prefix path edges) with endpoint details. getAllNodeNeighbors
now filters it by region + bbox + lastSeen + has_location.
- meshcore_node_direct_neighbors: per-node direct adjacency (both directions) with
neighbor details. getMeshcoreNodeNeighbors now filters it by node_public_key.
Also add the meshcore/SEA topic to the seattle region. Validated on a clean local
stack: migration 001->003 applies, both refreshable MVs create + refresh + populate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vendor the ingest service under ingest/ and move the web app under meshexplorer/.
The ingest builds the meshcoreingest daemon and the goose migration runner,
applies the meshcore ClickHouse schema (packets, adverts, unified node view),
and loads its MQTT broker list and ClickHouse settings entirely from environment
variables (MQTT_BROKERS as a JSON array, CLICKHOUSE_*). No credentials are baked
into the source.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Alex Vanderpot