mirror of
https://github.com/ajvpot/meshexplorer.git
synced 2026-07-29 04:42:23 +02:00
Derive more neighbor edges: hash-size-aware paths, anchors, confidence tiers
Rework the meshcore_all_neighbor_edges materialized view (migration 008) so the map's "show all neighbors" layer surfaces many more, and higher-quality, edges: - Parse routing paths at the packet's real hash_size (1/2/3 bytes per hop) instead of assuming 1-byte hops, so extended-hash packets are no longer mis-sliced, and wide (2-/3-byte) prefixes resolve uniquely. - Add anchored edges: resolve a hop against a fully-known endpoint (the advert originator or the uploading gateway) when exactly one in-region repeater of the right prefix sits within a plausible LoRa-hop distance. - Tag every edge with a derivation `method` and a `confidence` score, and aggregate undirected edges across observations. Frontend: expose method/confidence on the all-neighbors API and color edges by method / fade by confidence. The old "only MQTT neighbors" checkbox becomes the top notch of a single confidence selector (MQTT-direct only -> high -> standard -> all). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Vanderpot
@@ -0,0 +1,329 @@
|
||||
-- +goose Up
|
||||
-- Rework meshcore_all_neighbor_edges to derive far more (and higher-quality) neighbor edges.
|
||||
--
|
||||
-- The migration-004 path-edge logic had three defects:
|
||||
-- 1. It hard-coded 1-byte hops (substring(path, 2*i-1, 2)), so every extended-hash packet
|
||||
-- (hash_size 2/3, added in migration 007) was mis-sliced into garbage prefixes.
|
||||
-- 2. It kept a 1-byte prefix only if exactly one repeater in the region owned it
|
||||
-- (HAVING node_count = 1). With only 256 one-byte buckets and often more repeaters than
|
||||
-- that per region, birthday collisions discard most prefixes, and a 1-byte "unique" match
|
||||
-- is itself unreliable (it frequently stitches together non-adjacent nodes).
|
||||
-- 3. It could not exploit extended hashes, whose 2-/3-byte prefixes (65 536+ buckets) are
|
||||
-- almost always unique and resolve to genuinely adjacent nodes.
|
||||
--
|
||||
-- The new MV derives edges in confidence tiers and tags each with `method`/`confidence` so the
|
||||
-- map can filter. All path/anchor tiers read FLOOD packets (route_type 0/1, where the path is
|
||||
-- accumulated toward the gateway) over a 7-day window, parse hops at the packet's real hash_size,
|
||||
-- and aggregate prefix pairs BEFORE joining (keeps the refresh ~20s, not minutes):
|
||||
-- * direct - path_len=0 adverts: gateway heard advertiser at zero hops. (unchanged)
|
||||
-- * anchor-origin - advert originator (full key + location) was heard by the first path hop;
|
||||
-- resolve that hop to the single in-region repeater within MAX_HOP of the originator.
|
||||
-- * anchor-gateway- the uploading gateway (full key + location) heard the last path hop;
|
||||
-- resolve to the single in-region repeater within MAX_HOP of the gateway.
|
||||
-- MAX_HOP (206 km) is an upper bound on a plausible single LoRa hop between two located nodes;
|
||||
-- adjacencies longer than this are treated as hash-collision artifacts and dropped.
|
||||
-- * path-uniq-3b / -2b / -1b - consecutive path hops resolved by region-wide prefix uniqueness
|
||||
-- at the pair's own hash width (3-/2-byte are high quality; 1-byte is the noisy
|
||||
-- low-confidence fallback).
|
||||
-- Edges are canonicalized undirected and aggregated across all observations (observation_count).
|
||||
|
||||
DROP VIEW IF EXISTS meshcore_all_neighbor_edges;
|
||||
-- +goose StatementBegin
|
||||
CREATE MATERIALIZED VIEW IF NOT EXISTS meshcore_all_neighbor_edges
|
||||
REFRESH EVERY 1 HOUR
|
||||
ENGINE = MergeTree
|
||||
ORDER BY (region, source_node, target_node)
|
||||
AS
|
||||
WITH
|
||||
node_details AS (
|
||||
SELECT
|
||||
public_key,
|
||||
node_name,
|
||||
last_seen,
|
||||
ifNull(latitude, 0.) AS lat,
|
||||
ifNull(longitude, 0.) AS lon,
|
||||
-- (0,0) / missing coords are treated as "no usable location" (avoids edges drawn to null island)
|
||||
(has_location AND abs(ifNull(latitude, 0.)) > 0.01 AND abs(ifNull(longitude, 0.)) > 0.01) AS loc_ok
|
||||
FROM meshcore_adverts_latest
|
||||
),
|
||||
-- Repeaters seen in the last 2 days, region derived from their latest topic.
|
||||
repeaters AS (
|
||||
SELECT
|
||||
public_key,
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region,
|
||||
ifNull(latitude, 0.) AS lat,
|
||||
ifNull(longitude, 0.) AS lon,
|
||||
(has_location AND abs(ifNull(latitude, 0.)) > 0.01 AND abs(ifNull(longitude, 0.)) > 0.01) AS loc_ok
|
||||
FROM meshcore_adverts_latest
|
||||
WHERE is_repeater = 1 AND last_seen >= now() - INTERVAL 2 DAY
|
||||
),
|
||||
repeaters_loc AS (SELECT * FROM repeaters WHERE loc_ok AND region != ''),
|
||||
-- (region, width, prefix) -> representative key, with the count used to assert uniqueness (n = 1)
|
||||
-- at each possible hash width (1/2/3 bytes = 2/4/6 hex chars).
|
||||
prefix_uniq AS (
|
||||
SELECT region, 1 AS w, substring(public_key, 1, 2) AS prefix, any(public_key) AS key, count() AS n
|
||||
FROM repeaters WHERE region != '' GROUP BY region, substring(public_key, 1, 2)
|
||||
UNION ALL
|
||||
SELECT region, 2 AS w, substring(public_key, 1, 4) AS prefix, any(public_key) AS key, count() AS n
|
||||
FROM repeaters WHERE region != '' GROUP BY region, substring(public_key, 1, 4)
|
||||
UNION ALL
|
||||
SELECT region, 3 AS w, substring(public_key, 1, 6) AS prefix, any(public_key) AS key, count() AS n
|
||||
FROM repeaters WHERE region != '' GROUP BY region, substring(public_key, 1, 6)
|
||||
),
|
||||
-- Consecutive hop-prefix pairs across all flood packets, aggregated BEFORE the join. Each hop is
|
||||
-- 2*hash_size hex chars; pair i is (hop i, hop i+1). Distinct pairs are few, so this collapses the
|
||||
-- ~50M exploded pairs to a small set and keeps the downstream joins cheap.
|
||||
path_pair_counts AS (
|
||||
SELECT region, hash_size, tupleElement(pair, 1) AS src_prefix, tupleElement(pair, 2) AS dst_prefix, count() AS obs
|
||||
FROM (
|
||||
SELECT
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region,
|
||||
hash_size,
|
||||
arrayMap(i -> (substring(path, 2*hash_size*(i-1)+1, 2*hash_size), substring(path, 2*hash_size*i+1, 2*hash_size)), range(1, hop_count)) AS pairs
|
||||
FROM meshcore_packets
|
||||
WHERE ingest_timestamp >= now() - INTERVAL 7 DAY
|
||||
AND hash_size_code != 3 AND hash_size != 0 AND route_type IN (0, 1) AND hop_count >= 2
|
||||
) ARRAY JOIN pairs AS pair
|
||||
WHERE region != '' AND tupleElement(pair, 1) != tupleElement(pair, 2)
|
||||
GROUP BY region, hash_size, src_prefix, dst_prefix
|
||||
),
|
||||
edges_path_uniq AS (
|
||||
SELECT ppc.region AS region, su.key AS source_node, du.key AS target_node,
|
||||
multiIf(ppc.hash_size >= 3, 'path-uniq-3b', ppc.hash_size = 2, 'path-uniq-2b', 'path-uniq-1b') AS method,
|
||||
ppc.obs AS obs
|
||||
FROM path_pair_counts ppc
|
||||
INNER JOIN prefix_uniq su ON su.region = ppc.region AND su.w = ppc.hash_size AND su.prefix = ppc.src_prefix AND su.n = 1
|
||||
INNER JOIN prefix_uniq du ON du.region = ppc.region AND du.w = ppc.hash_size AND du.prefix = ppc.dst_prefix AND du.n = 1
|
||||
),
|
||||
-- anchor-gateway: the uploading gateway (full key) heard the last hop. Aggregate (gateway, last_prefix)
|
||||
-- first, then accept iff exactly one in-region repeater with that prefix is within 150 km of the gateway.
|
||||
last_hop_counts AS (
|
||||
SELECT region, hash_size, gateway_key, last_prefix, count() AS obs
|
||||
FROM (
|
||||
SELECT
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region,
|
||||
hash_size, hex(origin_pubkey) AS gateway_key,
|
||||
substring(path, 2*hash_size*(hop_count-1)+1, 2*hash_size) AS last_prefix
|
||||
FROM meshcore_packets
|
||||
WHERE ingest_timestamp >= now() - INTERVAL 7 DAY
|
||||
AND hash_size_code != 3 AND hash_size != 0 AND route_type IN (0, 1) AND hop_count >= 1
|
||||
)
|
||||
WHERE region != ''
|
||||
GROUP BY region, hash_size, gateway_key, last_prefix
|
||||
),
|
||||
edges_anchor_gw AS (
|
||||
SELECT lhc.region AS region, lhc.gateway_key AS source_node, any(R.public_key) AS target_node,
|
||||
'anchor-gateway' AS method, any(lhc.obs) AS obs
|
||||
FROM last_hop_counts lhc
|
||||
INNER JOIN node_details g ON g.public_key = lhc.gateway_key AND g.loc_ok
|
||||
INNER JOIN repeaters_loc R ON R.region = lhc.region AND substring(R.public_key, 1, 2*lhc.hash_size) = lhc.last_prefix
|
||||
AND R.public_key != lhc.gateway_key AND greatCircleDistance(R.lon, R.lat, g.lon, g.lat) <= 206000
|
||||
GROUP BY lhc.region, lhc.gateway_key, lhc.hash_size, lhc.last_prefix
|
||||
HAVING count() = 1
|
||||
),
|
||||
-- anchor-origin: an advert's originator (full key from the payload) was heard by the first hop.
|
||||
advert_first_counts AS (
|
||||
SELECT region, hash_size, origin_key, first_prefix, count() AS obs
|
||||
FROM (
|
||||
SELECT
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region,
|
||||
hash_size, hex(substring(payload, 1, 32)) AS origin_key, substring(path, 1, 2*hash_size) AS first_prefix
|
||||
FROM meshcore_packets
|
||||
WHERE payload_type = 4 AND ingest_timestamp >= now() - INTERVAL 7 DAY
|
||||
AND hash_size_code != 3 AND hash_size != 0 AND route_type IN (0, 1) AND hop_count >= 1
|
||||
)
|
||||
WHERE region != ''
|
||||
GROUP BY region, hash_size, origin_key, first_prefix
|
||||
),
|
||||
edges_anchor_origin AS (
|
||||
SELECT afc.region AS region, afc.origin_key AS source_node, any(R.public_key) AS target_node,
|
||||
'anchor-origin' AS method, any(afc.obs) AS obs
|
||||
FROM advert_first_counts afc
|
||||
INNER JOIN node_details o ON o.public_key = afc.origin_key AND o.loc_ok
|
||||
INNER JOIN repeaters_loc R ON R.region = afc.region AND substring(R.public_key, 1, 2*afc.hash_size) = afc.first_prefix
|
||||
AND R.public_key != afc.origin_key AND greatCircleDistance(R.lon, R.lat, o.lon, o.lat) <= 206000
|
||||
GROUP BY afc.region, afc.origin_key, afc.hash_size, afc.first_prefix
|
||||
HAVING count() = 1
|
||||
),
|
||||
-- Direct connections (path_len = 0 adverts): gateway heard the advertiser with no intermediate hops.
|
||||
direct_connections AS (
|
||||
SELECT
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region,
|
||||
hex(origin_pubkey) AS source_node, public_key AS target_node, count() AS obs
|
||||
FROM meshcore_adverts
|
||||
WHERE path_len = 0 AND hex(origin_pubkey) != public_key AND ingest_timestamp >= now() - INTERVAL 7 DAY
|
||||
GROUP BY region, source_node, target_node
|
||||
),
|
||||
all_edges AS (
|
||||
SELECT region, source_node, target_node, method, obs,
|
||||
multiIf(method = 'direct', 0, method LIKE 'anchor-%', 1, method = 'path-uniq-3b', 1, method = 'path-uniq-2b', 2, 3) AS rank
|
||||
FROM (
|
||||
SELECT region, source_node, target_node, method, obs FROM edges_path_uniq
|
||||
UNION ALL SELECT region, source_node, target_node, method, obs FROM edges_anchor_gw
|
||||
UNION ALL SELECT region, source_node, target_node, method, obs FROM edges_anchor_origin
|
||||
UNION ALL SELECT region, source_node, target_node, 'direct' AS method, obs FROM direct_connections
|
||||
)
|
||||
),
|
||||
-- Canonicalize undirected and merge across methods/observations: best (lowest-rank) method wins.
|
||||
edge_consensus AS (
|
||||
SELECT region, a AS source_node, b AS target_node,
|
||||
argMin(method, rank) AS method, min(rank) AS best_rank, sum(obs) AS observation_count
|
||||
FROM (
|
||||
SELECT region, method, obs, rank,
|
||||
least(source_node, target_node) AS a, greatest(source_node, target_node) AS b
|
||||
FROM all_edges
|
||||
WHERE source_node != target_node AND source_node != '' AND target_node != '' AND region != ''
|
||||
)
|
||||
GROUP BY region, a, b
|
||||
)
|
||||
SELECT
|
||||
ec.region AS region,
|
||||
ec.source_node AS source_node,
|
||||
ec.target_node AS target_node,
|
||||
-- back-compat connection_type: only a path_len=0 advert is a literal MQTT-direct edge; anchors
|
||||
-- are single-hop but inferred, so they group with 'path'. Fine-grained tier is in `method`.
|
||||
if(ec.method = 'direct', 'direct', 'path') AS connection_type,
|
||||
ec.method AS method,
|
||||
-- tier base (direct 1.0 .. path-uniq-1b 0.4) plus a small capped consensus bonus
|
||||
least(1.0, greatest(0.0, (1.0 - 0.2 * ec.best_rank) + least(0.08, 0.04 * log10(ec.observation_count + 1)))) AS confidence,
|
||||
ec.observation_count AS observation_count,
|
||||
ec.observation_count AS packet_count,
|
||||
sd.node_name AS source_name,
|
||||
if(sd.loc_ok, sd.lat, NULL) AS source_latitude,
|
||||
if(sd.loc_ok, sd.lon, NULL) AS source_longitude,
|
||||
toUInt8(sd.loc_ok) AS source_has_location,
|
||||
sd.last_seen AS source_last_seen,
|
||||
td.node_name AS target_name,
|
||||
if(td.loc_ok, td.lat, NULL) AS target_latitude,
|
||||
if(td.loc_ok, td.lon, NULL) AS target_longitude,
|
||||
toUInt8(td.loc_ok) AS target_has_location,
|
||||
td.last_seen AS target_last_seen
|
||||
FROM edge_consensus AS ec
|
||||
INNER JOIN node_details AS sd ON ec.source_node = sd.public_key
|
||||
INNER JOIN node_details AS td ON ec.target_node = td.public_key
|
||||
-- Backstop: drop geographically implausible adjacencies between two located nodes (a single LoRa
|
||||
-- hop beyond MAX_HOP / 206 km is not realistic and indicates a hash-collision false positive).
|
||||
WHERE NOT (sd.loc_ok AND td.loc_ok AND greatCircleDistance(sd.lon, sd.lat, td.lon, td.lat) > 206000);
|
||||
-- +goose StatementEnd
|
||||
|
||||
SYSTEM REFRESH VIEW meshcore_all_neighbor_edges;
|
||||
|
||||
|
||||
-- +goose Down
|
||||
-- Restore the migration-004 neighbor edge graph (1-byte hops, prefix-uniqueness only, no
|
||||
-- method/confidence columns).
|
||||
DROP VIEW IF EXISTS meshcore_all_neighbor_edges;
|
||||
-- +goose StatementBegin
|
||||
CREATE MATERIALIZED VIEW IF NOT EXISTS meshcore_all_neighbor_edges
|
||||
REFRESH EVERY 1 HOUR
|
||||
ENGINE = MergeTree
|
||||
ORDER BY (region, source_node, target_node)
|
||||
AS
|
||||
WITH
|
||||
node_details AS (
|
||||
SELECT
|
||||
public_key,
|
||||
node_name,
|
||||
last_seen,
|
||||
ifNull(latitude, 0.) AS lat,
|
||||
ifNull(longitude, 0.) AS lon,
|
||||
(has_location AND abs(ifNull(latitude, 0.)) > 0.01 AND abs(ifNull(longitude, 0.)) > 0.01) AS loc_ok
|
||||
FROM meshcore_adverts_latest
|
||||
),
|
||||
adverts_latest_r AS (
|
||||
SELECT
|
||||
public_key,
|
||||
is_repeater,
|
||||
last_seen,
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region
|
||||
FROM meshcore_adverts_latest
|
||||
),
|
||||
repeater_prefixes AS (
|
||||
SELECT
|
||||
region,
|
||||
substring(public_key, 1, 2) AS prefix,
|
||||
count() AS node_count,
|
||||
any(public_key) AS representative_key
|
||||
FROM adverts_latest_r
|
||||
WHERE is_repeater = 1 AND last_seen >= now() - INTERVAL 2 DAY AND region != ''
|
||||
GROUP BY region, prefix
|
||||
HAVING node_count = 1
|
||||
),
|
||||
path_src AS (
|
||||
SELECT DISTINCT
|
||||
payload,
|
||||
path,
|
||||
path_len,
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region
|
||||
FROM meshcore_packets
|
||||
WHERE path_len >= 2 AND ingest_timestamp >= now() - INTERVAL 1 DAY
|
||||
),
|
||||
path_pairs AS (
|
||||
SELECT DISTINCT
|
||||
region,
|
||||
payload,
|
||||
upper(substring(path, 2 * i - 1, 2)) AS source_prefix,
|
||||
upper(substring(path, 2 * i + 1, 2)) AS target_prefix
|
||||
FROM path_src
|
||||
ARRAY JOIN range(1, path_len) AS i
|
||||
WHERE i < path_len AND region != ''
|
||||
),
|
||||
path_neighbors AS (
|
||||
SELECT region, source_prefix, target_prefix, count() AS packet_count
|
||||
FROM path_pairs
|
||||
WHERE source_prefix != target_prefix
|
||||
GROUP BY region, source_prefix, target_prefix
|
||||
),
|
||||
path_connections AS (
|
||||
SELECT
|
||||
pn.region AS region,
|
||||
sm.representative_key AS source_node,
|
||||
tm.representative_key AS target_node,
|
||||
pn.packet_count AS packet_count
|
||||
FROM path_neighbors AS pn
|
||||
INNER JOIN repeater_prefixes AS sm ON sm.region = pn.region AND sm.prefix = pn.source_prefix
|
||||
INNER JOIN repeater_prefixes AS tm ON tm.region = pn.region AND tm.prefix = pn.target_prefix
|
||||
),
|
||||
direct_connections AS (
|
||||
SELECT DISTINCT
|
||||
multiIf(lower(topic) IN ('meshcore','meshcore/salish'), 'SEA', match(splitByChar('/', lower(topic))[2], '^[a-z]{3}$'), upper(splitByChar('/', lower(topic))[2]), '') AS region,
|
||||
hex(origin_pubkey) AS source_node,
|
||||
public_key AS target_node
|
||||
FROM meshcore_adverts
|
||||
WHERE path_len = 0
|
||||
AND hex(origin_pubkey) != public_key
|
||||
AND ingest_timestamp >= now() - INTERVAL 7 DAY
|
||||
),
|
||||
edges AS (
|
||||
SELECT region, source_node, target_node, 'path' AS connection_type, packet_count
|
||||
FROM path_connections
|
||||
UNION ALL
|
||||
SELECT d.region, d.source_node, d.target_node, 'direct' AS connection_type, CAST(1 AS UInt64) AS packet_count
|
||||
FROM direct_connections AS d
|
||||
WHERE d.region != ''
|
||||
AND (d.region, d.source_node, d.target_node) NOT IN (SELECT region, source_node, target_node FROM path_connections)
|
||||
AND (d.region, d.target_node, d.source_node) NOT IN (SELECT region, source_node, target_node FROM path_connections)
|
||||
)
|
||||
SELECT
|
||||
e.region AS region,
|
||||
e.source_node AS source_node,
|
||||
e.target_node AS target_node,
|
||||
e.connection_type AS connection_type,
|
||||
e.packet_count AS packet_count,
|
||||
sd.node_name AS source_name,
|
||||
if(sd.loc_ok, sd.lat, NULL) AS source_latitude,
|
||||
if(sd.loc_ok, sd.lon, NULL) AS source_longitude,
|
||||
toUInt8(sd.loc_ok) AS source_has_location,
|
||||
sd.last_seen AS source_last_seen,
|
||||
td.node_name AS target_name,
|
||||
if(td.loc_ok, td.lat, NULL) AS target_latitude,
|
||||
if(td.loc_ok, td.lon, NULL) AS target_longitude,
|
||||
toUInt8(td.loc_ok) AS target_has_location,
|
||||
td.last_seen AS target_last_seen
|
||||
FROM edges AS e
|
||||
INNER JOIN node_details AS sd ON e.source_node = sd.public_key
|
||||
INNER JOIN node_details AS td ON e.target_node = td.public_key
|
||||
WHERE NOT (sd.loc_ok AND td.loc_ok AND greatCircleDistance(sd.lon, sd.lat, td.lon, td.lat) > 150000);
|
||||
-- +goose StatementEnd
|
||||
|
||||
SYSTEM REFRESH VIEW meshcore_all_neighbor_edges;
|
||||
@@ -11,12 +11,13 @@ export async function GET(req: Request) {
|
||||
const nodeTypes = searchParams.getAll("nodeTypes");
|
||||
const lastSeen = searchParams.get("lastSeen");
|
||||
const region = searchParams.get("region");
|
||||
const minConfidence = searchParams.get("minConfidence");
|
||||
const includeNeighbors = searchParams.get("includeNeighbors") === "true";
|
||||
|
||||
|
||||
const positions = await getNodePositions({ minLat, maxLat, minLng, maxLng, nodeTypes, lastSeen });
|
||||
|
||||
|
||||
if (includeNeighbors) {
|
||||
const neighbors = await getAllNodeNeighbors(lastSeen, minLat, maxLat, minLng, maxLng, nodeTypes, region || undefined);
|
||||
const neighbors = await getAllNodeNeighbors(lastSeen, minLat, maxLat, minLng, maxLng, nodeTypes, region || undefined, minConfidence);
|
||||
return NextResponse.json({
|
||||
nodes: positions,
|
||||
neighbors: neighbors
|
||||
|
||||
@@ -11,8 +11,10 @@ export async function GET(req: Request) {
|
||||
const nodeTypes = searchParams.getAll("nodeTypes");
|
||||
const lastSeen = searchParams.get("lastSeen");
|
||||
const region = searchParams.get("region");
|
||||
const minConfidence = searchParams.get("minConfidence");
|
||||
const methods = searchParams.getAll("methods");
|
||||
|
||||
const neighbors = await getAllNodeNeighbors(lastSeen, minLat, maxLat, minLng, maxLng, nodeTypes, region || undefined);
|
||||
const neighbors = await getAllNodeNeighbors(lastSeen, minLat, maxLat, minLng, maxLng, nodeTypes, region || undefined, minConfidence, methods);
|
||||
|
||||
return NextResponse.json(neighbors);
|
||||
} catch (error) {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"use client";
|
||||
import React, { useState, useRef, useEffect } from 'react';
|
||||
import { useMapLayerSettings, TILE_LAYERS, NODE_TYPE_OPTIONS, type MapLayerSettings } from '@/hooks/useMapLayerSettings';
|
||||
import { useMapLayerSettings, TILE_LAYERS, NODE_TYPE_OPTIONS, NEIGHBOR_CONFIDENCE_OPTIONS, type MapLayerSettings } from '@/hooks/useMapLayerSettings';
|
||||
import { Square3Stack3DIcon } from '@heroicons/react/24/outline';
|
||||
|
||||
interface MapLayerSettingsProps {
|
||||
@@ -189,23 +189,38 @@ export default function MapLayerSettingsComponent({ onSettingsChange }: MapLayer
|
||||
</span>
|
||||
</label>
|
||||
|
||||
{/* Only show MQTT neighbors - indented sub-option */}
|
||||
<label className="flex items-center gap-2 ml-6 cursor-pointer">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={settings.onlyMqttNeighbors}
|
||||
onChange={(e) => updateSetting('onlyMqttNeighbors', e.target.checked)}
|
||||
disabled={!settings.showAllNeighbors}
|
||||
className="rounded"
|
||||
/>
|
||||
<span className={`text-sm ${
|
||||
{/* Neighbor confidence - indented sub-option (subsumes the old "only MQTT" toggle:
|
||||
"MQTT direct only" is the top tier) */}
|
||||
<div className="ml-6 mb-3">
|
||||
<label className={`block text-sm ${
|
||||
settings.showAllNeighbors
|
||||
? 'text-gray-700 dark:text-gray-300'
|
||||
: 'text-gray-400 dark:text-gray-500'
|
||||
} mb-1`}>
|
||||
Neighbor confidence
|
||||
</label>
|
||||
<select
|
||||
value={settings.minConfidence}
|
||||
onChange={(e) => updateSetting('minConfidence', parseFloat(e.target.value))}
|
||||
disabled={!settings.showAllNeighbors}
|
||||
className={`w-full p-2 border border-gray-300 dark:border-neutral-600 rounded text-sm ${
|
||||
settings.showAllNeighbors
|
||||
? 'bg-white dark:bg-neutral-800 text-gray-700 dark:text-gray-300'
|
||||
: 'bg-gray-100 dark:bg-neutral-700 text-gray-400 dark:text-gray-500 cursor-not-allowed'
|
||||
}`}
|
||||
>
|
||||
{NEIGHBOR_CONFIDENCE_OPTIONS.map(opt => (
|
||||
<option key={opt.value} value={opt.value}>{opt.label}</option>
|
||||
))}
|
||||
</select>
|
||||
<p className={`text-xs mt-1 ${
|
||||
settings.showAllNeighbors
|
||||
? 'text-gray-500 dark:text-gray-400'
|
||||
: 'text-gray-400 dark:text-gray-500'
|
||||
}`}>
|
||||
Only show MQTT neighbors
|
||||
</span>
|
||||
</label>
|
||||
Higher tiers show only the most trustworthy links (MQTT-direct, then anchored / extended-hash)
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Minimum packet count - indented sub-option */}
|
||||
<div className="ml-6 mb-3">
|
||||
|
||||
@@ -415,20 +415,18 @@ function NeighborLines({
|
||||
}
|
||||
|
||||
// Component to render all neighbor lines for all nodes
|
||||
function AllNeighborLines({
|
||||
connections,
|
||||
function AllNeighborLines({
|
||||
connections,
|
||||
nodes,
|
||||
useColors = true,
|
||||
minPacketCount = 1,
|
||||
strokeWidth = 2,
|
||||
onlyMqtt = false
|
||||
strokeWidth = 2
|
||||
}: {
|
||||
connections: AllNeighborsConnection[];
|
||||
nodes: NodePosition[];
|
||||
useColors?: boolean;
|
||||
minPacketCount?: number;
|
||||
strokeWidth?: number;
|
||||
onlyMqtt?: boolean;
|
||||
}) {
|
||||
if (connections.length === 0) return null;
|
||||
|
||||
@@ -436,12 +434,12 @@ function AllNeighborLines({
|
||||
const visibleNodeIds = new Set(nodes.map(node => node.node_id));
|
||||
|
||||
// Filter connections to only show lines between nodes that are visible on the map
|
||||
// and meet the minimum packet count threshold
|
||||
// and meet the minimum packet count threshold. Confidence filtering happens server-side
|
||||
// (the confidence selector drives a refetch), so no client-side confidence filter here.
|
||||
const visibleConnections = connections.filter(connection =>
|
||||
visibleNodeIds.has(connection.source_node) &&
|
||||
visibleNodeIds.has(connection.target_node) &&
|
||||
connection.packet_count >= minPacketCount &&
|
||||
(!onlyMqtt || connection.connection_type === 'direct')
|
||||
connection.packet_count >= minPacketCount
|
||||
);
|
||||
|
||||
// Calculate logarithmic thresholds based on packet counts for path connections
|
||||
@@ -488,39 +486,41 @@ function AllNeighborLines({
|
||||
[connection.target_latitude, connection.target_longitude]
|
||||
];
|
||||
|
||||
// Different colors based on connection type and logarithmic packet count
|
||||
const getConnectionColor = (connectionType: string, packetCount: number) => {
|
||||
// Color by derivation method; path-inferred edges use a logarithmic packet-count gradient.
|
||||
const getConnectionColor = (method: string, connectionType: string, packetCount: number) => {
|
||||
if (!useColors) {
|
||||
// If colors are disabled, use consistent colors based on connection type
|
||||
return connectionType === 'direct' ? '#8b5cf6' : '#6b7280'; // Purple for direct, gray for path
|
||||
return method === 'direct' ? '#8b5cf6'
|
||||
: method.startsWith('anchor-') ? '#3b82f6'
|
||||
: '#6b7280'; // purple direct, blue anchor, gray path
|
||||
}
|
||||
|
||||
if (connectionType === 'direct') {
|
||||
return '#8b5cf6'; // Purple for direct connections
|
||||
}
|
||||
|
||||
// For path connections, use logarithmic thresholds for color intensity
|
||||
|
||||
if (method === 'direct') return '#8b5cf6'; // Purple for literal MQTT-direct edges
|
||||
if (method.startsWith('anchor-')) return '#3b82f6'; // Blue for anchored single-hop edges
|
||||
|
||||
// For path-inferred connections, use logarithmic thresholds for color intensity
|
||||
if (packetCount >= thresholds.t4) return '#dc2626'; // Red for highest log range
|
||||
if (packetCount >= thresholds.t3) return '#ea580c'; // Dark orange
|
||||
if (packetCount >= thresholds.t3) return '#ea580c'; // Dark orange
|
||||
if (packetCount >= thresholds.t2) return '#f59e0b'; // Orange
|
||||
if (packetCount >= thresholds.t1) return '#eab308'; // Yellow
|
||||
if (packetCount > thresholds.min) return '#84cc16'; // Light green for above minimum
|
||||
return '#6b7280'; // Gray for minimum traffic
|
||||
};
|
||||
|
||||
const lineColor = getConnectionColor(connection.connection_type, connection.packet_count);
|
||||
|
||||
// Use strokeWidth setting for line weight
|
||||
|
||||
const lineColor = getConnectionColor(connection.method, connection.connection_type, connection.packet_count);
|
||||
|
||||
// Use strokeWidth setting for line weight; inferred (path) edges are drawn slightly thinner.
|
||||
const lineWeight = connection.connection_type === 'direct' ? strokeWidth : Math.max(1, strokeWidth - 1);
|
||||
|
||||
// Fade lower-confidence edges so the trustworthy ones stand out.
|
||||
const lineOpacity = Math.max(0.25, Math.min(0.9, 0.25 + 0.65 * (connection.confidence ?? 0.5)));
|
||||
|
||||
return (
|
||||
<Polyline
|
||||
key={`${connection.source_node}-${connection.target_node}-${connection.connection_type}`}
|
||||
key={`${connection.source_node}-${connection.target_node}-${connection.method}`}
|
||||
positions={positions}
|
||||
pathOptions={{
|
||||
color: lineColor,
|
||||
weight: lineWeight,
|
||||
opacity: 0.7,
|
||||
opacity: lineOpacity,
|
||||
}}
|
||||
/>
|
||||
);
|
||||
@@ -551,7 +551,7 @@ export default function MapView({ target = '_self' }: MapViewProps = {}) {
|
||||
tileLayer: "openstreetmap",
|
||||
showAllNeighbors: false,
|
||||
useColors: true,
|
||||
onlyMqttNeighbors: false,
|
||||
minConfidence: 0.5,
|
||||
nodeTypes: ["meshcore"],
|
||||
showMeshcoreCoverageOverlay: false,
|
||||
minPacketCount: 1,
|
||||
@@ -651,6 +651,7 @@ export default function MapView({ target = '_self' }: MapViewProps = {}) {
|
||||
}
|
||||
if (includeNeighbors) {
|
||||
params.push('includeNeighbors=true');
|
||||
params.push(`minConfidence=${mapLayerSettings.minConfidence}`);
|
||||
}
|
||||
if (params.length > 0) {
|
||||
url += `?${params.join("&")}`;
|
||||
@@ -696,7 +697,7 @@ export default function MapView({ target = '_self' }: MapViewProps = {}) {
|
||||
setAllNeighborsLoading(false);
|
||||
}
|
||||
});
|
||||
}, [mapLayerSettings.nodeTypes, config?.lastSeen, config?.selectedRegion]);
|
||||
}, [mapLayerSettings.nodeTypes, mapLayerSettings.minConfidence, config?.lastSeen, config?.selectedRegion]);
|
||||
|
||||
function isBoundsInside(inner: [[number, number], [number, number]], outer: [[number, number], [number, number]]) {
|
||||
// inner: [[minLat, minLng], [maxLat, maxLng]]
|
||||
@@ -889,19 +890,21 @@ export default function MapView({ target = '_self' }: MapViewProps = {}) {
|
||||
strokeWidth={mapLayerSettings.strokeWidth}
|
||||
/>
|
||||
{showAllNeighbors && (
|
||||
<AllNeighborLines
|
||||
<AllNeighborLines
|
||||
connections={allNeighborConnections}
|
||||
nodes={nodePositions}
|
||||
useColors={mapLayerSettings.useColors}
|
||||
minPacketCount={mapLayerSettings.minPacketCount}
|
||||
strokeWidth={mapLayerSettings.strokeWidth}
|
||||
onlyMqtt={mapLayerSettings.onlyMqttNeighbors}
|
||||
/>
|
||||
)}
|
||||
</MapContainer>
|
||||
|
||||
{/* Traffic Legend */}
|
||||
{showAllNeighbors && mapLayerSettings.useColors && allNeighborConnections.length > 0 && (() => {
|
||||
// At the top confidence notch only literal MQTT-direct edges are shown.
|
||||
const directOnly = mapLayerSettings.minConfidence >= 1;
|
||||
const hasAnchor = allNeighborConnections.some(conn => conn.method?.startsWith('anchor-'));
|
||||
// Calculate logarithmic thresholds for legend display
|
||||
const pathConnections = allNeighborConnections.filter(conn => conn.connection_type === 'path');
|
||||
const packetCounts = pathConnections.map(conn => conn.packet_count).sort((a, b) => a - b);
|
||||
@@ -925,7 +928,7 @@ export default function MapView({ target = '_self' }: MapViewProps = {}) {
|
||||
t4: Math.round(Math.pow(10, logMin + logRange * 0.8)),
|
||||
max
|
||||
};
|
||||
})() : (mapLayerSettings.onlyMqttNeighbors ? { min: 1, t1: 1, t2: 1, t3: 1, t4: 1, max: 1 } : null);
|
||||
})() : (directOnly ? { min: 1, t1: 1, t2: 1, t3: 1, t4: 1, max: 1 } : null);
|
||||
|
||||
return legendThresholds && (
|
||||
<div style={{
|
||||
@@ -941,10 +944,10 @@ export default function MapView({ target = '_self' }: MapViewProps = {}) {
|
||||
fontFamily: 'monospace'
|
||||
}}>
|
||||
<div style={{ fontWeight: 'bold', marginBottom: '8px' }}>
|
||||
{mapLayerSettings.onlyMqttNeighbors ? 'Neighbors' : 'Path Traffic'}
|
||||
{directOnly ? 'Neighbors' : 'Path Traffic'}
|
||||
</div>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: '4px' }}>
|
||||
{!mapLayerSettings.onlyMqttNeighbors && (
|
||||
{!directOnly && (
|
||||
<>
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: '8px' }}>
|
||||
<div style={{ width: '20px', height: '2px', backgroundColor: '#dc2626' }}></div>
|
||||
@@ -972,10 +975,16 @@ export default function MapView({ target = '_self' }: MapViewProps = {}) {
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: '8px', ...(mapLayerSettings.onlyMqttNeighbors ? {} : { marginTop: '4px', paddingTop: '4px', borderTop: '1px solid #e5e7eb' }) }}>
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: '8px', ...(directOnly ? {} : { marginTop: '4px', paddingTop: '4px', borderTop: '1px solid #e5e7eb' }) }}>
|
||||
<div style={{ width: '20px', height: '2px', backgroundColor: '#8b5cf6' }}></div>
|
||||
<span>MQTT connections</span>
|
||||
<span>MQTT direct</span>
|
||||
</div>
|
||||
{hasAnchor && !directOnly && (
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: '8px' }}>
|
||||
<div style={{ width: '20px', height: '2px', backgroundColor: '#3b82f6' }}></div>
|
||||
<span>Anchored neighbor</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -5,6 +5,9 @@ export interface AllNeighborsConnection {
|
||||
source_node: string;
|
||||
target_node: string;
|
||||
connection_type: string;
|
||||
method: string;
|
||||
confidence: number;
|
||||
observation_count: number;
|
||||
packet_count: number;
|
||||
source_name: string;
|
||||
source_latitude: number;
|
||||
@@ -24,21 +27,23 @@ interface UseAllNeighborsParams {
|
||||
nodeTypes?: string[];
|
||||
lastSeen?: number | null;
|
||||
region?: string;
|
||||
minConfidence?: number | null;
|
||||
enabled?: boolean;
|
||||
}
|
||||
|
||||
export function useAllNeighbors({
|
||||
minLat,
|
||||
maxLat,
|
||||
minLng,
|
||||
maxLng,
|
||||
nodeTypes,
|
||||
lastSeen,
|
||||
export function useAllNeighbors({
|
||||
minLat,
|
||||
maxLat,
|
||||
minLng,
|
||||
maxLng,
|
||||
nodeTypes,
|
||||
lastSeen,
|
||||
region,
|
||||
enabled = true
|
||||
minConfidence,
|
||||
enabled = true
|
||||
}: UseAllNeighborsParams) {
|
||||
return useQuery({
|
||||
queryKey: ['allNeighbors', minLat, maxLat, minLng, maxLng, nodeTypes, lastSeen, region],
|
||||
queryKey: ['allNeighbors', minLat, maxLat, minLng, maxLng, nodeTypes, lastSeen, region, minConfidence],
|
||||
queryFn: async (): Promise<AllNeighborsConnection[]> => {
|
||||
const params = new URLSearchParams();
|
||||
|
||||
@@ -63,7 +68,10 @@ export function useAllNeighbors({
|
||||
if (region) {
|
||||
params.append('region', region);
|
||||
}
|
||||
|
||||
if (minConfidence !== null && minConfidence !== undefined) {
|
||||
params.append('minConfidence', minConfidence.toString());
|
||||
}
|
||||
|
||||
const url = `/api/neighbors/all${params.toString() ? `?${params.toString()}` : ''}`;
|
||||
|
||||
const response = await fetch(buildApiUrl(url));
|
||||
|
||||
@@ -10,7 +10,9 @@ export interface MapLayerSettings {
|
||||
tileLayer: string;
|
||||
showAllNeighbors: boolean;
|
||||
useColors: boolean;
|
||||
onlyMqttNeighbors: boolean;
|
||||
// Minimum edge confidence to display. Subsumes the old "only MQTT neighbors" toggle: at 1.0 only
|
||||
// literal MQTT-direct edges show; lower values progressively include anchored and path-inferred edges.
|
||||
minConfidence: number;
|
||||
nodeTypes: NodeType[];
|
||||
showMeshcoreCoverageOverlay: boolean;
|
||||
minPacketCount: number;
|
||||
@@ -24,7 +26,7 @@ const DEFAULT_MAP_LAYER_SETTINGS: MapLayerSettings = {
|
||||
tileLayer: "openstreetmap",
|
||||
showAllNeighbors: false,
|
||||
useColors: true,
|
||||
onlyMqttNeighbors: false,
|
||||
minConfidence: 0.5,
|
||||
nodeTypes: ["meshcore"],
|
||||
showMeshcoreCoverageOverlay: false,
|
||||
minPacketCount: 1,
|
||||
@@ -44,3 +46,12 @@ export const TILE_LAYERS = [
|
||||
export const NODE_TYPE_OPTIONS = [
|
||||
{ key: "meshcore", label: "Meshcore" },
|
||||
];
|
||||
|
||||
// Neighbor confidence tiers, mapped to the minimum edge confidence emitted by
|
||||
// meshcore_all_neighbor_edges (direct=1.0, anchored/3-byte≈0.8, 2-byte≈0.6, 1-byte≈0.4).
|
||||
export const NEIGHBOR_CONFIDENCE_OPTIONS = [
|
||||
{ value: 1.0, label: "MQTT direct only" },
|
||||
{ value: 0.7, label: "High (anchored + extended hash)" },
|
||||
{ value: 0.5, label: "Standard" },
|
||||
{ value: 0, label: "All (include weak 1-byte)" },
|
||||
];
|
||||
|
||||
@@ -303,7 +303,7 @@ export async function getMeshcoreNodeInfo(publicKey: string, limit: number = 50)
|
||||
}
|
||||
}
|
||||
|
||||
export async function getAllNodeNeighbors(lastSeen: string | null = null, minLat?: string | null, maxLat?: string | null, minLng?: string | null, maxLng?: string | null, nodeTypes?: string[], region?: string) {
|
||||
export async function getAllNodeNeighbors(lastSeen: string | null = null, minLat?: string | null, maxLat?: string | null, minLng?: string | null, maxLng?: string | null, nodeTypes?: string[], region?: string, minConfidence?: string | null, methods?: string[]) {
|
||||
try {
|
||||
// Reads the precomputed (hourly-refreshed) neighbor edge graph and filters it
|
||||
// by region + bounding box + lastSeen. The heavy graph computation lives in the
|
||||
@@ -353,12 +353,25 @@ export async function getAllNodeNeighbors(lastSeen: string | null = null, minLat
|
||||
whereConditions.push("source_last_seen >= now() - INTERVAL {lastSeen:UInt32} SECOND AND target_last_seen >= now() - INTERVAL {lastSeen:UInt32} SECOND");
|
||||
params.lastSeen = Number(lastSeen);
|
||||
}
|
||||
// Confidence floor: hide low-confidence edges (e.g. the noisy 1-byte path tier) by default.
|
||||
if (minConfidence !== null && minConfidence !== undefined && minConfidence !== "") {
|
||||
whereConditions.push("confidence >= {minConfidence:Float32}");
|
||||
params.minConfidence = Number(minConfidence);
|
||||
}
|
||||
// Explicit derivation-method filter (e.g. only direct/anchor edges).
|
||||
if (methods && methods.length > 0) {
|
||||
whereConditions.push("method IN {methods:Array(String)}");
|
||||
params.methods = methods;
|
||||
}
|
||||
|
||||
const allNeighborsQuery = `
|
||||
SELECT
|
||||
source_node,
|
||||
target_node,
|
||||
connection_type,
|
||||
method,
|
||||
confidence,
|
||||
observation_count,
|
||||
packet_count,
|
||||
source_name,
|
||||
source_latitude,
|
||||
@@ -384,6 +397,9 @@ export async function getAllNodeNeighbors(lastSeen: string | null = null, minLat
|
||||
source_node: string;
|
||||
target_node: string;
|
||||
connection_type: string;
|
||||
method: string;
|
||||
confidence: number;
|
||||
observation_count: number;
|
||||
packet_count: number;
|
||||
source_name: string;
|
||||
source_latitude: number;
|
||||
|
||||
Reference in New Issue
Block a user