web: route-labelled requests, scrape-time node series, true node count (#1034)

CHANGELOG.md

@@ -189,6 +189,10 @@
 * Web: a federation instance row is replaced in one transaction: a failed write keeps the stored row, and two writes for one domain at once no longer fail on the unique domain index (SPEC FK1)
 * Web: a database error while a crawl stores one peer is logged and the crawl goes on with the next peer (SPEC FK2)
 * Web: a keyfile that cannot be parsed stops the boot with an error naming the file and is left as it is, instead of being replaced by a new key and instance id; a legacy keyfile that cannot be parsed is not copied, and the error names it; restore the named file from a backup or delete it to start with a new key (SPEC FK3)
+* Web: `/metrics` request metrics label `path` with the route that answered, such as `GET /api/nodes/:id`, or `static`, `metrics` or `unmatched`, instead of the request path; 8000 requests to distinct missing paths no longer add 8000 series (SPEC PG1)
+* Web: `/metrics` builds the per-node gauges from the stored node rows at each scrape: a renamed node keeps one `meshtastic_node` series, and after a restart every reported node is exported, not only the 1000 heard most recently in the last 7 days (SPEC PG2)
+* Web: `/metrics` exports no latitude, longitude or altitude for a position off the globe or at `(0, 0)` (SPEC PG3)
+* Web: `meshtastic_nodes` counts every node `/api/nodes` lists instead of stopping at 1000, and a node upload no longer reads 1000 node rows to set it (SPEC PG4)
 * Web: the node overlay takes neighbor and trace-hop roles and names from the dashboard's nodes instead of fetching the node list and each neighbor: 8 requests instead of 136 for a node with 127 neighbors (SPEC OV1/OV2)
 * Web: a node page fetches the node list beside the node's own reads and looks up only the neighbors and trace hops the list lacks: 9 requests instead of 136 for the same node (SPEC OV3)
 * Data: with `DEBUG=1`, `ignored-meshtastic.txt` keeps only each dropped packet's reason, port, channel, sender and recipient ids, packet id, receive time and payload size, no longer its text, payload or coordinates, and rotates at 10 MB, keeping one previous file; `ignored-meshcore.txt` rotates the same way (SPEC DC1-DC2)

PROMETHEUS.md

@@ -8,9 +8,9 @@ PotatoMesh exposes runtime telemetry at `/metrics` for Prometheus scraping.
 ## Runtime integration
 
 No configuration required. `/metrics` is served automatically on the same
-port as the dashboard as soon as the web app runs. Gauges are seeded from the
-database at startup and updated in near real time as `POST` ingest requests
-arrive.
+port as the dashboard as soon as the web app runs. Each scrape reads the
+per-node gauges from the stored node rows. `meshtastic_nodes` is set at startup
+and on each `POST /api/nodes`.
 
 ## Selecting which nodes are exported
 
@@ -21,15 +21,15 @@ time series):
 - `PROM_REPORT_IDS=*`: export metrics for every node.
 - `PROM_REPORT_IDS='!abcd1234,!0a2f0001'`: export metrics for the listed node ids only. Ids are `!` plus 8 lowercase hex digits, as `/api/nodes` shows them. Quote the value; shells and YAML treat `!` specially.
 
-Applies to both the initial refresh and incremental updates. Opted-out nodes are never exported, and with `PRIVATE=1` neither are hidden clients.
+Opted-out nodes are never exported, and with `PRIVATE=1` neither are hidden clients.
 
 ## Available metrics
 
 | Metric name | Type | Labels | Description |
 | --- | --- | --- | --- |
 | `meshtastic_messages_total` | Counter | _none_ | Increments each time the ingest pipeline accepts a new message payload. |
-| `meshtastic_nodes` | Gauge | _none_ | Tracks the number of nodes currently stored in the database. |
-| `meshtastic_node` | Gauge | `node`, `short_name`, `long_name`, `hw_model`, `role` | Reports a node as present (value `1`) along with identity metadata. |
+| `meshtastic_nodes` | Gauge | _none_ | Number of nodes heard in the last 7 days, without opted-out nodes and, with `PRIVATE=1`, hidden clients: the nodes `/api/nodes` lists, with no 1000-node cap. |
+| `meshtastic_node` | Gauge | `node`, `short_name`, `long_name`, `hw_model`, `role` | Reports a node as present (value `1`) with its current names, hardware model and role. One series per node. |
 | `meshtastic_node_battery_level` | Gauge | `node` | Most recent battery percentage reported by the node. |
 | `meshtastic_node_voltage` | Gauge | `node` | Most recent battery voltage reading. |
 | `meshtastic_node_uptime_seconds` | Gauge | `node` | Uptime reported by the device in seconds. |
@@ -38,10 +38,15 @@ Applies to both the initial refresh and incremental updates. Opted-out nodes are
 | `meshtastic_node_latitude` | Gauge | `node` | Latitude component of the last known position. |
 | `meshtastic_node_longitude` | Gauge | `node` | Longitude component of the last known position. |
 | `meshtastic_node_altitude` | Gauge | `node` | Altitude (in metres) of the last known position. |
+| `http_server_requests_total` | Counter | `code`, `method`, `path` | Requests answered. `path` is the route that answered, for example `GET /api/nodes/:id` or `GET /map`, also when it answers 404; `static` for a static file; `metrics` for `/metrics`; `unmatched` for any other request, such as an unknown path. |
+| `http_server_request_duration_seconds` | Histogram | `method`, `path` | Response time, with the same `path` values. |
+| `http_server_exceptions_total` | Counter | `exception` | Exceptions raised while answering, by class. |
 
 Per-node gauges are emitted only for ids in `PROM_REPORT_IDS`, and never for
 an opted-out node. A gauge does not appear until the device has sent the
-corresponding telemetry or position update at least once.
+corresponding telemetry or position update at least once. The latitude,
+longitude and altitude gauges do not appear for a position off the globe or at
+`(0, 0)`.
 
 ## Accessing the `/metrics` endpoint
 

web/lib/potato_mesh/application.rb

@@ -246,9 +246,9 @@ module PotatoMesh
       # pin stale JS between commits (SPEC AV6).
       use PotatoMesh::App::AssetCacheControl,
           immutable: APP_VERSION_PINNED
-      use ::Prometheus::Middleware::Collector
-      # The Exporter and the /metrics route print this one view, which drops
-      # per-node series of nodes /api/nodes would not list (SPEC PM1/PM2).
+      use App::Prometheus::RouteCollector # path label: the matched route (SPEC PG1)
+      # The Exporter and the /metrics route print this one view, which builds
+      # the per-node series from the rows /api/nodes would list (SPEC PM1, PG2).
       set :prometheus_export_registry,
           App::Prometheus::ExportRegistry.new(::Prometheus::Client.registry, self)
       use ::Prometheus::Middleware::Exporter, registry: prometheus_export_registry

web/lib/potato_mesh/application/data_processing/node_writes.rb

@@ -432,7 +432,7 @@ module PotatoMesh
       # @param protocol [String] protocol identifier (default +meshtastic+).
       # @return [void]
       def upsert_node(db, node_id, n, protocol: "meshtastic")
-        n = bound_node_payload(n) # SPEC SL3/SL10: bounded, Prometheus labels and gauges included
+        n = bound_node_payload(n) # SPEC SL3/SL10: bounded before any column, which /metrics reads (PG2)
         user = n["user"] || {}
         met = pick_alias(n, "deviceMetrics", "device_metrics") || {}
         pos = n["position"] || {}
@@ -503,13 +503,6 @@ module PotatoMesh
         # record under another key leaves the row's identity as stored.
         key_mismatch = record_under_another_key?(db, node_id, n, protocol)
 
-        # The prometheus helper still receives the raw `pos` so that gauges
-        # not affected by sentinel handling (e.g. precision_bits) keep
-        # updating; the latitude/longitude guards inside +update_prometheus_metrics+
-        # are responsible for skipping sentinel coordinates.  A record under
-        # another key reaches only the telemetry gauges.
-        update_prometheus_metrics(node_id, key_mismatch ? nil : user, role, met, key_mismatch ? nil : pos)
-
         lora_freq = coerce_integer(n["lora_freq"] || n["loraFrequency"])
         modem_preset = string_or_nil(n["modem_preset"] || n["modemPreset"])
         # Synthetic flag: true for placeholder nodes created from channel message
@@ -919,12 +912,6 @@ module PotatoMesh
         precision = coerce_integer(precision_bits)
         snr_val = coerce_float(snr)
 
-        update_prometheus_metrics(node_id, nil, nil, nil, {
-          "latitude" => lat,
-          "longitude" => lon,
-          "altitude" => alt,
-        })
-
         # When a position packet carries real coordinates but no usable
         # `position_time` (either omitted or collapsed by the sentinel guard
         # above), the `excluded.position_time IS NOT NULL` clauses below would
@@ -1058,14 +1045,6 @@ module PotatoMesh
         air_util_tx = coerce_float(metrics[:air_util_tx] || metrics["air_util_tx"])
         uptime = coerce_integer(metrics[:uptime_seconds] || metrics["uptime_seconds"])
 
-        update_prometheus_metrics(node_id, nil, nil, {
-          "batteryLevel" => battery,
-          "voltage" => voltage,
-          "uptimeSeconds" => uptime,
-          "channelUtilization" => channel_util,
-          "airUtilTx" => air_util_tx,
-        }, nil)
-
         assignments = []
         params = []
 

web/lib/potato_mesh/application/prometheus.rb

@@ -14,6 +14,8 @@
 
 # frozen_string_literal: true
 
+require "prometheus/middleware/collector"
+
 module PotatoMesh
   module App
     module Prometheus
@@ -101,168 +103,235 @@ module PotatoMesh
         # Ignore duplicate registrations when the code is reloaded.
       end
 
-      # Update per-node Prometheus gauges for a single node event.
-      #
-      # The method is a no-op when the configured report-ID list is empty or when
-      # +node_id+ does not match an entry in that list.  When the wildcard +*+ is
-      # present all nodes are reported.
-      #
-      # @param node_id [String, nil] canonical node identifier (+!xxxxxxxx+ form).
-      # @param user [Hash, nil] user payload hash containing +shortName+,
-      #   +longName+, and +hwModel+ keys.
-      # @param role [String] node role label; an empty string skips the NODE_GAUGE.
-      # @param met [Hash, nil] device metrics hash containing keys such as
-      #   +batteryLevel+, +voltage+, +uptimeSeconds+, +channelUtilization+, and
-      #   +airUtilTx+.
-      # @param pos [Hash, nil] position payload hash containing +latitude+,
-      #   +longitude+, and +altitude+.
-      # @return [void]
-      def update_prometheus_metrics(node_id, user = nil, role = "", met = nil, pos = nil)
-        ids = prom_report_ids
-        return if ids.empty? || !node_id
+      # Per-node gauges that read one node-row column each, mapped to that
+      # column (SPEC PG2).  {#prometheus_node_series} builds the presence
+      # gauge and the coordinate gauges itself.
+      NODE_COLUMN_GAUGES = {
+        NODE_BATTERY_LEVEL => "battery_level",
+        NODE_VOLTAGE => "voltage",
+        NODE_UPTIME => "uptime_seconds",
+        NODE_CHANNEL_UTIL => "channel_utilization",
+        NODE_AIR_UTIL_TX => "air_util_tx",
+      }.freeze
 
-        return unless ids[0] == "*" || ids.include?(node_id)
+      # Node-row columns one scrape reads (SPEC PG2).
+      NODE_SERIES_COLUMNS = %w[
+        node_id short_name long_name hw_model role latitude longitude altitude
+        battery_level voltage uptime_seconds channel_utilization air_util_tx
+      ].freeze
 
-        if user && user.is_a?(Hash) && role && role != ""
-          NODE_GAUGE.set(
-            1,
-            labels: {
-              node: node_id,
-              short_name: user["shortName"],
-              long_name: user["longName"],
-              hw_model: user["hwModel"],
-              role: role,
-            },
-          )
+      # Role the presence gauge reports for a row without one: the role
+      # +query_nodes+ gives it, so +/metrics+ and +GET /api/nodes+ agree (PG2).
+      DEFAULT_ROLE = "CLIENT"
+
+      # Latitudes on the globe, in degrees, ends included: the bound SPEC IB2
+      # gives +normalize_lat_lon+, mirrored for the coordinate gauges (PG3).
+      LATITUDE_RANGE = (-90.0..90.0)
+
+      # Longitudes on the globe, in degrees, ends included (SPEC IB2, PG3).
+      LONGITUDE_RANGE = (-180.0..180.0)
+
+      # +path+ label of a static file Sinatra's static handler sent (SPEC PG1).
+      STATIC_ROUTE = "static"
+
+      # +path+ label of the Exporter's own response (SPEC PG1).
+      METRICS_ROUTE = "metrics"
+
+      # +path+ label of any other request no Sinatra route answered, such as
+      # an unknown path (SPEC PG1).
+      UNMATCHED_ROUTE = "unmatched"
+
+      # Path the Exporter answers: the gem's default, which +application.rb+
+      # keeps.
+      METRICS_PATH = "/metrics"
+
+      # Rack middleware recording the HTTP request metrics of
+      # +Prometheus::Middleware::Collector+, labelled by route (SPEC PG1).
+      #
+      # The gem labels a request with its raw path and rewrites only UUID and
+      # all-digit segments, so every distinct path (a node id in
+      # +/api/nodes/:id+, any 404) added series that were never removed.  The
+      # metric names and the +code+, +method+ and +exception+ labels are the
+      # gem's.
+      class RouteCollector < ::Prometheus::Middleware::Collector
+        # Readable label of a route Sinatra matched.  Sinatra writes a
+        # regular-expression route as Ruby prints the regexp, slashes escaped
+        # (+GET \/map\/?+); the label unescapes them and drops an optional
+        # trailing slash (+GET /map+).  A string route reads as written
+        # (+GET /api/nodes/:id+).
+        #
+        # @param route [String] +env["sinatra.route"]+, +VERB pattern+.
+        # @return [String] route label.
+        def self.route_label(route)
+          route.gsub("\\/", "/").delete_suffix("/?")
         end
 
-        if met && met.is_a?(Hash)
-          if met["batteryLevel"]
-            NODE_BATTERY_LEVEL.set(met["batteryLevel"], labels: { node: node_id })
-          end
+        protected
 
-          if met["voltage"]
-            NODE_VOLTAGE.set(met["voltage"], labels: { node: node_id })
-          end
+        # The +path+ label of one request: the {.route_label} of the route
+        # Sinatra matched (+env["sinatra.route"]+), also when that route
+        # answers 404; else {STATIC_ROUTE} for a static file, {METRICS_ROUTE}
+        # for the Exporter's own response and {UNMATCHED_ROUTE} for any other
+        # request, all of which come before or without routing.
+        #
+        # @param env [Hash] Rack environment after the application answered.
+        # @return [String] route label.
+        def generate_path(env)
+          route = env["sinatra.route"]
+          return self.class.route_label(route) if route
+          return STATIC_ROUTE if env["sinatra.static_file"]
+          return METRICS_ROUTE if env["PATH_INFO"] == METRICS_PATH
 
-          if met["uptimeSeconds"]
-            NODE_UPTIME.set(met["uptimeSeconds"], labels: { node: node_id })
-          end
-
-          if met["channelUtilization"]
-            NODE_CHANNEL_UTIL.set(met["channelUtilization"], labels: { node: node_id })
-          end
-
-          if met["airUtilTx"]
-            NODE_AIR_UTIL_TX.set(met["airUtilTx"], labels: { node: node_id })
-          end
-        end
-
-        if pos && pos.is_a?(Hash)
-          lat = pos["latitude"]
-          lon = pos["longitude"]
-          # Issue #782: paired ``(0, 0)`` is the Meshtastic "no GPS lock"
-          # sentinel.  In Ruby ``0.0`` is truthy, so the previous
-          # ``if pos["latitude"]`` guard let the gauge be clobbered to 0
-          # on every sentinel nodeinfo.  Skip both gauges when the pair is
-          # sentinel so each retains its last real value.  Single-axis
-          # zero — a legitimate equator / prime-meridian fix — survives.
-          is_null_island = lat.is_a?(Numeric) && lon.is_a?(Numeric) &&
-                           lat.abs < 1e-9 && lon.abs < 1e-9
-          unless is_null_island
-            NODE_LATITUDE.set(lat, labels: { node: node_id }) if lat
-            NODE_LONGITUDE.set(lon, labels: { node: node_id }) if lon
-          end
-
-          if pos["altitude"]
-            NODE_ALTITUDE.set(pos["altitude"], labels: { node: node_id })
-          end
+          UNMATCHED_ROUTE
         end
       end
 
-      # Refresh all Prometheus node metrics from the current database snapshot.
+      # Number of nodes +GET /api/nodes+ lists, without its row cap (SPEC PG4).
       #
-      # Queries up to 1 000 nodes and updates the {NODES_GAUGE} with the total
-      # count.  For each node that matches the report-ID filter the per-node
-      # gauges are refreshed via {#update_prometheus_metrics}.
+      # The filters of the bulk read in +query_nodes+: heard since
+      # +node_window_floor+ (seven days), neither name carrying the opt-out
+      # marker and, with +PRIVATE=1+, a role other than +CLIENT_HIDDEN+.
+      #
+      # @param db [SQLite3::Database, nil] open handle to count on; nil opens
+      #   and closes a read-only one.
+      # @param now [Integer] reference unix time.
+      # @return [Integer] node count.
+      def prometheus_node_count(db = nil, now: Time.now.to_i)
+        handle = db || open_database(readonly: true)
+        where_clauses = ["last_heard >= ?"]
+        params = [node_window_floor(nil, now)]
+        where_clauses << hidden_client_filter if private_mode?
+        append_opt_out_filter(where_clauses, params, opt_out_self_filter)
+        handle.get_first_value("SELECT COUNT(*) FROM nodes WHERE #{where_clauses.join(" AND ")}", params).to_i
+      ensure
+        handle&.close unless db
+      end
+
+      # Seed {NODES_GAUGE} from the database at boot (SPEC PG4).  The per-node
+      # series need no seeding: every scrape builds them from the node rows
+      # (SPEC PG2).
       #
       # @return [void]
       def update_all_prometheus_metrics_from_nodes
-        nodes = query_nodes(1000)
-
-        NODES_GAUGE.set(nodes.size)
-
-        ids = prom_report_ids
-        unless ids.empty?
-          nodes.each do |n|
-            node_id = n["node_id"]
-
-            next if ids[0] != "*" && !ids.include?(node_id)
-
-            update_prometheus_metrics(
-              node_id,
-              {
-                "shortName" => n["short_name"] || "",
-                "longName" => n["long_name"] || "",
-                "hwModel" => n["hw_model"] || "",
-              },
-              n["role"] || "",
-              {
-                "batteryLevel" => n["battery_level"],
-                "voltage" => n["voltage"],
-                "uptimeSeconds" => n["uptime_seconds"],
-                "channelUtilization" => n["channel_utilization"],
-                "airUtilTx" => n["air_util_tx"],
-              },
-              {
-                "latitude" => n["latitude"],
-                "longitude" => n["longitude"],
-                "altitude" => n["altitude"],
-              },
-            )
-          end
-        end
+        NODES_GAUGE.set(prometheus_node_count)
       end
 
-      # Node ids whose per-node series a scrape may print (SPEC PM1/PM2).
+      # Node rows whose series a scrape exports (SPEC PM1, PM2, PG2).
       #
-      # Runs one read-only query with the filter +GET /api/nodes+ applies:
-      # neither name carries the opt-out marker and, with +PRIVATE=1+, the
-      # role is not +CLIENT_HIDDEN+.  It has no age window and no row cap,
-      # so the gauges keep their meaning; a node whose row retention deleted
-      # is absent, so its series are dropped too.
+      # One read-only query with the filter +GET /api/nodes+ applies: neither
+      # name carries the opt-out marker and, with +PRIVATE=1+, the role is not
+      # +CLIENT_HIDDEN+.  It has no age window and no row cap; a node whose row
+      # retention deleted is absent.  +PROM_REPORT_IDS+ narrows it: +*+ as the
+      # first entry keeps every node, a list keeps the nodes it names, and an
+      # empty setting runs no query.
       #
-      # @return [Set<String>] node ids a scrape may export.
-      def prometheus_visible_node_ids
+      # @param ids [Array<String>] configured report ids.
+      # @return [Array<Hash>] rows holding {NODE_SERIES_COLUMNS}, by node id.
+      def prometheus_node_rows(ids = prom_report_ids)
+        return [] if ids.empty?
+
         db = open_database(readonly: true)
         db.results_as_hash = true
         where_clauses = []
         params = []
         where_clauses << hidden_client_filter if private_mode?
         append_opt_out_filter(where_clauses, params, opt_out_self_filter)
-        sql = "SELECT node_id FROM nodes WHERE #{where_clauses.join(" AND ")}"
-        db.execute(sql, params).each_with_object(Set.new) { |row, ids| ids << row["node_id"] }
+        unless ids[0] == "*"
+          where_clauses << "node_id IN (#{Array.new(ids.length, "?").join(", ")})"
+          params.concat(ids)
+        end
+        db.execute(
+          "SELECT #{NODE_SERIES_COLUMNS.join(", ")} FROM nodes WHERE #{where_clauses.join(" AND ")} ORDER BY node_id",
+          params,
+        )
       ensure
         db&.close
       end
 
-      # Registry view that +/metrics+ prints in place of the registry
-      # (SPEC PM1/PM2).
+      # Per-node series of one scrape, built from {#prometheus_node_rows}
+      # (SPEC PG2, PG3).
       #
-      # Ingest keeps writing every series to the registry, and
-      # prometheus-client 5 cannot remove a label set, so the filter runs at
-      # export.  Each {#metrics} call asks +source+ once for the visible node
-      # ids and keeps a series of a family labelled +node+ only when its node
-      # is in that set.  Families without a +node+ label pass through
-      # unchanged.  If the lookup raises, the scrape carries no per-node
-      # family at all (fail closed).
+      # The series are the reported nodes' own: a rename replaces its node's
+      # presence series, and a node that leaves the rows (opt-out, retention,
+      # +PRIVATE=1+) takes every series with it.  Per row: +meshtastic_node+
+      # 1, labelled with the stored names and hardware model (NULL reads as
+      # empty) and the stored role, {DEFAULT_ROLE} when there is none; one
+      # series for each {NODE_COLUMN_GAUGES} column that holds a number; and,
+      # for a position {#prometheus_coordinates} keeps, its latitude,
+      # longitude and altitude, each when it is a number.
+      #
+      # @return [Hash{Symbol => Hash{Hash => Float}}] label set to value by
+      #   family name; a family without a series is absent.
+      def prometheus_node_series
+        prometheus_node_rows.each_with_object({}) do |row, series|
+          node = { node: row["node_id"] }
+          presence = node.merge(
+            short_name: row["short_name"].to_s,
+            long_name: row["long_name"].to_s,
+            hw_model: row["hw_model"].to_s,
+            role: (row["role"] || DEFAULT_ROLE).to_s,
+          )
+          prometheus_add_series(series, NODE_GAUGE, presence, 1.0)
+          NODE_COLUMN_GAUGES.each do |metric, column|
+            prometheus_add_series(series, metric, node, coerce_float(row[column]))
+          end
+          lat, lon = prometheus_coordinates(row["latitude"], row["longitude"])
+          next if lat.nil? && lon.nil?
+
+          prometheus_add_series(series, NODE_LATITUDE, node, lat)
+          prometheus_add_series(series, NODE_LONGITUDE, node, lon)
+          prometheus_add_series(series, NODE_ALTITUDE, node, coerce_float(row["altitude"]))
+        end
+      end
+
+      # Add one series to a {#prometheus_node_series} result.
+      #
+      # @param series [Hash{Symbol => Hash}] result being built.
+      # @param metric [Prometheus::Client::Metric] family of the series.
+      # @param labels [Hash{Symbol => String}] label set.
+      # @param value [Float, nil] value; nil adds nothing.
+      # @return [void]
+      def prometheus_add_series(series, metric, labels, value)
+        return if value.nil?
+
+        (series[metric.name] ||= {})[labels] = value
+      end
+
+      # Coordinates a node's gauges export (SPEC PG3).
+      #
+      # +normalize_lat_lon+ coerces both axes and drops the #782 +(0, 0)+
+      # sentinel; a latitude outside {LATITUDE_RANGE} or a longitude outside
+      # {LONGITUDE_RANGE} then drops the pair, one axis being enough, as SPEC
+      # IB2 has +normalize_lat_lon+ do.  A single axis on the globe is kept,
+      # as the write path keeps it.
+      #
+      # @param lat [Object] stored latitude.
+      # @param lon [Object] stored longitude.
+      # @return [Array(Float, Float)] +[lat, lon]+, nil on each axis dropped.
+      def prometheus_coordinates(lat, lon)
+        lat_f, lon_f = normalize_lat_lon(lat, lon)
+        return [nil, nil] if (lat_f && !LATITUDE_RANGE.cover?(lat_f)) || (lon_f && !LONGITUDE_RANGE.cover?(lon_f))
+
+        [lat_f, lon_f]
+      end
+
+      # Registry view that +/metrics+ prints in place of the registry
+      # (SPEC PM1, PM2, PG2).
+      #
+      # Nothing writes the per-node gauges: each {#metrics} call asks
+      # +source+ once for the per-node series of the visible, reported nodes,
+      # built from their stored rows, and a family labelled +node+ prints
+      # those series under its own name, type and help.  A family labelled
+      # +node+ that the build does not fill prints none.  Families without a
+      # +node+ label pass through unchanged.  If the lookup raises, the scrape
+      # carries no per-node family at all (fail closed).
       class ExportRegistry
         # Label that marks a metric family as per-node.
         NODE_LABEL = :node
 
-        # @param registry [#metrics] registry the gauges write to.
-        # @param source [#prometheus_visible_node_ids, #warn_log] object that
-        #   runs the visibility query and logs a failed lookup.
+        # @param registry [#metrics] registry the metrics are registered in.
+        # @param source [#prometheus_node_series, #warn_log] object that
+        #   builds the per-node series and logs a failed lookup.
         def initialize(registry, source)
           @registry = registry
           @source = source
@@ -275,22 +344,22 @@ module PotatoMesh
         # @return [Array<#name, #type, #docstring, #values>] families to
         #   print; per-node families are wrapped in {NodeFamily}.
         def metrics
-          visible = visible_node_ids
+          series = node_series
           @registry.metrics.filter_map do |metric|
             next metric unless metric.labels.include?(NODE_LABEL)
 
-            NodeFamily.new(metric, visible) if visible
+            NodeFamily.new(metric, series.fetch(metric.name, {})) if series
           end
         end
 
         private
 
-        # Look up the visible node ids, logging a failure instead of raising.
+        # Build the per-node series, logging a failure instead of raising.
         #
-        # @return [Set<String>, nil] visible node ids, or +nil+ when the
-        #   lookup raised.
-        def visible_node_ids
-          @source.prometheus_visible_node_ids
+        # @return [Hash{Symbol => Hash}, nil] series by family name, or +nil+
+        #   when the lookup raised.
+        def node_series
+          @source.prometheus_node_series
         rescue StandardError => e
           @source.warn_log(
             "Withheld per-node metrics: visible node lookup failed",
@@ -301,16 +370,15 @@ module PotatoMesh
           nil
         end
 
-        # One per-node family limited to the series of visible nodes.  It
-        # offers the readers the text formatter calls (+name+, +type+,
-        # +docstring+, +values+) plus +labels+, and never writes to the
-        # wrapped metric.
+        # One per-node family holding the series one scrape built.  It offers
+        # the readers the text formatter calls (+name+, +type+, +docstring+,
+        # +values+) plus +labels+, and never writes to the wrapped metric.
         class NodeFamily
           # @param metric [Prometheus::Client::Metric] family labelled +node+.
-          # @param visible_node_ids [#include?] node ids whose series stay.
-          def initialize(metric, visible_node_ids)
+          # @param values [Hash{Hash => Float}] label set to value.
+          def initialize(metric, values)
             @metric = metric
-            @visible_node_ids = visible_node_ids
+            @values = values
           end
 
           # @return [Symbol] metric name.
@@ -333,11 +401,11 @@ module PotatoMesh
             @metric.labels
           end
 
-          # Series of visible nodes with their current values.
+          # Series of the family's reported nodes, with their stored values.
           #
-          # @return [Hash{Hash => Object}] label set to value.
+          # @return [Hash{Hash => Float}] label set to value.
           def values
-            @metric.values.select { |label_set, _value| @visible_node_ids.include?(label_set[NODE_LABEL]) }
+            @values
           end
         end
       end

web/lib/potato_mesh/application/routes/ingest.rb

@@ -61,7 +61,7 @@ module PotatoMesh
               per_node = normalize_protocol_value(node["protocol"])
               upsert_node(db, node_id, node, protocol: per_node || batch_protocol)
             end
-            PotatoMesh::App::Prometheus::NODES_GAUGE.set(query_nodes(1000).length)
+            PotatoMesh::App::Prometheus::NODES_GAUGE.set(prometheus_node_count(db))
             PotatoMesh::App::ApiCache.invalidate_prefix("api:nodes:", "api:stats:")
             PotatoMesh::App::PubSub.publish("nodes", private_mode: private_mode?)
             status 201

web/spec/data_processing_spec.rb

@@ -2435,12 +2435,6 @@ RSpec.describe PotatoMesh::App::DataProcessing do
           yield
         end
 
-        def update_prometheus_metrics(*); end
-
-        def prom_report_ids
-          []
-        end
-
         def private_mode?
           false
         end
@@ -2874,12 +2868,6 @@ RSpec.describe PotatoMesh::App::DataProcessing do
           yield
         end
 
-        def update_prometheus_metrics(*); end
-
-        def prom_report_ids
-          []
-        end
-
         def private_mode?
           false
         end
@@ -2981,12 +2969,6 @@ RSpec.describe PotatoMesh::App::DataProcessing do
           yield
         end
 
-        def update_prometheus_metrics(*); end
-
-        def prom_report_ids
-          []
-        end
-
         def private_mode?
           false
         end
@@ -3113,12 +3095,6 @@ RSpec.describe PotatoMesh::App::DataProcessing do
           yield
         end
 
-        def update_prometheus_metrics(*); end
-
-        def prom_report_ids
-          []
-        end
-
         def private_mode?
           false
         end
@@ -3186,12 +3162,6 @@ RSpec.describe PotatoMesh::App::DataProcessing do
           yield
         end
 
-        def update_prometheus_metrics(*); end
-
-        def prom_report_ids
-          []
-        end
-
         def private_mode?
           false
         end

web/spec/field_limits_spec.rb

@@ -20,6 +20,7 @@ require "json"
 require "openssl"
 require "sqlite3"
 require "uri"
+require_relative "support/metrics_spec_helpers"
 require_relative "support/data_processing_harness"
 require_relative "support/federation_identity"
 require_relative "support/ingest_spec_helpers"
@@ -687,6 +688,8 @@ RSpec.describe "Ingest field limits" do
   end
 
   describe "keyed evidence and metric labels" do
+    include MetricsSpecHelpers
+
     it "stores a public key over its cap as NULL, so it is no keyed evidence (MR1)" do
       [["!5c000015", "k" * 512], ["!5c000016", "k" * 513]].each do |node_id, key|
         post_ok("/api/nodes", node_body(node_id, "user" => { "longName" => "Keyed", "publicKey" => key, "role" => "COMPANION" }).merge("protocol" => "meshcore"))
@@ -699,17 +702,20 @@ RSpec.describe "Ingest field limits" do
     end
 
     it "labels the node gauge with the bounded names" do
-      allow_any_instance_of(Sinatra::Application).to receive(:prom_report_ids).and_return(["*"])
-      label_sets = []
-      allow(PotatoMesh::App::Prometheus::NODE_GAUGE).to receive(:set) { |_value, labels: {}| label_sets << labels }
+      allow(PotatoMesh::Config).to receive(:prom_report_id_list).and_return(["*"])
 
       post_ok("/api/nodes", node_body("!5c000017", "user" => { "longName" => "L" * 1_000, "shortName" => "S" * 100, "hwModel" => "H" * 100, "role" => "CLIENT" }))
 
-      expect(label_sets).to eq([{ node: "!5c000017", short_name: "S" * 16, long_name: "L" * 512, hw_model: nil, role: "CLIENT" }])
+      # The series reads the stored row (SPEC PG2), which holds the bounded names.
+      expect(scraped_samples("meshtastic_node", "!5c000017")).to eq(
+        [%(meshtastic_node{node="!5c000017",short_name="#{"S" * 16}",long_name="#{"L" * 512}",hw_model="",role="CLIENT"} 1.0)],
+      )
     end
   end
 
   describe "numeric fields (SL-A7)" do
+    include MetricsSpecHelpers
+
     # 100 kB of text that is no number.
     let(:junk) { "x" * 100_000 }
 
@@ -775,17 +781,16 @@ RSpec.describe "Ingest field limits" do
     end
 
     it "hands the node gauges numbers only" do
-      allow_any_instance_of(Sinatra::Application).to receive(:prom_report_ids).and_return(["*"])
-      gauges = %i[NODE_GAUGE NODE_BATTERY_LEVEL NODE_VOLTAGE NODE_UPTIME NODE_CHANNEL_UTIL NODE_AIR_UTIL_TX NODE_LATITUDE NODE_LONGITUDE NODE_ALTITUDE]
-      gauges.each { |name| allow(PotatoMesh::App::Prometheus.const_get(name)).to receive(:set) }
+      allow(PotatoMesh::Config).to receive(:prom_report_id_list).and_return(["*"])
 
       post_ok("/api/nodes", node_body("!5c000027", "deviceMetrics" => { "batteryLevel" => junk, "voltage" => "4.1" },
                                                    "position" => { "latitude" => 52.5, "longitude" => "13.4", "altitude" => junk }))
 
-      expect(PotatoMesh::App::Prometheus::NODE_BATTERY_LEVEL).not_to have_received(:set)
-      expect(PotatoMesh::App::Prometheus::NODE_ALTITUDE).not_to have_received(:set)
-      expect(PotatoMesh::App::Prometheus::NODE_VOLTAGE).to have_received(:set).with(4.1, labels: { node: "!5c000027" })
-      expect(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to have_received(:set).with(13.4, labels: { node: "!5c000027" })
+      # The gauges read the stored row (SPEC PG2): NULL for text, a number for a numeric string.
+      expect(scraped_samples("meshtastic_node_battery_level", "!5c000027")).to eq([])
+      expect(scraped_samples("meshtastic_node_altitude", "!5c000027")).to eq([])
+      expect(scraped_samples("meshtastic_node_voltage", "!5c000027")).to eq(['meshtastic_node_voltage{node="!5c000027"} 4.1'])
+      expect(scraped_samples("meshtastic_node_longitude", "!5c000027")).to eq(['meshtastic_node_longitude{node="!5c000027"} 13.4'])
     end
 
     it "stores a message's numbers as NULL when they are no number, and converts numeric strings" do

web/spec/metrics_routes_spec.rb

@@ -0,0 +1,289 @@
+# Copyright © 2025-26 l5yth & contributors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# frozen_string_literal: true
+
+require "spec_helper"
+require "prometheus/client"
+require_relative "support/metrics_spec_helpers"
+
+# What +/metrics+ prints, driven through the API with the app's full
+# middleware stack (SPEC PG1-PG4, ACCEPTANCE PG-A1-PG-A4; review items M1,
+# L-PROMLABELS, IB-G1 and M5). Every example reads only what an unfixed tree
+# also has, so the file loads and fails there on its assertions.
+RSpec.describe "/metrics request labels, node series and node count" do
+  include MetricsSpecHelpers
+
+  let(:app) { Sinatra::Application }
+  let(:api_token) { "metrics-routes-token" }
+  let(:auth_headers) do
+    { "CONTENT_TYPE" => "application/json", "HTTP_AUTHORIZATION" => "Bearer #{api_token}" }
+  end
+  let(:now) { Time.now.to_i }
+  let(:marker) { PotatoMesh::Config.node_opt_out_marker }
+
+  around do |example|
+    saved = ENV.to_h.slice("API_TOKEN", "PROM_REPORT_IDS", "PRIVATE")
+    ENV["API_TOKEN"] = api_token
+    ENV["PROM_REPORT_IDS"] = "*"
+    ENV.delete("PRIVATE")
+    Dir.mktmpdir("metrics-routes-spec-") do |dir|
+      RSpec::Mocks.with_temporary_scope do
+        allow(PotatoMesh::Config).to receive(:db_path).and_return(File.join(dir, "mesh.db"))
+        db_helper = Object.new.extend(PotatoMesh::App::Database)
+        db_helper.init_db
+        db_helper.ensure_schema_upgrades
+        PotatoMesh::App::ApiCache.invalidate_all
+        example.run
+      end
+    end
+  ensure
+    %w[API_TOKEN PROM_REPORT_IDS PRIVATE].each do |key|
+      saved.key?(key) ? ENV[key] = saved[key] : ENV.delete(key)
+    end
+  end
+
+  # Run statements against the spec database in one transaction.
+  #
+  # @yieldparam db [SQLite3::Database] open database handle.
+  # @return [void]
+  def with_spec_db
+    db = SQLite3::Database.new(PotatoMesh::Config.db_path)
+    db.transaction { yield db }
+  ensure
+    db&.close
+  end
+
+  # Store a node row directly, as one written before this process started.
+  #
+  # @param db [SQLite3::Database] open database handle.
+  # @param node_id [String] canonical node id.
+  # @param heard [Integer] last-heard time.
+  # @param columns [Hash{Symbol => Object}] further columns and their values.
+  # @return [void]
+  def store_node(db, node_id, heard:, **columns)
+    row = { "node_id" => node_id, "long_name" => "Stored #{node_id}", "short_name" => "ST", "role" => "CLIENT",
+            "last_heard" => heard, "first_heard" => heard }.merge(columns.transform_keys(&:to_s))
+    db.execute("INSERT INTO nodes(#{row.keys.join(", ")}) VALUES (#{(["?"] * row.size).join(", ")})", row.values)
+  end
+
+  # One node entry of a +POST /api/nodes+ body.
+  #
+  # @param long_name [String] display name.
+  # @param heard [Integer] lastHeard, so a later entry wins the upsert guard.
+  # @param position [Hash, nil] position section.
+  # @return [Hash] node entry.
+  def node_entry(long_name, heard, position: nil)
+    entry = {
+      "user" => { "longName" => long_name, "shortName" => "PG", "hwModel" => "TBEAM", "role" => "CLIENT" },
+      "deviceMetrics" => { "batteryLevel" => 80, "voltage" => 4.0 },
+      "lastHeard" => heard,
+    }
+    entry["position"] = position.merge("time" => heard) if position
+    entry
+  end
+
+  # POST nodes and expect the route to accept them.
+  #
+  # @param entries [Hash{String => Hash}] node id to node entry.
+  # @return [void]
+  def post_nodes(entries)
+    post "/api/nodes", entries.to_json, auth_headers
+    expect(last_response.status).to eq(201), "POST /api/nodes answered #{last_response.status}: #{last_response.body}"
+  end
+
+  # The HTTP request counter the request-metrics middleware registered.
+  #
+  # @return [Prometheus::Client::Counter] +http_server_requests_total+.
+  def request_counter
+    ::Prometheus::Client.registry.get(:http_server_requests_total)
+  end
+
+  # Distinct +path+ values of the request counter and duration histogram.
+  #
+  # @return [Array<String>] path label values recorded so far.
+  def request_paths
+    %i[http_server_requests_total http_server_request_duration_seconds].flat_map do |name|
+      ::Prometheus::Client.registry.get(name).values.keys.map { |labels| labels[:path] }
+    end.uniq
+  end
+
+  # Every label the collector can give a request: one per route of
+  # Sinatra's table, and the three fixed labels.
+  #
+  # @return [Array<String>] the route labels, then +static+, +metrics+ and
+  #   +unmatched+.
+  def path_labels
+    routes = PotatoMesh::Application.routes.flat_map do |verb, table|
+      table.map { |pattern, _conditions, _block| PotatoMesh::App::Prometheus::RouteCollector.route_label("#{verb} #{pattern}") }
+    end
+    routes + %w[static metrics unmatched]
+  end
+
+  describe "HTTP request metrics (SPEC PG1)" do
+    it "labels a per-node read with its route, never with the node id from the URL" do
+      get "/api/nodes/!0b0b0001"
+      expect(last_response.status).to eq(404)
+
+      expect(request_paths).to include("GET /api/nodes/:id")
+      expect(request_paths.grep(/0b0b0001/)).to eq([])
+      expect(request_counter.get(labels: { code: "404", method: "get", path: "GET /api/nodes/:id" })).to be >= 1
+    end
+
+    it "keeps 1000 distinct 404 paths to one series, every label a route's or a fixed one" do
+      get "/version"
+      before = request_counter.get(labels: { code: "404", method: "get", path: "unmatched" })
+
+      # Letters keep each segment from the gem's all-digit rewrite.
+      1000.times { |index| get "/pg-missing/n#{index}" }
+
+      expect(last_response.status).to eq(404)
+      paths = request_paths
+      expect(paths.grep(%r{/pg-missing/})).to eq([])
+      expect(paths - path_labels).to eq([])
+      expect(request_counter.get(labels: { code: "404", method: "get", path: "unmatched" }) - before).to eq(1000)
+    end
+
+    it "labels a static file static and the Exporter's own /metrics answer metrics" do
+      get "/version"
+      counts = lambda do
+        %w[static metrics unmatched].to_h { |path| [path, request_counter.get(labels: { code: "200", method: "get", path: path })] }
+      end
+      before = counts.call
+
+      get "/potatomesh-logo.svg"
+      expect(last_response.status).to eq(200)
+      get "/metrics"
+      expect(last_response.status).to eq(200)
+
+      expect(counts.call.to_h { |path, count| [path, count - before[path]] }).to eq("static" => 1.0, "metrics" => 1.0, "unmatched" => 0.0)
+      expect(request_paths).not_to include("/potatomesh-logo.svg", "/metrics")
+    end
+
+    it "labels a regular-expression route readably, with or without its trailing slash" do
+      get "/map"
+      get "/map/"
+      expect(last_response.status).to eq(200)
+
+      expect(request_counter.get(labels: { code: "200", method: "get", path: "GET /map" })).to be >= 2
+      expect(request_paths.grep(/\\/)).to eq([])
+    end
+  end
+
+  describe "per-node series built from the node rows (SPEC PG2)" do
+    it "replaces a renamed node's presence series instead of adding one" do
+      node = "!0b0b0002"
+      post_nodes(node => node_entry("First Name", now - 60))
+      post_nodes(node => node_entry("Second Name", now - 30))
+
+      expect(scraped_samples("meshtastic_node", node)).to eq(
+        [%(meshtastic_node{node="#{node}",short_name="PG",long_name="Second Name",hw_model="TBEAM",role="CLIENT"} 1.0)],
+      )
+    end
+
+    it "exports a reported node from its stored row, also one this process never ingested" do
+      node = "!0b0b0003"
+      with_spec_db do |db|
+        store_node(db, node, heard: now - 40 * 86_400, battery_level: 77, voltage: 3.9, uptime_seconds: 600,
+                             channel_utilization: 12.5, air_util_tx: 0.75, latitude: 48.1, longitude: 11.5, altitude: 520)
+      end
+
+      expect(scraped_samples("meshtastic_node_battery_level", node)).to eq([%(meshtastic_node_battery_level{node="#{node}"} 77.0)])
+      expect(scraped_samples("meshtastic_node_uptime_seconds", node)).to eq([%(meshtastic_node_uptime_seconds{node="#{node}"} 600.0)])
+      expect(scraped_samples("meshtastic_node_altitude", node)).to eq([%(meshtastic_node_altitude{node="#{node}"} 520.0)])
+      expect(scraped_samples("meshtastic_node", node).length).to eq(1)
+    end
+
+    it "follows the row when a telemetry or position packet updates it" do
+      node = "!0b0b0004"
+      post_nodes(node => node_entry("Mover", now - 60, position: { "latitude" => 52.5, "longitude" => 13.4 }))
+      post "/api/telemetry", [{ id: 940_001, node_id: node, rx_time: now - 20, battery_level: 55, voltage: 3.7 }].to_json, auth_headers
+      post "/api/positions", [{ id: 940_002, node_id: node, rx_time: now - 10, latitude: 48.1, longitude: 11.5, altitude: 520 }].to_json, auth_headers
+
+      expect(scraped_samples("meshtastic_node_battery_level", node)).to eq([%(meshtastic_node_battery_level{node="#{node}"} 55.0)])
+      expect(scraped_samples("meshtastic_node_latitude", node)).to eq([%(meshtastic_node_latitude{node="#{node}"} 48.1)])
+      expect(scraped_samples("meshtastic_node_longitude", node)).to eq([%(meshtastic_node_longitude{node="#{node}"} 11.5)])
+    end
+  end
+
+  describe "coordinate gauges (SPEC PG3)" do
+    it "exports no coordinate gauge for a posted off-globe position" do
+      far = "!0b0b0005"
+      near = "!0b0b0006"
+      post_nodes(
+        far => node_entry("Far", now - 30, position: { "latitude" => 95.0, "longitude" => 190.0, "altitude" => 40 }),
+        near => node_entry("Near", now - 30, position: { "latitude" => 52.5, "longitude" => 13.4, "altitude" => 40 }),
+      )
+
+      %w[latitude longitude altitude].each do |axis|
+        expect(scraped_samples("meshtastic_node_#{axis}", far)).to eq([]), "#{axis} of the off-globe node was exported"
+      end
+      expect(scraped_samples("meshtastic_node_latitude", near)).to eq([%(meshtastic_node_latitude{node="#{near}"} 52.5)])
+      expect(scraped_samples("meshtastic_node_altitude", near)).to eq([%(meshtastic_node_altitude{node="#{near}"} 40.0)])
+    end
+
+    it "exports no coordinate gauge for a stored off-globe row or the (0, 0) sentinel, and keeps the globe's edges" do
+      with_spec_db do |db|
+        store_node(db, "!0b0b0007", heard: now, latitude: 398.761944, longitude: 332.909167, altitude: 10)
+        store_node(db, "!0b0b0008", heard: now, latitude: 0.0, longitude: 0.0, altitude: 10)
+        store_node(db, "!0b0b0009", heard: now, latitude: 52.5, longitude: 200.0, altitude: 10)
+        store_node(db, "!0b0b000a", heard: now, latitude: -90.0, longitude: 180.0, altitude: 10)
+      end
+
+      %w[!0b0b0007 !0b0b0008 !0b0b0009].each do |node|
+        %w[latitude longitude altitude].each do |axis|
+          expect(scraped_samples("meshtastic_node_#{axis}", node)).to eq([]), "#{axis} of #{node} was exported"
+        end
+        expect(scraped_samples("meshtastic_node", node).length).to eq(1)
+      end
+      expect(scraped_samples("meshtastic_node_latitude", "!0b0b000a")).to eq([%(meshtastic_node_latitude{node="!0b0b000a"} -90.0)])
+      expect(scraped_samples("meshtastic_node_longitude", "!0b0b000a")).to eq([%(meshtastic_node_longitude{node="!0b0b000a"} 180.0)])
+      expect(scraped_samples("meshtastic_node_altitude", "!0b0b000a")).to eq([%(meshtastic_node_altitude{node="!0b0b000a"} 10.0)])
+    end
+  end
+
+  describe "node count gauge (SPEC PG4)" do
+    it "counts every node GET /api/nodes lists, past its 1000-row cap" do
+      with_spec_db do |db|
+        3000.times { |index| store_node(db, format("!0c%06x", index), heard: now - 60 - index) }
+      end
+
+      post_nodes("!0b0b0fff" => node_entry("Trigger", now))
+
+      expect(scraped_samples("meshtastic_nodes")).to eq(["meshtastic_nodes 3001.0"])
+    end
+
+    it "counts with the filters of GET /api/nodes and seeds the gauge at boot" do
+      with_spec_db do |db|
+        store_node(db, "!0b0b1001", heard: now - 60)
+        store_node(db, "!0b0b1002", heard: now - 60, long_name: "Quiet #{marker} Station")
+        store_node(db, "!0b0b1003", heard: now - 60, short_name: "Q#{marker}")
+        store_node(db, "!0b0b1004", heard: now - 8 * 86_400)
+        store_node(db, "!0b0b1005", heard: now - 60, role: "CLIENT_HIDDEN")
+      end
+
+      PotatoMesh::Application.update_all_prometheus_metrics_from_nodes
+      expect(scraped_samples("meshtastic_nodes")).to eq(["meshtastic_nodes 2.0"])
+      get "/api/nodes?limit=1000"
+      expect(JSON.parse(last_response.body).length).to eq(2)
+
+      ENV["PRIVATE"] = "1"
+      PotatoMesh::App::ApiCache.invalidate_all
+      post_nodes("!0b0b1006" => node_entry("Private Trigger", now))
+      expect(scraped_samples("meshtastic_nodes")).to eq(["meshtastic_nodes 2.0"])
+      get "/api/nodes?limit=1000"
+      expect(JSON.parse(last_response.body).length).to eq(2)
+    end
+  end
+end

web/spec/node_identity_binding_spec.rb

@@ -104,14 +104,13 @@ RSpec.describe "Node identity binding" do
     end
 
     it "keeps names, role, hardware model, key and position against a record under another key" do
-      allow(dp).to receive(:update_prometheus_metrics)
       db = open_db
       seed_owner(db)
       post_profile(db, key: k2)
       row = node_row(db, id)
       db.close
-      # The record's profile and position reach no metric; its telemetry does.
-      expect(dp).to have_received(:update_prometheus_metrics).with(id, nil, "CLIENT", { "batteryLevel" => 42 }, nil)
+      # /metrics reads this row (SPEC PG2), so the record's profile and
+      # position reach no metric; its telemetry does.
       expect(row.slice("long_name", "short_name", "role", "hw_model", "public_key", "latitude", "last_advert_heard")).to eq(
         "long_name" => "Owner", "short_name" => "OWNR", "role" => "ROUTER", "hw_model" => "RAK4631",
         "public_key" => k1, "latitude" => 52.52, "last_advert_heard" => now - 600,

web/spec/prometheus_spec.rb

@@ -16,8 +16,11 @@
 
 require "spec_helper"
 require "prometheus/client"
+require_relative "support/metrics_spec_helpers"
 
 RSpec.describe PotatoMesh::App::Prometheus do
+  include MetricsSpecHelpers
+
   # Build a host class mixing in the module so we can call instance methods.
   let(:harness_class) do
     Class.new do
@@ -54,8 +57,6 @@ RSpec.describe PotatoMesh::App::Prometheus do
         yield
       end
 
-      def update_prometheus_metrics(*); end
-
       def resolve_protocol(_db, _ingestor, cache: nil)
         "meshtastic"
       end
@@ -109,18 +110,6 @@ RSpec.describe PotatoMesh::App::Prometheus do
     )
   end
 
-  # Sample lines labelled with one node id, comments excluded.
-  #
-  # @param text [String] Prometheus text exposition.
-  # @param node_id [String] canonical node id.
-  # @return [Array<String>] exposition lines for that node.
-  def node_samples(text, node_id)
-    text.each_line
-      .reject { |line| line.start_with?("#") }
-      .select { |line| line.include?(%(node="#{node_id}")) }
-      .map(&:chomp)
-  end
-
   # ---------------------------------------------------------------------------
   # Module-level metric constants
   # ---------------------------------------------------------------------------
@@ -147,197 +136,229 @@ RSpec.describe PotatoMesh::App::Prometheus do
       expect(PotatoMesh::App::Prometheus::METRICS).to be_an(Array)
       expect(PotatoMesh::App::Prometheus::METRICS).not_to be_empty
     end
+
+    it "reads the column of every column gauge" do
+      columns = PotatoMesh::App::Prometheus::NODE_COLUMN_GAUGES.values
+      expect(PotatoMesh::App::Prometheus::NODE_SERIES_COLUMNS).to include(*columns)
+    end
+
+    it "builds every per-node family of METRICS" do
+      built = PotatoMesh::App::Prometheus::NODE_COLUMN_GAUGES.keys + [
+        PotatoMesh::App::Prometheus::NODE_GAUGE,
+        PotatoMesh::App::Prometheus::NODE_LATITUDE,
+        PotatoMesh::App::Prometheus::NODE_LONGITUDE,
+        PotatoMesh::App::Prometheus::NODE_ALTITUDE,
+      ]
+      expect(built).to match_array(PotatoMesh::App::Prometheus::METRICS.select { |metric| metric.labels.include?(:node) })
+    end
   end
 
   # ---------------------------------------------------------------------------
-  # update_prometheus_metrics
+  # RouteCollector: the request metrics labelled by route (SPEC PG1)
   # ---------------------------------------------------------------------------
-  describe "#update_prometheus_metrics" do
-    # Re-include the real implementation so we can test it.
-    let(:real_class) do
-      Class.new do
-        include PotatoMesh::App::Prometheus
+  describe PotatoMesh::App::Prometheus::RouteCollector do
+    # A private registry keeps these series out of the global one.
+    let(:registry) { ::Prometheus::Client::Registry.new }
 
-        def prom_report_ids
-          ["*"]
+    # Send GETs through one collector around +inner+.
+    #
+    # @param inner [#call] Rack application the collector wraps.
+    # @param paths [Array<String>] request paths, in order.
+    # @return [void]
+    def collect(inner, *paths)
+      collector = described_class.new(inner, registry: registry)
+      paths.each { |path| collector.call(Rack::MockRequest.env_for(path)) }
+    end
+
+    it "labels a request with the route Sinatra matched" do
+      routed = lambda do |env|
+        env["sinatra.route"] = "GET /api/nodes/:id"
+        [200, {}, []]
+      end
+
+      collect(routed, "/api/nodes/!0a2f0401")
+
+      expect(registry.get(:http_server_requests_total).values.keys).to eq([{ code: "200", method: "get", path: "GET /api/nodes/:id" }])
+      expect(registry.get(:http_server_request_duration_seconds).values.keys).to eq([{ method: "get", path: "GET /api/nodes/:id" }])
+    end
+
+    it "labels a static file static, the Exporter's path metrics and any other unrouted request unmatched" do
+      inner = lambda do |env|
+        case env["PATH_INFO"]
+        when "/potatomesh-logo.svg"
+          # Sinatra's static handler names the file it sends.
+          env["sinatra.static_file"] = "/srv/public/potatomesh-logo.svg"
+          [200, {}, []]
+        when "/metrics" then [200, {}, []]
+        else [404, {}, []]
         end
       end
+      collect(inner, "/potatomesh-logo.svg", "/metrics", "/pg-nowhere/0a2f0402")
+
+      expect(registry.get(:http_server_requests_total).values.keys).to eq(
+        [
+          { code: "200", method: "get", path: "static" },
+          { code: "200", method: "get", path: "metrics" },
+          { code: "404", method: "get", path: "unmatched" },
+        ],
+      )
+    end
+
+    it "makes a regular-expression route readable and leaves a string route as written" do
+      expect(described_class.route_label("GET \\/map\\/?")).to eq("GET /map")
+      expect(described_class.route_label("GET /api/nodes/:id")).to eq("GET /api/nodes/:id")
+      expect(described_class.route_label("GET /")).to eq("GET /")
+    end
+
+    it "gives every route of the app its own readable label" do
+      raw = PotatoMesh::Application.routes.flat_map do |verb, table|
+        table.map { |pattern, _conditions, _block| "#{verb} #{pattern}" }
+      end
+      labels = raw.map { |route| described_class.route_label(route) }
+
+      expect(labels.uniq.length).to eq(raw.length)
+      expect(labels.grep(/\\/)).to eq([])
+      expect(labels).to include("GET /map", "HEAD /map", "GET /nodes", "GET /nodes/:id", "GET /federation")
+      expect(labels & %w[static metrics unmatched]).to eq([])
+    end
+
+    it "keeps the gem's metric names and labels" do
+      collect(->(_env) { [200, {}, []] }, "/")
+
+      expect(registry.metrics.map { |metric| [metric.name, metric.labels] }).to eq(
+        [
+          [:http_server_requests_total, %i[code method path]],
+          [:http_server_request_duration_seconds, %i[method path]],
+          [:http_server_exceptions_total, [:exception]],
+        ],
+      )
+    end
+  end
+
+  # ---------------------------------------------------------------------------
+  # prometheus_coordinates (SPEC PG3)
+  # ---------------------------------------------------------------------------
+  describe "#prometheus_coordinates" do
+    it "keeps a pair on the globe, its edges included" do
+      expect(prometheus.prometheus_coordinates(52.5, 13.4)).to eq([52.5, 13.4])
+      expect(prometheus.prometheus_coordinates(90, -180)).to eq([90.0, -180.0])
+      expect(prometheus.prometheus_coordinates(-90.0, 180.0)).to eq([-90.0, 180.0])
+    end
+
+    it "drops the pair when either axis is off the globe, also with the other axis absent" do
+      [[90.0001, 13.4], [52.5, -180.5], [398.761944, 332.909167], [214.7483647, -214.7483647], [95.0, nil], [nil, 190.0]].each do |lat, lon|
+        expect(prometheus.prometheus_coordinates(lat, lon)).to eq([nil, nil]), "kept (#{lat.inspect}, #{lon.inspect})"
+      end
     end
 
-    subject(:prom_obj) { real_class.new }
-
-    it "is a no-op when ids list is empty" do
-      allow(prom_obj).to receive(:prom_report_ids).and_return([])
-      expect { prom_obj.update_prometheus_metrics("!aabb1234") }.not_to raise_error
+    # Issue #782: a paired ``(0, 0)`` is the Meshtastic "no GPS lock"
+    # sentinel, while a single-axis zero is a fix on the equator or the prime
+    # meridian.
+    it "drops the (0, 0) sentinel and keeps a single-axis zero" do
+      expect(prometheus.prometheus_coordinates(0.0, 0.0)).to eq([nil, nil])
+      expect(prometheus.prometheus_coordinates(0.0, 13.4)).to eq([0.0, 13.4])
+      expect(prometheus.prometheus_coordinates(52.5, 0.0)).to eq([52.5, 0.0])
     end
 
-    it "is a no-op when node_id is nil" do
-      expect { prom_obj.update_prometheus_metrics(nil) }.not_to raise_error
-    end
-
-    it "skips when node is not in the allowed id list" do
-      allow(prom_obj).to receive(:prom_report_ids).and_return(["!other"])
-      expect(PotatoMesh::App::Prometheus::NODE_GAUGE).not_to receive(:set)
-      prom_obj.update_prometheus_metrics("!aabb1234")
-    end
-
-    it "sets NODE_GAUGE when user data and role are present" do
-      allow(PotatoMesh::App::Prometheus::NODE_GAUGE).to receive(:set)
-      prom_obj.update_prometheus_metrics(
-        "!aabb1234",
-        { "shortName" => "T", "longName" => "Test", "hwModel" => "TBEAM" },
-        "CLIENT",
-      )
-      expect(PotatoMesh::App::Prometheus::NODE_GAUGE).to have_received(:set).once
-    end
-
-    it "sets battery level gauge when provided" do
-      allow(PotatoMesh::App::Prometheus::NODE_BATTERY_LEVEL).to receive(:set)
-      prom_obj.update_prometheus_metrics(
-        "!aabb1234",
-        nil,
-        "",
-        { "batteryLevel" => 75 },
-      )
-      expect(PotatoMesh::App::Prometheus::NODE_BATTERY_LEVEL).to have_received(:set).with(75, labels: { node: "!aabb1234" })
-    end
-
-    it "sets latitude/longitude when position is present" do
-      allow(PotatoMesh::App::Prometheus::NODE_LATITUDE).to receive(:set)
-      allow(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to receive(:set)
-      prom_obj.update_prometheus_metrics(
-        "!aabb1234",
-        nil,
-        "",
-        nil,
-        { "latitude" => 52.0, "longitude" => 13.0 },
-      )
-      expect(PotatoMesh::App::Prometheus::NODE_LATITUDE).to have_received(:set).with(52.0, labels: { node: "!aabb1234" })
-      expect(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to have_received(:set).with(13.0, labels: { node: "!aabb1234" })
-    end
-
-    # Issue #782: a Meshtastic node without a GPS lock emits ``(0, 0)`` on
-    # every nodeinfo.  The previous truthy-zero guard clobbered NODE_LATITUDE
-    # and NODE_LONGITUDE to 0 on each update; after the fix the gauges retain
-    # their last real value and the sentinel is silently ignored.
-    it "skips the lat/lon gauges for the (0, 0) Null Island sentinel" do
-      allow(PotatoMesh::App::Prometheus::NODE_LATITUDE).to receive(:set)
-      allow(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to receive(:set)
-      prom_obj.update_prometheus_metrics(
-        "!aabb1234",
-        nil,
-        "",
-        nil,
-        { "latitude" => 0.0, "longitude" => 0.0 },
-      )
-      expect(PotatoMesh::App::Prometheus::NODE_LATITUDE).not_to have_received(:set)
-      expect(PotatoMesh::App::Prometheus::NODE_LONGITUDE).not_to have_received(:set)
-    end
-
-    it "preserves an equator fix (lat=0, lon!=0)" do
-      allow(PotatoMesh::App::Prometheus::NODE_LATITUDE).to receive(:set)
-      allow(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to receive(:set)
-      prom_obj.update_prometheus_metrics(
-        "!aabb1234",
-        nil,
-        "",
-        nil,
-        { "latitude" => 0.0, "longitude" => 13.4 },
-      )
-      expect(PotatoMesh::App::Prometheus::NODE_LATITUDE).to have_received(:set).with(0.0, labels: { node: "!aabb1234" })
-      expect(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to have_received(:set).with(13.4, labels: { node: "!aabb1234" })
-    end
-
-    it "preserves a prime-meridian fix (lat!=0, lon=0)" do
-      allow(PotatoMesh::App::Prometheus::NODE_LATITUDE).to receive(:set)
-      allow(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to receive(:set)
-      prom_obj.update_prometheus_metrics(
-        "!aabb1234",
-        nil,
-        "",
-        nil,
-        { "latitude" => 52.5, "longitude" => 0.0 },
-      )
-      expect(PotatoMesh::App::Prometheus::NODE_LATITUDE).to have_received(:set).with(52.5, labels: { node: "!aabb1234" })
-      expect(PotatoMesh::App::Prometheus::NODE_LONGITUDE).to have_received(:set).with(0.0, labels: { node: "!aabb1234" })
+    it "keeps a single axis on the globe and drops an axis that is no number" do
+      expect(prometheus.prometheus_coordinates(52.5, nil)).to eq([52.5, nil])
+      expect(prometheus.prometheus_coordinates("52.5", "east")).to eq([52.5, nil])
+      expect(prometheus.prometheus_coordinates(Float::INFINITY, 13.4)).to eq([nil, 13.4])
     end
   end
 
   # ---------------------------------------------------------------------------
-  # update_all_prometheus_metrics_from_nodes
+  # prometheus_node_count and the boot seed (SPEC PG4)
   # ---------------------------------------------------------------------------
-  describe "#update_all_prometheus_metrics_from_nodes" do
-    it "sets NODES_GAUGE to the count of returned nodes" do
-      nodes = [
-        { "node_id" => "!aabb1234", "short_name" => "A", "long_name" => "Alpha", "hw_model" => "TBEAM", "role" => "CLIENT" },
-      ]
-      allow(prometheus).to receive(:query_nodes).and_return(nodes)
-      allow(prometheus).to receive(:update_prometheus_metrics)
-      allow(PotatoMesh::App::Prometheus::NODES_GAUGE).to receive(:set)
-
-      prometheus.update_all_prometheus_metrics_from_nodes
-
-      expect(PotatoMesh::App::Prometheus::NODES_GAUGE).to have_received(:set).with(1)
-    end
-
-    it "iterates over all nodes when prom_report_ids includes wildcard" do
-      nodes = [
-        { "node_id" => "!aabb1234", "short_name" => "A", "long_name" => "Alpha", "hw_model" => "TBEAM", "role" => "CLIENT" },
-      ]
-      allow(prometheus).to receive(:query_nodes).and_return(nodes)
-      allow(prometheus).to receive(:update_prometheus_metrics)
-      allow(PotatoMesh::App::Prometheus::NODES_GAUGE).to receive(:set)
-
-      prometheus.update_all_prometheus_metrics_from_nodes
-
-      expect(prometheus).to have_received(:update_prometheus_metrics).once
-    end
-
-    it "skips metric updates when prom_report_ids is empty" do
-      # Override prom_report_ids to return empty list for this test.
-      klass = Class.new do
-        include PotatoMesh::App::Prometheus
-        include PotatoMesh::App::Queries
-        include PotatoMesh::App::Helpers
-        include PotatoMesh::App::DataProcessing
-
-        def prom_report_ids
-          []
-        end
-
-        def private_mode?; false; end
-        def debug_log(m, **); end
-        def warn_log(m, **); end
-        def open_database(**); SQLite3::Database.new(PotatoMesh::Config.db_path); end
-        def normalize_node_id(*); nil; end
-        def with_busy_retry; yield; end
-        def update_prometheus_metrics(*); end
-        def resolve_protocol(*); "meshtastic"; end
-      end
-
-      obj = klass.new
-      allow(obj).to receive(:query_nodes).and_return([{ "node_id" => "!aabb1234" }])
-      allow(obj).to receive(:update_prometheus_metrics)
-      allow(PotatoMesh::App::Prometheus::NODES_GAUGE).to receive(:set)
-
-      obj.update_all_prometheus_metrics_from_nodes
-
-      expect(obj).not_to have_received(:update_prometheus_metrics)
-    end
-  end
-
-  # ---------------------------------------------------------------------------
-  # prometheus_visible_node_ids (SPEC PM1/PM2)
-  # ---------------------------------------------------------------------------
-  describe "#prometheus_visible_node_ids" do
+  describe "#prometheus_node_count" do
     let(:marker) { PotatoMesh::Config.node_opt_out_marker }
     let(:now) { Time.now.to_i }
 
-    it "returns every node without the marker, also one older than every API window" do
-      insert_node_row("!0a2f0101", last_heard: now)
-      insert_node_row("!0a2f0102", last_heard: now - 60 * 86_400)
+    it "counts what GET /api/nodes lists: heard within the week, not opted out, not hidden under PRIVATE=1" do
+      insert_node_row("!0a2f0501", last_heard: now - 60)
+      insert_node_row("!0a2f0502", last_heard: now - 8 * 86_400)
+      insert_node_row("!0a2f0503", long_name: "Quiet #{marker} Station")
+      insert_node_row("!0a2f0504", short_name: "Q#{marker}")
+      insert_node_row("!0a2f0505", role: "CLIENT_HIDDEN")
 
-      expect(prometheus.prometheus_visible_node_ids).to eq(Set["!0a2f0101", "!0a2f0102"])
+      expect(prometheus.prometheus_node_count).to eq(2)
+      expect(prometheus.query_nodes(1000).length).to eq(2)
+
+      allow(prometheus).to receive(:private_mode?).and_return(true)
+      expect(prometheus.prometheus_node_count).to eq(1)
+      expect(prometheus.query_nodes(1000).length).to eq(1)
+    end
+
+    it "takes the week's floor inclusively" do
+      insert_node_row("!0a2f0506", last_heard: now - 604_800)
+      insert_node_row("!0a2f0507", last_heard: now - 604_801)
+
+      expect(prometheus.prometheus_node_count(now: now)).to eq(1)
+    end
+
+    it "counts on a handle it is given and leaves it open, else opens and closes its own" do
+      insert_node_row("!0a2f0508")
+      handle = prometheus.open_database
+      opened = []
+      allow(prometheus).to receive(:open_database).and_wrap_original do |original, **options|
+        original.call(**options).tap { |db| opened << db }
+      end
+
+      expect(prometheus.prometheus_node_count(handle)).to eq(1)
+      expect(handle).not_to be_closed
+      expect(opened).to be_empty
+
+      expect(prometheus.prometheus_node_count).to eq(1)
+      expect(opened.length).to eq(1)
+      expect(opened.first).to be_closed
+    ensure
+      handle&.close
+    end
+
+    it "raises when it cannot open the database" do
+      missing = File.join(File.dirname(PotatoMesh::Config.db_path), "missing", "mesh.db")
+      allow(PotatoMesh::Config).to receive(:db_path).and_return(missing)
+
+      expect { prometheus.prometheus_node_count }.to raise_error(SQLite3::CantOpenException)
+    end
+  end
+
+  describe "#update_all_prometheus_metrics_from_nodes" do
+    it "seeds the node-count gauge from the database and sets no per-node gauge" do
+      insert_node_row("!0a2f0601")
+      insert_node_row("!0a2f0602")
+      per_node = PotatoMesh::App::Prometheus::METRICS.select { |metric| metric.labels.include?(:node) }
+      per_node.each { |metric| allow(metric).to receive(:set) }
+      allow(PotatoMesh::App::Prometheus::NODES_GAUGE).to receive(:set)
+
+      prometheus.update_all_prometheus_metrics_from_nodes
+
+      expect(PotatoMesh::App::Prometheus::NODES_GAUGE).to have_received(:set).with(2)
+      per_node.each { |metric| expect(metric).not_to have_received(:set) }
+    end
+  end
+
+  # ---------------------------------------------------------------------------
+  # prometheus_node_rows: PM1's visible set, narrowed by PROM_REPORT_IDS
+  # (SPEC PM1, PM2, PG2)
+  # ---------------------------------------------------------------------------
+  describe "#prometheus_node_rows" do
+    let(:marker) { PotatoMesh::Config.node_opt_out_marker }
+    let(:now) { Time.now.to_i }
+
+    # Node ids of the rows a scrape reads.
+    #
+    # @param ids [Array<String>] report ids.
+    # @return [Array<String>] node ids in row order.
+    def row_ids(ids = ["*"])
+      prometheus.prometheus_node_rows(ids).map { |row| row["node_id"] }
+    end
+
+    it "returns every node without the marker by node id, also one older than every API window" do
+      insert_node_row("!0a2f0102", last_heard: now - 60 * 86_400)
+      insert_node_row("!0a2f0101", last_heard: now)
+
+      expect(row_ids).to eq(%w[!0a2f0101 !0a2f0102])
     end
 
     it "leaves out a node whose long or short name carries the opt-out marker" do
@@ -345,7 +366,7 @@ RSpec.describe PotatoMesh::App::Prometheus do
       insert_node_row("!0a2f0104", long_name: "Quiet #{marker} Station")
       insert_node_row("!0a2f0105", short_name: "Q#{marker}")
 
-      expect(prometheus.prometheus_visible_node_ids).to eq(Set["!0a2f0103"])
+      expect(row_ids).to eq(["!0a2f0103"])
     end
 
     it "leaves out a node whose row the retention purge deleted" do
@@ -354,37 +375,133 @@ RSpec.describe PotatoMesh::App::Prometheus do
       allow(PotatoMesh::Application).to receive(:info_log)
       PotatoMesh::Application.purge_old_data!(now: now)
 
-      expect(prometheus.prometheus_visible_node_ids).to eq(Set["!0a2f0106"])
+      expect(row_ids).to eq(["!0a2f0106"])
     end
 
     it "returns a node again once it removes the marker" do
       insert_node_row("!0a2f0108", long_name: "Quiet #{marker} Station")
-      expect(prometheus.prometheus_visible_node_ids).to be_empty
+      expect(row_ids).to be_empty
 
       execute_sql("UPDATE nodes SET long_name = ? WHERE node_id = ?", ["Quiet Station", "!0a2f0108"])
-      expect(prometheus.prometheus_visible_node_ids).to eq(Set["!0a2f0108"])
+      expect(row_ids).to eq(["!0a2f0108"])
     end
 
     it "leaves out CLIENT_HIDDEN nodes under PRIVATE=1 only, keeping a NULL role" do
       insert_node_row("!0a2f0109", role: "CLIENT_HIDDEN")
       insert_node_row("!0a2f010a", role: nil)
       insert_node_row("!0a2f010b", role: "ROUTER")
-      expect(prometheus.prometheus_visible_node_ids).to eq(Set["!0a2f0109", "!0a2f010a", "!0a2f010b"])
+      expect(row_ids).to eq(%w[!0a2f0109 !0a2f010a !0a2f010b])
 
       allow(prometheus).to receive(:private_mode?).and_return(true)
-      expect(prometheus.prometheus_visible_node_ids).to eq(Set["!0a2f010a", "!0a2f010b"])
+      expect(row_ids).to eq(%w[!0a2f010a !0a2f010b])
+    end
+
+    it "keeps the visible nodes a list names, every node for a leading *, and runs no query without ids" do
+      insert_node_row("!0a2f0110")
+      insert_node_row("!0a2f0111")
+      insert_node_row("!0a2f0112", long_name: "Quiet #{marker} Station")
+
+      expect(row_ids(%w[!0a2f0111 !0a2f0112 !0a2f01ff])).to eq(["!0a2f0111"])
+      # Only a leading * is the wildcard; anywhere else it names no node.
+      expect(row_ids(%w[!0a2f0110 *])).to eq(["!0a2f0110"])
+      expect(row_ids(["*"])).to eq(%w[!0a2f0110 !0a2f0111])
+
+      expect(prometheus).not_to receive(:open_database)
+      expect(prometheus.prometheus_node_rows([])).to eq([])
+    end
+
+    it "reads the configured report ids and every column the series need" do
+      insert_node_row("!0a2f0113")
+
+      rows = prometheus.prometheus_node_rows
+
+      expect(rows.map { |row| row["node_id"] }).to eq(["!0a2f0113"])
+      expect(rows.first.keys).to match_array(PotatoMesh::App::Prometheus::NODE_SERIES_COLUMNS)
     end
   end
 
   # ---------------------------------------------------------------------------
-  # ExportRegistry: the view /metrics prints (SPEC PM1/PM2)
+  # prometheus_node_series: the per-node series of one scrape (SPEC PG2, PG3)
+  # ---------------------------------------------------------------------------
+  describe "#prometheus_node_series" do
+    let(:now) { Time.now.to_i }
+
+    # Store measurements on a node row.
+    #
+    # @param node_id [String] canonical node id.
+    # @param columns [Hash{Symbol => Object}] column values.
+    # @return [void]
+    def store_columns(node_id, **columns)
+      assignments = columns.keys.map { |column| "#{column} = ?" }.join(", ")
+      execute_sql("UPDATE nodes SET #{assignments} WHERE node_id = ?", columns.values + [node_id])
+    end
+
+    it "builds every per-node family from a full row" do
+      insert_node_row("!0a2f0701", long_name: "Full Station", short_name: "FS")
+      store_columns("!0a2f0701", hw_model: "TBEAM", battery_level: 80, voltage: 4.1, uptime_seconds: 3600,
+                                 channel_utilization: 12.5, air_util_tx: 1.25, latitude: 52.5, longitude: 13.4, altitude: 40)
+      node = { node: "!0a2f0701" }
+
+      expect(prometheus.prometheus_node_series).to eq(
+        meshtastic_node: { node.merge(short_name: "FS", long_name: "Full Station", hw_model: "TBEAM", role: "CLIENT") => 1.0 },
+        meshtastic_node_battery_level: { node => 80.0 },
+        meshtastic_node_voltage: { node => 4.1 },
+        meshtastic_node_uptime_seconds: { node => 3600.0 },
+        meshtastic_node_channel_utilization: { node => 12.5 },
+        meshtastic_node_transmit_air_utilization: { node => 1.25 },
+        meshtastic_node_latitude: { node => 52.5 },
+        meshtastic_node_longitude: { node => 13.4 },
+        meshtastic_node_altitude: { node => 40.0 },
+      )
+    end
+
+    it "labels presence with empty names and model and the CLIENT role when they are NULL, and skips columns that hold no number" do
+      execute_sql("INSERT INTO nodes(node_id, last_heard, voltage, altitude) VALUES (?, ?, ?, ?)", ["!0a2f0702", now, "four volts", 10])
+
+      expect(prometheus.prometheus_node_series).to eq(
+        meshtastic_node: { { node: "!0a2f0702", short_name: "", long_name: "", hw_model: "", role: "CLIENT" } => 1.0 },
+      )
+      # GET /api/nodes shows the same role for the row.
+      expect(prometheus.query_nodes(10).map { |row| row["role"] }).to eq(["CLIENT"])
+    end
+
+    it "exports no coordinate gauge for a position off the globe or at (0, 0), and the altitude only beside a kept axis" do
+      { "!0a2f0703" => [95.0, 13.4], "!0a2f0704" => [0.0, 0.0], "!0a2f0705" => [52.5, nil], "!0a2f0706" => [nil, nil] }.each do |node_id, (lat, lon)|
+        insert_node_row(node_id)
+        store_columns(node_id, latitude: lat, longitude: lon, altitude: 10)
+      end
+
+      series = prometheus.prometheus_node_series
+
+      expect(series[:meshtastic_node_latitude]).to eq({ { node: "!0a2f0705" } => 52.5 })
+      expect(series).not_to have_key(:meshtastic_node_longitude)
+      expect(series[:meshtastic_node_altitude]).to eq({ { node: "!0a2f0705" } => 10.0 })
+      expect(series[:meshtastic_node].length).to eq(4)
+    end
+
+    it "builds nothing without report ids" do
+      insert_node_row("!0a2f0707")
+      allow(prometheus).to receive(:prom_report_ids).and_return([])
+
+      expect(prometheus.prometheus_node_series).to eq({})
+    end
+  end
+
+  # ---------------------------------------------------------------------------
+  # ExportRegistry: the view /metrics prints (SPEC PM1, PM2, PG2)
   # ---------------------------------------------------------------------------
   describe PotatoMesh::App::Prometheus::ExportRegistry do
     let(:marker) { PotatoMesh::Config.node_opt_out_marker }
-    # A private registry keeps these series out of the global one.
-    let(:registry) { ::Prometheus::Client::Registry.new }
-    let!(:battery) { registry.gauge(:spec_node_battery_level, docstring: "Battery", labels: [:node]) }
-    let!(:presence) { registry.gauge(:spec_node, docstring: "Presence", labels: %i[node long_name]) }
+    # A private registry with two of the real per-node families, a
+    # node-labelled family the build does not fill, and two families without
+    # a node label.
+    let(:registry) do
+      ::Prometheus::Client::Registry.new.tap do |private_registry|
+        private_registry.register(PotatoMesh::App::Prometheus::NODE_BATTERY_LEVEL)
+        private_registry.register(PotatoMesh::App::Prometheus::NODE_GAUGE)
+      end
+    end
+    let!(:unbuilt) { registry.gauge(:spec_node_unbuilt, docstring: "Unbuilt", labels: [:node]) }
     let!(:requests) { registry.counter(:spec_requests_total, docstring: "Requests", labels: [:path]) }
     let!(:messages) { registry.counter(:spec_messages_total, docstring: "Messages") }
 
@@ -397,69 +514,91 @@ RSpec.describe PotatoMesh::App::Prometheus do
       ::Prometheus::Client::Formats::Text.marshal(export)
     end
 
-    it "keeps visible nodes' series with their values and drops opted-out and deleted nodes" do
+    # Store a battery level on a node row.
+    #
+    # @param node_id [String] canonical node id.
+    # @param level [Numeric] battery level.
+    # @return [void]
+    def store_battery(node_id, level)
+      execute_sql("UPDATE nodes SET battery_level = ? WHERE node_id = ?", [level, node_id])
+    end
+
+    # The presence line +insert_node_row+'s defaults print.
+    #
+    # @param node_id [String] canonical node id.
+    # @param long_name [String] stored long name.
+    # @return [String] +meshtastic_node+ sample.
+    def presence(node_id, long_name = "Spec Node")
+      %(meshtastic_node{node="#{node_id}",short_name="SN",long_name="#{long_name}",hw_model="",role="CLIENT"} 1.0)
+    end
+
+    it "prints the stored series of visible nodes, none of opted-out or deleted nodes" do
       insert_node_row("!0a2f0201")
       insert_node_row("!0a2f0202", long_name: "Quiet #{marker} Station")
-      # !0a2f0203 has series but no row, as after a retention purge.
-      %w[!0a2f0201 !0a2f0202 !0a2f0203].each_with_index do |node_id, index|
-        battery.set(50 + index, labels: { node: node_id })
-        presence.set(1, labels: { node: node_id, long_name: "Station #{index}" })
-      end
+      store_battery("!0a2f0201", 50)
+      store_battery("!0a2f0202", 51)
+      # !0a2f0203 has a registry series but no row, as after a retention purge.
+      unbuilt.set(52, labels: { node: "!0a2f0203" })
 
       text = scrape
 
-      expect(node_samples(text, "!0a2f0201")).to eq(
-        [
-          'spec_node_battery_level{node="!0a2f0201"} 50.0',
-          'spec_node{node="!0a2f0201",long_name="Station 0"} 1.0',
-        ],
-      )
+      expect(node_samples(text, "!0a2f0201")).to eq(['meshtastic_node_battery_level{node="!0a2f0201"} 50.0', presence("!0a2f0201")])
       expect(node_samples(text, "!0a2f0202")).to eq([])
       expect(node_samples(text, "!0a2f0203")).to eq([])
-      # The registry itself keeps every series; only the view filters.
-      expect(battery.values.keys.map { |label_set| label_set[:node] }).to eq(%w[!0a2f0201 !0a2f0202 !0a2f0203])
+    end
+
+    it "prints no series of a node-labelled family the build does not fill, whatever the registry holds" do
+      insert_node_row("!0a2f0209")
+      unbuilt.set(9, labels: { node: "!0a2f0209" })
+
+      text = scrape
+
+      expect(text).to include("# TYPE spec_node_unbuilt gauge")
+      expect(node_samples(text, "!0a2f0209")).to eq([presence("!0a2f0209")])
     end
 
     it "passes families without a node label through unchanged" do
-      requests.increment(labels: { path: "/api/nodes/!0a2f0202" })
+      requests.increment(labels: { path: "GET /api/nodes/:id" })
       messages.increment
 
       families = export.metrics
 
-      expect(families.map(&:name)).to eq(%i[spec_node_battery_level spec_node spec_requests_total spec_messages_total])
-      expect(families[2]).to be(requests)
-      expect(families[3]).to be(messages)
-      expect(scrape).to include(%(spec_requests_total{path="/api/nodes/!0a2f0202"} 1.0), "spec_messages_total 1.0")
+      expect(families.map(&:name)).to eq(%i[meshtastic_node_battery_level meshtastic_node spec_node_unbuilt spec_requests_total spec_messages_total])
+      expect(families[3]).to be(requests)
+      expect(families[4]).to be(messages)
+      expect(scrape).to include(%(spec_requests_total{path="GET /api/nodes/:id"} 1.0), "spec_messages_total 1.0")
     end
 
     it "wraps a per-node family without changing its name, type, help or labels" do
       family = export.metrics.first
 
       expect(family).to be_a(described_class::NodeFamily)
-      expect([family.name, family.type, family.docstring, family.labels]).to eq([:spec_node_battery_level, :gauge, "Battery", [:node]])
+      expect([family.name, family.type, family.docstring, family.labels]).to eq(
+        [:meshtastic_node_battery_level, :gauge, "Battery level of a Meshtastic node", [:node]],
+      )
     end
 
-    it "runs one visibility lookup per scrape" do
-      allow(prometheus).to receive(:prometheus_visible_node_ids).and_call_original
+    it "runs one node lookup per scrape" do
+      allow(prometheus).to receive(:prometheus_node_series).and_call_original
 
       scrape
 
-      expect(prometheus).to have_received(:prometheus_visible_node_ids).once
+      expect(prometheus).to have_received(:prometheus_node_series).once
     end
 
     it "withholds every per-node family and logs a warning when the lookup raises" do
       insert_node_row("!0a2f0204")
-      battery.set(70, labels: { node: "!0a2f0204" })
+      store_battery("!0a2f0204", 70)
       messages.increment
-      # The visibility query cannot open the database.
+      # The node query cannot open the database.
       missing = File.join(File.dirname(PotatoMesh::Config.db_path), "missing", "mesh.db")
       allow(PotatoMesh::Config).to receive(:db_path).and_return(missing)
       allow(prometheus).to receive(:warn_log)
 
       text = scrape
 
-      # No sample and no TYPE/HELP line of either per-node family.
-      expect(text).not_to include("spec_node")
+      # No sample and no TYPE/HELP line of any per-node family.
+      expect(text).not_to include("meshtastic_node", "spec_node")
       expect(text).to include("# TYPE spec_requests_total counter", "spec_messages_total 1.0")
       expect(prometheus).to have_received(:warn_log).with(
         "Withheld per-node metrics: visible node lookup failed",
@@ -470,40 +609,34 @@ RSpec.describe PotatoMesh::App::Prometheus do
     end
 
     it "exports a node in an explicit PROM_REPORT_IDS list only while /api/nodes lists it" do
-      reporter = Class.new do
-        include PotatoMesh::App::Prometheus
-
-        def prom_report_ids
-          ["!0a2f0205", "!0a2f0206"]
-        end
-      end.new
+      allow(prometheus).to receive(:prom_report_ids).and_return(["!0a2f0205", "!0a2f0206"])
       insert_node_row("!0a2f0205")
       insert_node_row("!0a2f0206", long_name: "Quiet #{marker} Station")
       insert_node_row("!0a2f0207")
-      %w[!0a2f0205 !0a2f0206 !0a2f0207].each do |node_id|
-        reporter.update_prometheus_metrics(node_id, nil, "", { "batteryLevel" => 40 })
-      end
+      %w[!0a2f0205 !0a2f0206 !0a2f0207].each { |node_id| store_battery(node_id, 40) }
 
-      text = ::Prometheus::Client::Formats::Text.marshal(described_class.new(::Prometheus::Client.registry, prometheus))
+      text = scrape
 
-      expect(node_samples(text, "!0a2f0205")).to eq(['meshtastic_node_battery_level{node="!0a2f0205"} 40.0'])
+      expect(node_samples(text, "!0a2f0205")).to eq(['meshtastic_node_battery_level{node="!0a2f0205"} 40.0', presence("!0a2f0205")])
       expect(node_samples(text, "!0a2f0206")).to eq([])
       expect(node_samples(text, "!0a2f0207")).to eq([])
     end
 
-    it "gives a node its series back, with their last values, once it removes the marker" do
+    it "gives a node its series back, with its stored values, once it removes the marker" do
       insert_node_row("!0a2f0208", long_name: "Quiet #{marker} Station")
-      battery.set(33, labels: { node: "!0a2f0208" })
+      store_battery("!0a2f0208", 33)
       expect(node_samples(scrape, "!0a2f0208")).to eq([])
 
       execute_sql("UPDATE nodes SET long_name = ? WHERE node_id = ?", ["Quiet Station", "!0a2f0208"])
 
-      expect(node_samples(scrape, "!0a2f0208")).to eq(['spec_node_battery_level{node="!0a2f0208"} 33.0'])
+      expect(node_samples(scrape, "!0a2f0208")).to eq(
+        ['meshtastic_node_battery_level{node="!0a2f0208"} 33.0', presence("!0a2f0208", "Quiet Station")],
+      )
     end
   end
 
   # ---------------------------------------------------------------------------
-  # /metrics wiring: the Exporter and the route print one view (SPEC PM1)
+  # /metrics wiring: the collector, the Exporter and the route (SPEC PM1, PG1)
   # ---------------------------------------------------------------------------
   describe "/metrics wiring" do
     let(:export) { PotatoMesh::Application.settings.prometheus_export_registry }
@@ -515,20 +648,33 @@ RSpec.describe PotatoMesh::App::Prometheus do
       expect(exporter[1].first[:registry]).to be(export)
     end
 
+    it "records requests through the route-labelled collector, outside the Exporter" do
+      classes = PotatoMesh::Application.middleware.map(&:first)
+
+      expect(classes).not_to include(::Prometheus::Middleware::Collector)
+      expect(classes.index(PotatoMesh::App::Prometheus::RouteCollector)).to be < classes.index(::Prometheus::Middleware::Exporter)
+    end
+
+    it "mounts the Exporter at the path the collector labels metrics" do
+      exporter = PotatoMesh::Application.middleware.find { |klass, _args, _block| klass == ::Prometheus::Middleware::Exporter }
+
+      expect(exporter[1].first).not_to have_key(:path)
+      expect(::Prometheus::Middleware::Exporter.new(->(_env) { [200, {}, []] }).path).to eq(PotatoMesh::App::Prometheus::METRICS_PATH)
+    end
+
     it "prints the same view from the /metrics route" do
       marker = PotatoMesh::Config.node_opt_out_marker
+      allow(PotatoMesh::Config).to receive(:prom_report_id_list).and_return(["*"])
       insert_node_row("!0a2f0301")
       insert_node_row("!0a2f0302", long_name: "Quiet #{marker} Station")
-      %w[!0a2f0301 !0a2f0302].each do |node_id|
-        PotatoMesh::App::Prometheus::NODE_BATTERY_LEVEL.set(61, labels: { node: node_id })
-      end
+      execute_sql("UPDATE nodes SET battery_level = ?", [61])
 
       # The bare instance has no middleware, so the Sinatra route answers.
       response = Rack::MockRequest.new(PotatoMesh::Application.new!).get("/metrics")
 
       expect(response.status).to eq(200)
       expect(response.content_type).to start_with(::Prometheus::Client::Formats::Text::CONTENT_TYPE)
-      expect(node_samples(response.body, "!0a2f0301")).to eq(['meshtastic_node_battery_level{node="!0a2f0301"} 61.0'])
+      expect(node_samples(response.body, "!0a2f0301")).to include('meshtastic_node_battery_level{node="!0a2f0301"} 61.0')
       expect(node_samples(response.body, "!0a2f0302")).to eq([])
     end
   end
@@ -581,16 +727,6 @@ RSpec.describe PotatoMesh::App::Prometheus do
       }
     end
 
-    # Scrape +/metrics+ and keep the samples labelled with one node id.
-    #
-    # @param node_id [String] canonical node id.
-    # @return [Array<String>] exposition lines for that node, comments excluded.
-    def scraped_series(node_id)
-      get "/metrics"
-      expect(last_response.status).to eq(200)
-      node_samples(last_response.body, node_id)
-    end
-
     # Node ids +GET /api/nodes+ serves.
     #
     # @return [Array<String>] listed node ids.
@@ -605,15 +741,15 @@ RSpec.describe PotatoMesh::App::Prometheus do
       post "/api/nodes", { visible => node_record("Loud Station", now - 60), opted => node_record("Quiet Station", now - 60) }.to_json, auth_headers
       expect(last_response.status).to eq(201)
       # Before the opt-out the node is exported like any other.
-      expect(scraped_series(opted).length).to eq(per_node_families)
+      expect(scraped_node_samples(opted).length).to eq(per_node_families)
 
       post "/api/nodes", { opted => node_record("Quiet #{marker} Station", now - 30) }.to_json, auth_headers
       post "/api/telemetry", [{ id: 920_001, node_id: opted, rx_time: now - 20, battery_level: 55, voltage: 3.7 }].to_json, auth_headers
       post "/api/positions", [{ id: 920_002, node_id: opted, rx_time: now - 10, latitude: 48.1, longitude: 11.5, altitude: 520 }].to_json, auth_headers
       expect(listed_node_ids).to eq([visible])
 
-      expect(scraped_series(opted)).to eq([])
-      expect(scraped_series(visible).length).to eq(per_node_families)
+      expect(scraped_node_samples(opted)).to eq([])
+      expect(scraped_node_samples(visible).length).to eq(per_node_families)
     end
 
     it "serves no series of an opted-out node named in an explicit PROM_REPORT_IDS list" do
@@ -623,8 +759,8 @@ RSpec.describe PotatoMesh::App::Prometheus do
       post "/api/nodes", { visible => node_record("Loud Station", now - 30), opted => node_record("Quiet #{marker} Station", now - 30) }.to_json, auth_headers
       expect(listed_node_ids).to eq([visible])
 
-      expect(scraped_series(opted)).to eq([])
-      expect(scraped_series(visible).length).to eq(per_node_families)
+      expect(scraped_node_samples(opted)).to eq([])
+      expect(scraped_node_samples(visible).length).to eq(per_node_families)
     end
 
     it "serves no series of an opted-out Reticulum node announced under an unmarked aspect" do
@@ -646,7 +782,7 @@ RSpec.describe PotatoMesh::App::Prometheus do
       end
       post "/api/nodes", announce.call("Argos Station", "lxmf.delivery", "PEER", "22", now - 40).to_json, auth_headers
       # Before the opt-out the Reticulum node is exported like any other.
-      expect(scraped_series(node_id)).not_to be_empty
+      expect(scraped_node_samples(node_id)).not_to be_empty
 
       post "/api/nodes", announce.call("Quiet #{marker} Node", "nomadnetwork.node", "NODE", "11", now - 30).to_json, auth_headers
       # The PEER aspect announces again, without the marker. RE10 keeps the
@@ -654,7 +790,7 @@ RSpec.describe PotatoMesh::App::Prometheus do
       post "/api/nodes", announce.call("Argos Station", "lxmf.delivery", "PEER", "22", now - 10).to_json, auth_headers
       expect(listed_node_ids).to eq([])
 
-      expect(scraped_series(node_id)).to eq([])
+      expect(scraped_node_samples(node_id)).to eq([])
     end
 
     it "serves no series of a CLIENT_HIDDEN node under PRIVATE=1 while a visible node keeps every family" do
@@ -665,8 +801,8 @@ RSpec.describe PotatoMesh::App::Prometheus do
       expect(last_response.status).to eq(201)
       expect(listed_node_ids).to eq([visible])
 
-      expect(scraped_series(hidden)).to eq([])
-      expect(scraped_series(visible).length).to eq(per_node_families)
+      expect(scraped_node_samples(hidden)).to eq([])
+      expect(scraped_node_samples(visible).length).to eq(per_node_families)
     end
   end
 end

web/spec/queries_spec.rb

@@ -32,11 +32,6 @@ RSpec.describe PotatoMesh::App::Queries do
         false
       end
 
-      # Stub prom_report_ids so tests do not depend on prometheus env.
-      def prom_report_ids
-        []
-      end
-
       # No-op for debug_log calls inside query helpers.
       def debug_log(message, **); end
 
@@ -61,8 +56,6 @@ RSpec.describe PotatoMesh::App::Queries do
         yield
       end
 
-      def update_prometheus_metrics(*); end
-
       def resolve_protocol(_db, _ingestor, cache: nil)
         "meshtastic"
       end

web/spec/support/data_processing_harness.rb

@@ -21,7 +21,7 @@ require "tmpdir"
 # in isolation (+data_processing_spec.rb+, +meshcore_dedup_spec.rb+).
 module DataProcessingHarness
   # Build a minimal host class so the module methods can be called in
-  # isolation: logging, retry, metrics and privacy collaborators are stubbed,
+  # isolation: logging, retry and privacy collaborators are stubbed,
   # and node references normalize to their canonical id without a lookup.
   #
   # @param protocol [String] value the stubbed +resolve_protocol+ returns.
@@ -43,12 +43,6 @@ module DataProcessingHarness
         yield
       end
 
-      def update_prometheus_metrics(*); end
-
-      def prom_report_ids
-        []
-      end
-
       def private_mode?
         false
       end

web/spec/support/metrics_spec_helpers.rb

@@ -0,0 +1,77 @@
+# Copyright © 2025-26 l5yth & contributors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# frozen_string_literal: true
+
+# Exposition readers shared by the specs that read +/metrics+
+# (+prometheus_spec.rb+, +metrics_routes_spec.rb+, +field_limits_spec.rb+).
+# The scrape helpers request +/metrics+ through Rack::Test.
+module MetricsSpecHelpers
+  # Sample lines of a text exposition, comments excluded.
+  #
+  # @param text [String] Prometheus text exposition.
+  # @return [Array<String>] sample lines.
+  def sample_lines(text)
+    text.each_line.map(&:chomp).reject { |line| line.start_with?("#") }
+  end
+
+  # Sample lines of one metric family in a text exposition.
+  #
+  # @param text [String] Prometheus text exposition.
+  # @param family [String] metric name; a longer name sharing its prefix is
+  #   another family.
+  # @param node_id [String, nil] keep only this node's samples when given.
+  # @return [Array<String>] sample lines, comments excluded.
+  def family_samples(text, family, node_id = nil)
+    sample_lines(text).select do |line|
+      (line.start_with?("#{family}{") || line.start_with?("#{family} ")) &&
+        (node_id.nil? || line.include?(%(node="#{node_id}")))
+    end
+  end
+
+  # Sample lines labelled with one node id, of every family.
+  #
+  # @param text [String] Prometheus text exposition.
+  # @param node_id [String] canonical node id.
+  # @return [Array<String>] sample lines for that node, comments excluded.
+  def node_samples(text, node_id)
+    sample_lines(text).select { |line| line.include?(%(node="#{node_id}")) }
+  end
+
+  # Scrape +/metrics+ through the app and expect it to answer.
+  #
+  # @return [String] text exposition.
+  def scraped_metrics
+    get "/metrics"
+    expect(last_response.status).to eq(200)
+    last_response.body
+  end
+
+  # Scrape +/metrics+ and keep the samples of one family.
+  #
+  # @param family [String] metric name.
+  # @param node_id [String, nil] keep only this node's samples when given.
+  # @return [Array<String>] sample lines, comments excluded.
+  def scraped_samples(family, node_id = nil)
+    family_samples(scraped_metrics, family, node_id)
+  end
+
+  # Scrape +/metrics+ and keep the samples labelled with one node id.
+  #
+  # @param node_id [String] canonical node id.
+  # @return [Array<String>] sample lines for that node, comments excluded.
+  def scraped_node_samples(node_id)
+    node_samples(scraped_metrics, node_id)
+  end
+end