data,web: /api/stats packets as <scope>.packets.hour; total sums protocols

This commit is contained in:
l5y
2026-07-26 12:33:48 +02:00
parent 433d48e84a
commit 0bc6ba7185
7 changed files with 29 additions and 23 deletions
+6 -5
View File
@@ -4041,11 +4041,12 @@ older than the configured window (≥ 24 h), so the table cannot grow unbounded.
```
**Expected:** pass. With two `meshcore` ingestors reporting different 24 h packet
totals, the meshcore rate = `MAX(total_A, total_B) ÷ 24` — the busiest single
vantage, so the quieter ingestor and any overlap never inflate it (`total` is the
`MAX` across **all** ingestors regardless of protocol; a protocol with no active
ingestor reads `0`; rows older than 24 h do not contribute). `query_packets_per_hour`
returns these per-protocol rates, which the `GET /api/stats` route folds into each
scope as `<scope>.packets.hour` (MA-A5).
vantage, so the quieter ingestor and any overlap never inflate it. `total` is the
**SUM** of the per-protocol rates (distinct protocols never share the air, so they
add — e.g. meshcore + meshtastic), a protocol with no active ingestor reads `0`,
and rows older than 24 h do not contribute. `query_packets_per_hour` returns these
per-protocol rates, which the `GET /api/stats` route folds into each scope as
`<scope>.packets.hour` (MA-A5).
### MA-A5 — `/api/stats` exposes packets as an additive `<scope>.packets.hour` metric — MA5
```bash
+1 -1
View File
@@ -22,7 +22,7 @@ table, and reported-only disclosure rows.
* Web: nodes table gains grouped headers, curated mobile columns (Battery survives), a per-row disclosure of hidden fields, row hover/click, numeric alignment, captions/scopes
### Fixes
* Web/data: `GET /api/stats` exposes the mesh-activity packets/hour rate as `<scope>.packets.hour` (folded into the S1 scope→metric→window tree) rather than a top-level `packets_per_hour` map; the ingestor announcement dogfeed reads the new path (SPEC MA5, pre-release amendment of the merged-but-unreleased #859)
* Web/data: `GET /api/stats` exposes the mesh-activity packets/hour rate as `<scope>.packets.hour` (folded into the S1 scope→metric→window tree) rather than a top-level `packets_per_hour` map, and `total.packets.hour` now **sums** the per-protocol rates instead of taking a single MAX vantage (distinct protocols never share the air, so they add); the ingestor announcement dogfeed reads the new path (SPEC MA4/MA5, pre-release amendment of the merged-but-unreleased #859)
* Web: WCAG AA text contrast — role-badge text computed by background luminance; chat log entries, error text, links, and focus rings tokenised (`--danger`, single `--accent`)
* Web: federation table's undefined CSS tokens aliased — row borders and the sticky header render again
* Web: legend expanded by default on /map, honest toggle label (filter suffix only when filters are active), line-style key on the neighbor/trace toggles; fixed the malformed `data-legend-collapsed` attribute
+1 -1
View File
@@ -1096,7 +1096,7 @@ contradicted.
| **MA1** | **Merged packet counter — count everything, at the earliest seam.** Each ingestor maintains one merged `packets` counter incremented on **every received frame***all of them, including ignored / errored / unimplemented / unsupported-port packets* — counted at the earliest common receive seam (`handlers/_state._mark_packet_seen`, already invoked by both the Meshtastic `on_receive` path and every MeshCore handler/telemetry path *before* any dispatch or filtering), **plus every ingestor-initiated transmission** (the announcement itself and the existing MeshCore telemetry/status polls). RX and TX are deliberately **merged** into a single figure (no separate breakdown). "We don't want to under-report" is satisfied at source: counting precedes every drop/ignore decision. A build-phase check confirms each protocol's RX entry funnels through the seam so no family is missed. | interview |
| **MA2** | **Heartbeat carries the per-interval delta.** `POST /api/ingestors` gains an additive optional `packets` field = frames counted since the previous heartbeat (a per-interval **delta**, not a since-boot cumulative), reset to zero each time a heartbeat is queued. Per-interval deltas map one-to-one onto time-series rows and are restart-safe (a reboot simply starts a fresh interval). Absent ⇒ treated as `0`, so pre-feature ingestors and the existing `tests/test_mesh.py` fixtures are unaffected (D8). | interview |
| **MA3** | **Per-ingestor activity time-series (the moving-average schema).** A new append-only table `ingestor_activity` (`ingestor_id TEXT`, `at INTEGER`, `packets INTEGER`, `protocol TEXT`, indexed on `at`) records one row per heartbeat delta. This is the schema that makes a packets/hour moving average computable while distinguishing **multiple protocols and multiple ingestors per protocol** (each ingestor's contribution is kept separate rather than pre-summed). The `ingestors` snapshot table (one row per node) is unchanged. Rows are pruned by the existing retention worker (window ≥ the 24 h the announcement needs; sized with the other activity floors). | proposed |
| **MA4** | **Aggregation = MAX per protocol (dedup-free "in the air").** The mesh-wide packets/hour for a protocol is `MAX` over that protocol's ingestors of *(that ingestor's total `packets` reported in the last 24 h ÷ 24)*. A single radio can only hear ≤ what is actually transmitted, so the busiest single vantage is the best dedup-free estimate of unique air traffic and can never double-count a frame heard by two radios (true per-frame dedup is impossible anyway — ignored/errored frames carry no id). The fixed `÷ 24` denominator (not ÷ elapsed) keeps the rate stable and avoids divide-by-small spikes; an ingestor with < 24 h of data simply reads lower and ramps up (moot in practice — the announcement fires only ≥ 24 h after start, MA7). *Accepted limitation:* MAX under-estimates the true union when different radios are busiest in different hours. | interview |
| **MA4** | **Aggregation = MAX per protocol; `total` = SUM across protocols (dedup-free "in the air").** The mesh-wide packets/hour for a protocol is `MAX` over that protocol's ingestors of *(that ingestor's total `packets` reported in the last 24 h ÷ 24)*. A single radio can only hear ≤ what is actually transmitted, so the busiest single vantage is the best dedup-free estimate of unique air traffic and can never double-count a frame heard by two radios (true per-frame dedup is impossible anyway — ignored/errored frames carry no id). The **`total`** rate is the **SUM** of the per-protocol rates — different protocols ride different frequencies/channels, so their frames never overlap in the air and add rather than dedup. **Amended pre-release:** `total` first shipped as a MAX over *every* ingestor regardless of protocol, which under-counted it to the single busiest protocol (e.g. meshcore 44 + meshtastic 76 rendered as 76, not 120); corrected to the cross-protocol SUM. The fixed `÷ 24` denominator (not ÷ elapsed) keeps the rate stable and avoids divide-by-small spikes; an ingestor with < 24 h of data simply reads lower and ramps up (moot in practice — the announcement fires only ≥ 24 h after start, MA7). *Accepted limitation:* the per-protocol MAX under-estimates the true union when different radios are busiest in different hours. | interview |
| **MA5** | **Exposure as an additive per-scope `packets` metric.** `GET /api/stats` exposes the MA4 24 h MAX-aggregated moving average under each scope as `<scope>.packets.hour` — a `packets` metric carrying a single `hour` window (it is a rate, not a windowed count, so no day/week/month keys), consistent with the S1 `scope → metric → window` layout. `reticulum.packets.hour` is a forward-looking `0` stub (S6); `total.packets.hour` is the MAX across all ingestors regardless of protocol. The rest of the S1 tree is untouched, so this needs **no** version bump. The announcement's active-node figure reuses the existing `<protocol>.nodes.day` count (already deduped by `node_id`); only packets/hour is new. **Amended pre-release:** the field originally shipped (unreleased, on `main`) as a top-level `packets_per_hour: { total, meshcore, meshtastic, reticulum }` map; it is superseded here by the per-scope `packets.hour` form for consistency with the S1 tree. Because it was merged but **not yet in any tagged release** and is **not** on the signed federation wire, the reshape carries **no** version bump, federation-compat fallback, or signature break — only the in-repo `announce.py` dogfeed reader (MA6) and the specs/docs move with it. | proposed (amended) |
| **MA6** | **Announcement content, drawn from the instance's own API (dogfeeding).** Each ingestor broadcasts, for its own configured protocol, a single line formatted to that protocol's character limit: `"<Protocol> activity in the last 24h: <N> active nodes, <M> packets/hour. https://<domain>"`. `<N>` and `<M>` are fetched **from the target instance** (`GET /api/stats``<protocol>.nodes.day` and `<protocol>.packets.hour`), never computed from the ingestor's local view — because one ingestor may not see the whole mesh. `<domain>` is the configured `INSTANCE_DOMAIN`. Reporting (MA1MA4) is independent and continues regardless of announcement state. | interview |
| **MA7** | **Announcement is triple-gated.** An announcement is transmitted only when **all** hold: (a) `RX_ONLY` is unset — the **reused** receive-only flag (default `0`) is the single transmit gate; `RX_ONLY=1` forbids *every* ingestor TX (the MeshCore polls and the announcement alike), so it is the sole opt-out and **no separate `ENABLE_TX` env is added**; (b) the target instance reports **non-private** — the ingestor GETs `<domain>/version` and honors `config.private_mode`, re-checked every cycle, and **fails closed** (skips) on any fetch/parse error, so a privacy signal is never missed (Invariant II); (c) **≥ 24 h have elapsed since ingestor start** — so the first numbers are accurate over a full window and restarts cannot spam the channel. | interview |
+4 -3
View File
@@ -332,9 +332,10 @@ do **not** accept `before`.
`MAX` over that protocol's ingestors of *(the ingestor's `packets` total in the
last 24 h ÷ 24)* — a single radio hears ≤ what is actually transmitted, so the
busiest vantage is the best dedup-free estimate of air traffic and never
double-counts a frame heard by two radios. `total.packets.hour` is the same MAX
over **every** ingestor regardless of protocol; `reticulum.packets.hour` is the
always-zero forward-looking stub. Unlike `messages`, it is **not** privacy-gated
double-counts a frame heard by two radios. `total.packets.hour` is the **SUM**
of the per-protocol rates (distinct protocols ride distinct frequencies, so they
add rather than dedup); `reticulum.packets.hour` is the always-zero
forward-looking stub. Unlike `messages`, it is **not** privacy-gated
(packets are a public aggregate, no message content). Additive to the 0.7.x
`/api/stats` tree — no version bump; the ingestor dogfeeds it for the activity
announcement (MA6).
@@ -39,10 +39,12 @@ module PotatoMesh
# traffic and can never double-count a frame heard by two radios (true
# per-frame dedup is impossible — ignored/errored frames carry no id).
# For each protocol the rate is
# +MAX(ingestor's 24 h packet total) ÷ 24+, rounded; +total+ is the same
# MAX taken over **every** ingestor regardless of protocol (so an ingestor
# reporting several protocols contributes its combined total). +reticulum+
# is a forward-looking always-zero stub (SPEC S6/MA5).
# +MAX(ingestor's 24 h packet total) ÷ 24+, rounded. +total+ is the **SUM**
# of the per-protocol rates: different protocols ride different
# frequencies/channels, so their frames never overlap in the air and add
# rather than dedup (a same-protocol frame heard by two radios is still
# deduped by the per-protocol MAX). +reticulum+ is a forward-looking
# always-zero stub (SPEC S6/MA5).
#
# @param now [Integer] reference unix timestamp in seconds.
# @param db [SQLite3::Database, nil] optional open database handle to reuse.
@@ -63,16 +65,18 @@ module PotatoMesh
end
per_protocol_max = Hash.new(0)
per_ingestor_total = Hash.new(0)
rows.each do |row|
protocol = row["p"]
total = row["total"].to_i
per_protocol_max[protocol] = [per_protocol_max[protocol], total].max if protocol
per_ingestor_total[row["ingestor_id"]] += total
next unless protocol
per_protocol_max[protocol] = [per_protocol_max[protocol], row["total"].to_i].max
end
{
"total" => packets_per_hour_rate(per_ingestor_total.values.max || 0),
# +total+ sums the per-protocol MAX vantages: distinct protocols ride
# distinct frequencies/channels, so their frames never overlap in the
# air and add rather than dedup (a same-protocol frame heard by two
# radios is still deduped by the per-protocol MAX above).
"total" => packets_per_hour_rate(per_protocol_max.values.sum),
"meshcore" => packets_per_hour_rate(per_protocol_max["meshcore"]),
"meshtastic" => packets_per_hour_rate(per_protocol_max["meshtastic"]),
"reticulum" => 0,
+1 -1
View File
@@ -6742,7 +6742,7 @@ RSpec.describe "Potato Mesh Sinatra app" do
expect(payload).not_to have_key("packets_per_hour")
expect(payload["meshcore"]["packets"]).to eq("hour" => 50)
expect(payload["meshtastic"]["packets"]).to eq("hour" => 30)
expect(payload["total"]["packets"]).to eq("hour" => 50) # MAX over every ingestor = 1200/24
expect(payload["total"]["packets"]).to eq("hour" => 80) # SUM of per-protocol MAX = (1200+720)/24
expect(payload["reticulum"]["packets"]).to eq("hour" => 0)
# The additive field leaves the S1 scope × metric × window tree intact.
expect(payload["sampled"]).to eq(false)
+3 -3
View File
@@ -1304,7 +1304,7 @@ RSpec.describe PotatoMesh::App::Queries do
result = queries.query_packets_per_hour(now: now)
expect(result["meshcore"]).to eq(50) # MAX(1200, 900) / 24
expect(result["meshtastic"]).to eq(30) # 720 / 24
expect(result["total"]).to eq(50) # MAX over every ingestor = 1200 / 24
expect(result["total"]).to eq(80) # SUM of per-protocol MAX = (1200 + 720) / 24
expect(result["reticulum"]).to eq(0)
end
@@ -1332,8 +1332,8 @@ RSpec.describe PotatoMesh::App::Queries do
# The reticulum scope is always emitted as zero (forward-looking stub)…
expect(result["reticulum"]).to eq(0)
# …but a reticulum ingestor still contributes to the protocol-agnostic
# total (MAX across every ingestor).
expect(result["total"]).to eq(30) # 720 / 24
# total (summed across protocols).
expect(result["total"]).to eq(30) # sole protocol: 720 / 24
end
it "rounds the hourly rate to the nearest integer" do