web: add pagination to all APIs

This commit is contained in:
l5y committed 2026-06-24 12:57:24 +02:00
1 parent 7eccbfb89a
commit d99f722ba2
10 files changed
+624 -32

No files matched your search

+99
View File
@@ -1176,3 +1176,102 @@ new helper file must carry the full Apache block); **D1** (`/version` config blo
the shared layout's behavior is unchanged). Any existing view/app spec that
asserted an exact *unversioned* asset string (e.g. `src="/assets/js/app/index.js"`)
is **updated** to the `?v=` form, **not** removed.
---
## Feature: Uniform backward pagination (`?before=`) for bulk collection APIs
Maps to SPEC decisions **BP1–BP9**. `?before=<unix_seconds>` is added as an
inclusive upper-bound keyset cursor to the six bulk collection GETs — `/api/nodes`,
`/api/positions`, `/api/telemetry`, `/api/neighbors`, `/api/traces`,
`/api/ingestors` — mirroring the existing `/api/messages` cursor (**C7**). The
logic lives in `web/lib/potato_mesh/application/routes/api.rb` and the `query_*`
helpers under `web/lib/potato_mesh/application/queries/`; the cursor is documented
in `data/mesh_ingestor/CONTRACTS.md`. Unless a check says otherwise, start the
server in public mode
(`API_TOKEN=acctest PRIVATE=0 FEDERATION=0 bundle exec ruby app.rb`).
### BP-A1 — Every bulk collection pages backward through the full window — BP1, BP2, BP3
```bash
( cd web && bundle exec rspec spec/app_spec.rb -e "before pagination" )
```
**Expected:** pass. For **each** of `/api/nodes`, `/api/positions`,
`/api/telemetry`, `/api/neighbors`, `/api/traces`, and `/api/ingestors`, seeding
more than `MAX_QUERY_LIMIT` (1000) rows inside the route's window and walking
newest → oldest — each page `limit=MAX_QUERY_LIMIT`, then `before=<oldest
primary-sort value seen>`, de-duplicating by id — recovers **every** in-window row
(the walk does not stall at the newest 1000). No single response exceeds
`MAX_QUERY_LIMIT`. The cursor bounds the route's primary sort column inclusively:
`rx_time` for positions/telemetry/neighbors/traces, `last_heard` for nodes,
`last_seen_time` for ingestors.
### BP-A2 — `before` only narrows; the floor still bounds the window — BP2
```bash
( cd web && bundle exec rspec spec/app_spec.rb -e "before cannot widen the window" )
```
**Expected:** pass. A `before` newer than `now` returns the same rows as no
`before` (a no-op upper bound). A `before` older than the route's floor, combined
with the floor-clamped lower bound, returns **nothing beyond the floor** — a row
older than the 7-day / 28-day floor stays excluded, so `before` cannot reach past
it (preserves **C4**). A non-positive or non-integer `before` (`0`, `-5`, `abc`)
is ignored as absent (parity with the messages `coerce_positive_or_nil`), so the
unfiltered newest page is returned.
### BP-A3 — Inclusive boundary, protocol-neutral cursor — BP3, BP5
```bash
( cd web && bundle exec rspec spec/app_spec.rb -e "before pagination boundary" )
```
**Expected:** pass. Two rows sharing the exact boundary second are **both**
returned when that second is passed as `before` (the inclusive `<=` ceiling never
skips a boundary row — client dedup collapses the one-row overlap between pages).
`?before=` composes with `?protocol=`: a backward walk filtered by
`protocol=meshcore` returns only MeshCore rows and still recovers all of them,
with neither protocol privileged.
### BP-A4 — History pages bypass the response cache — BP7
```bash
( cd web && bundle exec rspec spec/app_spec.rb -e "before bypasses the response cache" )
```
**Expected:** pass. A request carrying `before` is served from a fresh query, not
the short-lived `ApiCache` newest-page entry, and issuing it does **not** overwrite
or evict that hot entry — a subsequent no-`before` request still returns the cached
newest page. Matches the established `/api/messages` behavior (a `since > 0` or
`before` request skips the cache; the cache key for the default path is unchanged).
### BP-A5 — Privacy, opt-out, and apex are untouched — BP6
```bash
( cd web && bundle exec rspec spec/app_spec.rb -e "before pagination honors privacy" )
git grep -niE 'mqtt|mosquitto|paho|amqp|kafka|broker' -- web/Gemfile web/Gemfile.lock | grep -viE 'via_?mqtt'
```
**Expected:** the rspec passes and the grep prints nothing. A backward walk over
`/api/nodes` still excludes opted-out nodes (`NODE_OPT_OUT_MARKER`) and, in private
mode, `CLIENT_HIDDEN` nodes — `before` only narrows, so it can never surface a row
the route would otherwise hide (**A2c**, Invariant II). No manifest gains a broker
dependency (Invariant I / **A1a**): the change is a read-side query param only.
### BP-A6 — Cursor documented; deferred scope recorded — BP1, BP8, BP9
```bash
git grep -n 'before' -- data/mesh_ingestor/CONTRACTS.md
```
**Expected:** the `CONTRACTS.md` "GET endpoint time windows" section documents the
`?before=` inclusive upper-bound cursor and names the six collections that accept
it. The deferred items in **BP9** are out of scope and must **not** appear in this
change: `/api/instances` still lacks `limit`/`since`/`protocol`, and
`/api/telemetry/aggregated` still uses camelCase `windowSeconds`/`bucketSeconds`
(no snake_case alias) — these stay tracked follow-ups, not regressions.
### BP-R1 — Regression: prior acceptance still holds
```bash
( cd web && bundle exec rspec ) && ( cd web && npm test )
( . .venv/bin/activate && pytest -q tests/ )
```
**Expected:** every prior check still passes. At risk and explicitly required to
remain green: **C7** (messages backward pagination — its keyset mechanism is now
shared by six more routes, but `/api/messages` behavior is unchanged); **C4**
(window floors — `before` only narrows, never widens); **A2 / A2a / A2c** (privacy
& opt-out — a narrowing upper bound exposes no hidden row, and `/api/messages`
still 404s in private mode); **A4a** (`KNOWN_PROTOCOLS` unchanged); **PL-A1 /
PL-A2** and **FC-A2** (the frontend message pager and cache seed-then-delta are
untouched — frontend `before` adoption is deferred per **BP9**); and **B1** (all
suites). No POST/event contract changes, so **C2** and the Python suite are
unaffected (the only `data/` touch is the `CONTRACTS.md` GET-window documentation).
+41
View File
@@ -342,3 +342,44 @@ helper + import-map builder) and the asset references in `views/layouts/app.erb`
| **AV3** | **Full module-graph busting via import map.** Because `?v=` on an entry-point URL does **not** propagate to that module's relative `import './x.js'`, the layout `<head>` emits exactly one `<script type="importmap">` (before any module loads) mapping every served production `/assets/js/**/*.js` module (excluding `__tests__`) to its `?v=<APP_VERSION>` URL. This invalidates the whole transitive graph (e.g. `main.js` + its 33 imports), not just the entry points. **Safety property:** a module absent from the map degrades to today's unversioned-but-working load — a missing entry can never break a working import. Browsers without import-map support degrade to today's behavior (entry points still busted via AV2). | interview |
| **AV4** | **Scope = JS + CSS only; native, no new egress.** Only executable/style assets are versioned (all JS + `base.css`). Images, favicons, and SVG icons keep today's `Last-Modified`/`ETag` revalidation (a stale logo is cosmetic, not behavioral). The mechanism uses the native browser import-map feature — **no new dependency, build step, external host, or analytics param** — so apex (I), privacy (II), federation (III), parity (IV), fixed-stack (D7), contract (D8), and no-phone-home (D11) all hold unchanged. The version query is not part of any `/api/*` contract (D8): Sinatra serves the same static file regardless of query string. | interview |
| **AV5** | **Engineering bar (D9).** The helper and import-map builder ship with 100% unit tests + RDoc, Apache headers, `rufo`-clean; all existing suites stay green. View/app specs that assert exact asset markup are **updated** to the versioned form, not removed. | CLAUDE.md |
---
## Feature: Uniform backward pagination (`?before=`) for bulk collection APIs
Generalizes the `/api/messages` backward-pagination cursor (issue #796, **C7**) to
the other five bulk collection GETs so a client can page **backward through** the
visibility window instead of stalling at the newest `MAX_QUERY_LIMIT` (1000) rows.
Motivated by an external consumer (`l5yth/meshint`) that could not retrieve more
than 1000 nodes from `GET /api/nodes`. Read-side only; integrates with the GET
routes in `web/lib/potato_mesh/application/routes/api.rb`, the query helpers in
`web/lib/potato_mesh/application/queries/` (`node_queries.rb`,
`telemetry_queries.rb`, `federation_queries.rb`; `common.rb` already provides the
`coerce_positive_or_nil` cursor coercion), and the GET-window documentation in
`data/mesh_ingestor/CONTRACTS.md`.
**Conflict check against existing decisions.** *D8 (stable contract + GET window
floors)* — **extends**: `before` is a new optional, additive parameter (absent ⇒
today's behavior) and only ever *narrows*, so it needs no version bump and no
federation-compat fallback (unlike the 0.7.0 `/api/stats` break, **S1**). *C4
(floors cannot be widened)* / *C7 (messages `before`)* — **extends**: the proven
#796 keyset cursor is applied uniformly; the `MAX(since, floor)` clamp and
`MAX_QUERY_LIMIT` cap are untouched. *Invariant I (apex)* — **consistent**:
read-side query param, no broker/dependency/egress. *Invariant II (privacy)* —
**consistent**: opt-out / `CLIENT_HIDDEN` / private-mode gates are unchanged and a
narrowing upper bound can never surface a hidden row; no new on-disk retention.
*Invariant IV (parity)* — **extends**: the cursor is protocol-neutral and composes
with the existing `?protocol=` filter. No invariant is contradicted, so this
feature adds new decisions without amending any prior one.
| # | Decision | Source |
| --- | --- | --- |
| **BP1** | **Uniform `?before=` on the six bulk collections.** `GET /api/{nodes, positions, telemetry, neighbors, traces, ingestors}` accept an optional `?before=<unix_seconds>` upper-bound cursor, mirroring the existing `GET /api/messages` behavior (**C7**). `before` is an **inclusive** (`<=`) ceiling on each route's **primary sort column** — the column it already `ORDER BY … DESC`: `rx_time` for `positions`/`telemetry`/`neighbors`/`traces`, `last_heard` for `nodes`, `last_seen_time` for `ingestors`. The per-id routes (`/api/.../:id`) and `/api/instances` are **out of scope**. | interview |
| **BP2** | **Narrows only — never widens (preserves C4/D8).** `before` composes with the existing server-side window floor as an *additional upper bound*: the effective window is `MAX(since, floor) ≤ t ≤ before`. Because `before` only removes *newer* rows, no value can reach past the 7-day / 28-day floor (a `before` older than the floor simply yields fewer/zero rows; a `before` in the future is a no-op). It therefore needs **no clamp** of its own (unlike `since`, which is clamped *up* to the floor). A non-positive or non-integer `before` is ignored as absent via the existing `coerce_positive_or_nil`, identical to messages. `MAX_QUERY_LIMIT` per request is unchanged — `before` pages *through* the window, not *beyond* it. | interview + code |
| **BP3** | **Keyset mechanics identical to messages (#796).** The caller walks newest → oldest: each page is `ORDER BY <sort_col> DESC LIMIT n`, then the **oldest `<sort_col>` value of the page** becomes the next `before`, de-duplicating by the collection's id client-side. The inclusive boundary deliberately overlaps consecutive pages by any rows sharing the boundary second so none is skipped; client dedup collapses the overlap. This is the established **C7 / PL-A2** walk applied uniformly. | interview + code |
| **BP4** | **Additive & backward-compatible — no version bump.** `before` is a new optional query parameter; when absent every route behaves exactly as today, and existing consumers (including the federation crawl) are unaffected. Unlike **S1**, this is *not* a breaking change: no manifest version bump and no old/new shape fallback are required. **Extends** D8's backward-compatibility rule rather than amending it. | interview |
| **BP5** | **Protocol-neutral (Invariant IV).** The cursor is identical for all rows regardless of protocol and composes with the existing `?protocol=` filter; neither Meshtastic nor MeshCore is privileged. | interview |
| **BP6** | **Apex & privacy untouched (Invariants I/II).** Read-side only — no broker, dependency, or egress (I). The existing opt-out, `CLIENT_HIDDEN`, and private-mode behaviors are unchanged, and because `before` only narrows it can never expose a row a route would otherwise hide (II). No new on-disk retention is introduced. | interview + code |
| **BP7** | **Cache-bypass parity with messages.** A request carrying `before` (like one carrying `since > 0`) **bypasses** the shared `ApiCache` response cache, which only memoises the default newest-page feed; the cache key for the cached (no-`before`, no-`since`) path is unchanged, so history pages never pollute or evict the hot newest-page entry. | code |
| **BP8** | **Engineering bar (D9).** Ships with 100% unit tests across the route and query layers (mirroring the existing messages-`before` specs), RDoc on every edited method, Apache headers intact, `rufo`-clean; all existing Ruby/JS/Python suites stay green. | CLAUDE.md |
| **BP9** | **Out of scope (deferred, tracked).** This feature ships the **server capability only**. (a) Wiring the browser data cache / JS data-fetchers to *backfill* collections via `before` (beyond the existing messages pager) is a tracked follow-up — the capability is unblocked, not wired. (b) The lone camelCase query params `windowSeconds` / `bucketSeconds` on `GET /api/telemetry/aggregated` and (c) the missing `limit` / `since` / `protocol` params on `GET /api/instances` are recorded here as known API-handling inconsistencies and tracked as **separate follow-ups**, deliberately excluded to keep this feature compartmentalized. | interview |
+37
View File
@@ -197,6 +197,43 @@ Federation peers should not assume an unbounded historical window: a peer that r
The constants live in `web/lib/potato_mesh/config.rb` (`week_seconds`, `four_weeks_seconds`).
### GET endpoint backward pagination (`?before=`)
The six bulk collection endpoints — `GET /api/nodes`, `/api/positions`,
`/api/telemetry`, `/api/neighbors`, `/api/traces`, and `/api/ingestors` — plus the
pre-existing `GET /api/messages` cursor accept an optional `?before=<unix_seconds>`
**inclusive upper-bound cursor** for backward pagination. It is the companion to
`?since=`: where `since` raises the lower bound of the window, `before` lowers the
upper bound. `before` bounds each route's **primary sort column** — the column it
already orders by, newest first:
| Route | `before` bounds |
| --- | --- |
| `GET /api/nodes` | `last_heard` |
| `GET /api/messages` | `rx_time` |
| `GET /api/positions` | `rx_time` |
| `GET /api/telemetry` | `rx_time` |
| `GET /api/neighbors` | `rx_time` |
| `GET /api/traces` | `rx_time` |
| `GET /api/ingestors` | `last_seen_time` |
To page backward through more than one `limit`-sized response (the per-request cap
is `MAX_QUERY_LIMIT` = 1000), walk newest → oldest: fetch a page, then re-request
with `before` set to the **oldest sort-column value** in the page just received,
de-duplicating rows by their id. The inclusive `<=` boundary intentionally repeats
any row that shares the boundary second, so none is skipped across the page break;
the client's id-dedup collapses the one-row overlap. Repeat until a short page
(fewer than `limit` rows) signals the window is exhausted. This is how a client
retrieves **every** in-window row instead of stalling at the newest 1000.
`before` **only ever narrows** the result set, so — exactly like `since` — it
**cannot widen** the window past the route's floor in the table above: a `before`
older than the floor merely returns fewer rows (the floor still clamps the lower
bound), and a `before` newer than "now" is a no-op. A non-positive or non-integer
`before` is ignored (treated as absent). The cursor composes with `?protocol=` and
is protocol-neutral. The per-id routes (`GET /api/.../:id`) and `GET /api/instances`
do **not** accept `before`.
### GET /api/stats response shape
> **Breaking change in 0.7.0.** Before 0.7.0 the payload was flat —
@@ -44,18 +44,11 @@ module PotatoMesh
where_clauses << "m.rx_time >= ?"
params << since_threshold
# Upper-bound cursor for backward pagination (issue #796). When set,
# +before+ is an *inclusive* ceiling on +rx_time+: the client walks it
# backward (newest -> oldest), passing the oldest +rx_time+ of each page
# as the next cursor and de-duplicating by +id+ client-side, so rows that
# share the boundary second are never skipped. Because the cursor only
# ever *narrows* the result set, the seven-day floor above still bounds
# the window — callers cannot use +before+ to reach further back.
before_cursor = coerce_positive_or_nil(before)
if before_cursor
where_clauses << "m.rx_time <= ?"
params << before_cursor
end
# Inclusive upper-bound cursor for backward pagination (issue #796;
# SPEC BP1-BP3), shared with every bulk collection via the helper. It
# only ever *narrows*, so the seven-day floor above still bounds the
# window — callers cannot use +before+ to reach further back.
append_before_filter(where_clauses, params, before, column: "m.rx_time")
unless include_encrypted
where_clauses << "COALESCE(TRIM(m.encrypted), '') = ''"
@@ -142,6 +142,40 @@ module PotatoMesh
params << protocol
end
# Append an inclusive upper-bound (+before+) cursor to an in-flight WHERE
# clause builder for backward pagination of a bulk collection endpoint
# (SPEC BP1-BP3; the +/api/messages+ #796 cursor generalised to every
# list route).
#
# +before+ is an *inclusive* (+<=+) ceiling on the route's primary sort
# column — the column it already +ORDER BY ... DESC+ — so a client can
# walk the feed newest -> oldest by passing the oldest sort value of each
# page as the next +before+ and de-duplicating by id, never skipping a row
# that shares the boundary second. Because the cursor only ever *narrows*
# the result set it cannot widen the server-side window floor
# (SPEC BP2 / acceptance C4): a +before+ older than the floor merely
# returns fewer rows, and a +before+ in the future is a no-op.
# Non-positive / non-integer values are ignored as absent via
# {coerce_positive_or_nil}, matching the messages route.
#
# @param where_clauses [Array<String>] accumulating WHERE conditions (mutated).
# @param params [Array] accumulating bind parameters (mutated).
# @param before [Object] raw +before+ cursor from the request.
# @param column [String] qualified sort column to bound (e.g. ``"last_heard"``,
# ``"m.rx_time"``). Must match {SAFE_COLUMN_IDENTIFIER}; arbitrary user
# input is not accepted.
# @return [Integer, nil] the coerced cursor that was applied, or +nil+ when
# +before+ was absent/invalid and no clause was added.
def append_before_filter(where_clauses, params, before, column:)
assert_safe_column_identifier!(column)
cursor = coerce_positive_or_nil(before)
return nil unless cursor
where_clauses << "#{column} <= ?"
params << cursor
cursor
end
# Normalise a caller-provided limit to a sane, positive integer.
#
# @param limit [Object] value coerced to an integer.
@@ -22,8 +22,10 @@ module PotatoMesh
# @param limit [Integer] maximum number of rows to return.
# @param node_ref [String, Integer, nil] optional node reference to scope results.
# @param since [Integer] unix timestamp threshold applied in addition to the rolling window.
# @param before [Integer, nil] inclusive upper-bound +rx_time+ cursor for
# backward pagination (SPEC BP1); rows newer than this are excluded.
# @return [Array<Hash>] compacted position rows suitable for API responses.
def query_positions(limit, node_ref: nil, since: 0, protocol: nil)
def query_positions(limit, node_ref: nil, since: 0, before: nil, protocol: nil)
limit = coerce_query_limit(limit)
db = open_database(readonly: true)
db.results_as_hash = true
@@ -44,6 +46,10 @@ module PotatoMesh
params.concat(clause.last)
end
# Inclusive upper-bound cursor for backward pagination (SPEC BP1);
# bounds the +rx_time+ sort column.
append_before_filter(where_clauses, params, before, column: "rx_time")
append_opt_out_filter(where_clauses, params, opt_out_node_id_filter("node_id"))
append_protocol_filter(where_clauses, params, protocol)
@@ -84,8 +90,10 @@ module PotatoMesh
# @param limit [Integer] maximum number of rows to return.
# @param node_ref [String, Integer, nil] optional node reference to scope results.
# @param since [Integer] unix timestamp threshold applied in addition to the rolling window for collections.
# @param before [Integer, nil] inclusive upper-bound +rx_time+ cursor for
# backward pagination (SPEC BP1); rows newer than this are excluded.
# @return [Array<Hash>] compacted neighbor rows suitable for API responses.
def query_neighbors(limit, node_ref: nil, since: 0, protocol: nil)
def query_neighbors(limit, node_ref: nil, since: 0, before: nil, protocol: nil)
limit = coerce_query_limit(limit)
db = open_database(readonly: true)
db.results_as_hash = true
@@ -110,6 +118,10 @@ module PotatoMesh
# Either endpoint of the neighbour relationship may carry the
# opt-out marker — filter both so a silenced node never appears as
# a source or destination of an RF link.
# Inclusive upper-bound cursor for backward pagination (SPEC BP1);
# bounds the +rx_time+ sort column.
append_before_filter(where_clauses, params, before, column: "rx_time")
append_opt_out_filter(where_clauses, params, opt_out_node_id_filter("node_id"))
append_opt_out_filter(where_clauses, params, opt_out_node_id_filter("neighbor_id"))
append_protocol_filter(where_clauses, params, protocol)
@@ -141,8 +153,10 @@ module PotatoMesh
# @param limit [Integer] maximum number of rows to return.
# @param node_ref [String, Integer, nil] optional node reference to scope results.
# @param since [Integer] unix timestamp threshold applied in addition to the rolling window.
# @param before [Integer, nil] inclusive upper-bound +rx_time+ cursor for
# backward pagination (SPEC BP1); rows newer than this are excluded.
# @return [Array<Hash>] compacted trace rows suitable for API responses.
def query_traces(limit, node_ref: nil, since: 0, protocol: nil)
def query_traces(limit, node_ref: nil, since: 0, before: nil, protocol: nil)
limit = coerce_query_limit(limit)
db = open_database(readonly: true)
db.results_as_hash = true
@@ -173,6 +187,10 @@ module PotatoMesh
# filtered separately at hydration time so a trace that only relays
# through a silenced node still surfaces with the offending hop
# removed.
# Inclusive upper-bound cursor for backward pagination (SPEC BP1);
# bounds the +rx_time+ sort column.
append_before_filter(where_clauses, params, before, column: "rx_time")
append_opt_out_filter(where_clauses, params, opt_out_node_num_filter("src"))
append_opt_out_filter(where_clauses, params, opt_out_node_num_filter("dest"))
append_protocol_filter(where_clauses, params, protocol)
@@ -135,8 +135,10 @@ module PotatoMesh
# @param limit [Integer] maximum number of rows to return.
# @param node_ref [String, Integer, nil] optional node reference to narrow results.
# @param since [Integer] unix timestamp threshold applied in addition to the rolling window for collections.
# @param before [Integer, nil] inclusive upper-bound +last_heard+ cursor for
# backward pagination (SPEC BP1); nodes newer than this are excluded.
# @return [Array<Hash>] compacted node rows suitable for API responses.
def query_nodes(limit, node_ref: nil, since: 0, protocol: nil)
def query_nodes(limit, node_ref: nil, since: 0, before: nil, protocol: nil)
limit = coerce_query_limit(limit)
db = open_database(readonly: true)
db.results_as_hash = true
@@ -159,6 +161,13 @@ module PotatoMesh
params << since_threshold
end
# Inclusive upper-bound cursor for backward pagination (SPEC BP1-BP3).
# Bounds the +last_heard+ sort column so callers can page newest ->
# oldest past the +MAX_QUERY_LIMIT+ cap; only ever narrows (BP2). The
# per-id route never supplies +before+, so single-node lookups are
# unaffected.
append_before_filter(where_clauses, params, before, column: "last_heard")
if private_mode?
where_clauses << "(role IS NULL OR role <> 'CLIENT_HIDDEN')"
end
@@ -224,8 +233,10 @@ module PotatoMesh
#
# @param limit [Integer] maximum number of ingestors to return.
# @param since [Integer] unix timestamp threshold applied in addition to the rolling window for collections.
# @param before [Integer, nil] inclusive upper-bound +last_seen_time+ cursor
# for backward pagination (SPEC BP1); ingestors newer than this are excluded.
# @return [Array<Hash>] compacted ingestor rows suitable for API responses.
def query_ingestors(limit, since: 0, protocol: nil)
def query_ingestors(limit, since: 0, before: nil, protocol: nil)
limit = coerce_query_limit(limit)
db = open_database(readonly: true)
db.results_as_hash = true
@@ -237,6 +248,9 @@ module PotatoMesh
since_threshold = normalize_since_threshold(since, floor: cutoff)
where_clauses = ["last_seen_time >= ?"]
params = [since_threshold]
# Inclusive upper-bound cursor for backward pagination (SPEC BP1);
# bounds the +last_seen_time+ sort column.
append_before_filter(where_clauses, params, before, column: "last_seen_time")
append_opt_out_filter(where_clauses, params, opt_out_node_id_filter("node_id"))
append_protocol_filter(where_clauses, params, protocol)
sql = <<~SQL
@@ -22,8 +22,10 @@ module PotatoMesh
# @param limit [Integer] maximum number of rows to return.
# @param node_ref [String, Integer, nil] optional node reference to scope results.
# @param since [Integer] unix timestamp threshold applied in addition to the rolling window for collections.
# @param before [Integer, nil] inclusive upper-bound +rx_time+ cursor for
# backward pagination (SPEC BP1); rows newer than this are excluded.
# @return [Array<Hash>] compacted telemetry rows suitable for API responses.
def query_telemetry(limit, node_ref: nil, since: 0, protocol: nil)
def query_telemetry(limit, node_ref: nil, since: 0, before: nil, protocol: nil)
limit = coerce_query_limit(limit)
db = open_database(readonly: true)
db.results_as_hash = true
@@ -44,6 +46,10 @@ module PotatoMesh
params.concat(clause.last)
end
# Inclusive upper-bound cursor for backward pagination (SPEC BP1);
# bounds the +rx_time+ sort column.
append_before_filter(where_clauses, params, before, column: "rx_time")
append_opt_out_filter(where_clauses, params, opt_out_node_id_filter("node_id"))
append_protocol_filter(where_clauses, params, protocol)
+28 -14
View File
@@ -89,10 +89,15 @@ module PotatoMesh
since = params["since"]
protocol = sanitize_protocol(params["protocol"])
since_val = coerce_integer(since) || 0
# Inclusive upper-bound cursor for backward pagination (SPEC BP1). A
# request carrying +before+ is a history page, so — like a +since+
# query — it bypasses the shared cache (which only memoises the
# default newest-page feed).
before = coerce_positive_or_nil(params["before"])
priv = private_mode? ? 1 : 0
if since_val > 0
json_body = query_nodes(limit, since: since, protocol: protocol).to_json
if since_val > 0 || before
json_body = query_nodes(limit, since: since, before: before, protocol: protocol).to_json
etag Digest::MD5.hexdigest(json_body), kind: :weak
api_cache_control
json_body
@@ -139,9 +144,11 @@ module PotatoMesh
protocol = sanitize_protocol(params["protocol"])
since = params["since"]
since_val = coerce_integer(since) || 0
# Backward-pagination cursor (SPEC BP1); bypasses the cache like +since+.
before = coerce_positive_or_nil(params["before"])
if since_val > 0
json_body = query_ingestors(limit, since: since, protocol: protocol).to_json
if since_val > 0 || before
json_body = query_ingestors(limit, since: since, before: before, protocol: protocol).to_json
etag Digest::MD5.hexdigest(json_body), kind: :weak
api_cache_control
json_body
@@ -164,8 +171,7 @@ module PotatoMesh
# Upper-bound cursor for backward pagination (issue #796). A request
# carrying +before+ is a history page, so it bypasses the shared
# response cache (which only memoises the default newest-page feed).
before = coerce_integer(params["before"])
before = nil if before && before <= 0
before = coerce_positive_or_nil(params["before"])
protocol = sanitize_protocol(params["protocol"])
enc_key = include_encrypted ? "1" : "0"
@@ -210,9 +216,11 @@ module PotatoMesh
since = params["since"]
protocol = sanitize_protocol(params["protocol"])
since_val = coerce_integer(since) || 0
# Backward-pagination cursor (SPEC BP1); bypasses the cache like +since+.
before = coerce_positive_or_nil(params["before"])
if since_val > 0
json_body = query_positions(limit, since: since, protocol: protocol).to_json
if since_val > 0 || before
json_body = query_positions(limit, since: since, before: before, protocol: protocol).to_json
etag Digest::MD5.hexdigest(json_body), kind: :weak
api_cache_control
json_body
@@ -243,9 +251,11 @@ module PotatoMesh
since = params["since"]
protocol = sanitize_protocol(params["protocol"])
since_val = coerce_integer(since) || 0
# Backward-pagination cursor (SPEC BP1); bypasses the cache like +since+.
before = coerce_positive_or_nil(params["before"])
if since_val > 0
json_body = query_neighbors(limit, since: since, protocol: protocol).to_json
if since_val > 0 || before
json_body = query_neighbors(limit, since: since, before: before, protocol: protocol).to_json
etag Digest::MD5.hexdigest(json_body), kind: :weak
api_cache_control
json_body
@@ -276,9 +286,11 @@ module PotatoMesh
since = params["since"]
protocol = sanitize_protocol(params["protocol"])
since_val = coerce_integer(since) || 0
# Backward-pagination cursor (SPEC BP1); bypasses the cache like +since+.
before = coerce_positive_or_nil(params["before"])
if since_val > 0
json_body = query_telemetry(limit, since: since, protocol: protocol).to_json
if since_val > 0 || before
json_body = query_telemetry(limit, since: since, before: before, protocol: protocol).to_json
etag Digest::MD5.hexdigest(json_body), kind: :weak
api_cache_control
json_body
@@ -362,9 +374,11 @@ module PotatoMesh
since = params["since"]
protocol = sanitize_protocol(params["protocol"])
since_val = coerce_integer(since) || 0
# Backward-pagination cursor (SPEC BP1); bypasses the cache like +since+.
before = coerce_positive_or_nil(params["before"])
if since_val > 0
json_body = query_traces(limit, since: since, protocol: protocol).to_json
if since_val > 0 || before
json_body = query_traces(limit, since: since, before: before, protocol: protocol).to_json
etag Digest::MD5.hexdigest(json_body), kind: :weak
api_cache_control
json_body
+336
View File
@@ -112,6 +112,39 @@ RSpec.describe "Potato Mesh Sinatra app" do
ensure_self_instance_record!
end
# Walk a bulk collection endpoint backward with the +before+ cursor the way an
# API client must, returning the set of unique record ids recovered. Generic
# over the route path, the id field used for client-side de-duplication, and
# the primary sort field the cursor advances on (SPEC BP1-BP3).
#
# @param path [String] collection route, e.g. ``"/api/positions"``.
# @param id_key [String] response field uniquely identifying a row.
# @param sort_key [String] response field the +before+ cursor bounds.
# @return [Array] the de-duplicated ids recovered across every page.
def walk_before(path, id_key:, sort_key:)
cap = PotatoMesh::App::Queries::MAX_QUERY_LIMIT
seen = {}
cursor = nil
pages = 0
loop do
url = "#{path}?limit=#{cap}"
url += "&before=#{cursor}" if cursor
get url
expect(last_response).to be_ok
rows = JSON.parse(last_response.body)
# The per-request cap is never exceeded by a single response.
expect(rows.size).to be <= cap
added = rows.reject { |row| seen.key?(row[id_key]) }
added.each { |row| seen[row[id_key]] = true }
pages += 1
break if rows.size < cap # window exhausted
break if added.empty? # no progress (server ignoring `before`)
break if pages >= 10 # hard safety bound against an infinite loop
cursor = rows.map { |row| row[sort_key] }.min
end
seen.keys
end
# Retrieve the number of rows stored in the instances table.
#
# @return [Integer] count of stored instance records.
@@ -6107,6 +6140,204 @@ RSpec.describe "Potato Mesh Sinatra app" do
end
describe "GET /api/nodes" do
# Regression-style coverage for SPEC BP1-BP6: more than MAX_QUERY_LIMIT
# nodes inside the seven-day window must all remain reachable by paging
# backward with an inclusive `before` cursor (the `meshint` 1000-node cap).
it "exposes every in-window node through before pagination" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
cap = PotatoMesh::App::Queries::MAX_QUERY_LIMIT
total = cap + 500
# Seed more than one page of nodes, all inside the seven-day window, each
# with a distinct last_heard so the keyset cursor is unambiguous.
with_db do |db|
db.transaction do
total.times do |i|
lh = now - 60 - i * 30
db.execute(
"INSERT INTO nodes(node_id, num, short_name, long_name, hw_model, role, last_heard, first_heard) VALUES(?,?,?,?,?,?,?,?)",
["!%08x" % (0x1000 + i), 0x1000 + i, "n#{i}", "Node #{i}", "TBEAM", "CLIENT", lh, lh],
)
end
end
end
# Walk the listing the way an external client (l5yth/meshint) must: pull a
# page, then ask for everything at-or-before the oldest last_heard already
# seen. Without an upper-bound cursor the server cannot return anything
# past the newest `cap` rows, so the walk would stall before reaching them.
recovered = walk_before("/api/nodes", id_key: "node_id", sort_key: "last_heard")
expect(recovered.size).to eq(total)
end
it "before cannot widen the window past the floor and ignores a non-positive cursor" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
stale_last = now - (PotatoMesh::Config.week_seconds + 4 * 60 * 60)
fresh_last = now - 30
with_db do |db|
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard) VALUES(?,?,?,?,?,?,?)",
["!stalenode", "stal", "Stale", "TBEAM", "CLIENT", stale_last, stale_last],
)
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard) VALUES(?,?,?,?,?,?,?)",
["!freshnode", "frsh", "Fresh", "TBEAM", "CLIENT", fresh_last, fresh_last],
)
end
# A `before` at "now" cannot reach the stale row: the seven-day floor still
# clamps the lower bound, so `before` only ever narrows, never widens.
get "/api/nodes?before=#{now}"
expect(last_response).to be_ok
ids = JSON.parse(last_response.body).map { |r| r["node_id"] }
expect(ids).to include("!freshnode")
expect(ids).not_to include("!stalenode")
# A `before` older than the floor returns nothing beyond the floor (the only
# in-window row is newer than this ceiling, so the result is empty).
get "/api/nodes?before=#{stale_last}"
expect(last_response).to be_ok
expect(JSON.parse(last_response.body)).to be_empty
# A non-positive / non-integer `before` is ignored (treated as absent), so
# the fresh row still comes back rather than being filtered by a bogus
# ceiling — parity with the /api/messages coercion.
["0", "-5", "abc"].each do |bogus|
get "/api/nodes?before=#{bogus}"
expect(last_response).to be_ok
ids = JSON.parse(last_response.body).map { |r| r["node_id"] }
expect(ids).to include("!freshnode"), "before=#{bogus} should be ignored"
end
end
it "before pagination boundary is inclusive and protocol-neutral" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
boundary = now - 100
with_db do |db|
# Two nodes sharing the exact boundary last_heard, one per protocol, plus
# a newer node above the boundary.
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard, protocol) VALUES(?,?,?,?,?,?,?,?)",
["!edgecore0", "ec", "Edge Core", "TBEAM", "CLIENT", boundary, boundary, "meshcore"],
)
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard, protocol) VALUES(?,?,?,?,?,?,?,?)",
["!edgetast0", "et", "Edge Tastic", "TBEAM", "CLIENT", boundary, boundary, "meshtastic"],
)
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard, protocol) VALUES(?,?,?,?,?,?,?,?)",
["!newer0001", "nw", "Newer", "TBEAM", "CLIENT", now - 10, now - 10, "meshcore"],
)
end
# Inclusive <= ceiling: both rows sharing the boundary second are returned
# when the boundary is used as `before` (the one-row overlap a client dedups).
get "/api/nodes?before=#{boundary}"
expect(last_response).to be_ok
ids = JSON.parse(last_response.body).map { |r| r["node_id"] }
expect(ids).to include("!edgecore0", "!edgetast0")
expect(ids).not_to include("!newer0001")
# `before` composes with `protocol`, privileging neither protocol.
get "/api/nodes?before=#{boundary}&protocol=meshcore"
expect(last_response).to be_ok
ids = JSON.parse(last_response.body).map { |r| r["node_id"] }
expect(ids).to eq(["!edgecore0"])
end
it "before pagination honors privacy and opt-out filters" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
marker = PotatoMesh::Config.node_opt_out_marker
with_db do |db|
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard) VALUES(?,?,?,?,?,?,?)",
["!plainnode", "pln", "Plain", "TBEAM", "CLIENT", now - 50, now - 50],
)
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard) VALUES(?,?,?,?,?,?,?)",
["!optoutnod", "opt", "Quiet #{marker} One", "TBEAM", "CLIENT", now - 60, now - 60],
)
db.execute(
"INSERT INTO nodes(node_id, short_name, long_name, hw_model, role, last_heard, first_heard) VALUES(?,?,?,?,?,?,?)",
["!hiddennod", "hid", "Hidden", "TBEAM", "CLIENT_HIDDEN", now - 70, now - 70],
)
end
# A backward-paginated request still applies the opt-out filter: a node
# carrying the marker never appears even when within the `before` ceiling.
get "/api/nodes?before=#{now}"
expect(last_response).to be_ok
ids = JSON.parse(last_response.body).map { |r| r["node_id"] }
expect(ids).to include("!plainnode")
expect(ids).not_to include("!optoutnod")
# In private mode the CLIENT_HIDDEN exclusion also still applies under a
# `before` walk — narrowing can never surface a hidden row (Invariant II).
allow(PotatoMesh::Config).to receive(:private_mode_enabled?).and_return(true)
get "/api/nodes?before=#{now}"
expect(last_response).to be_ok
ids = JSON.parse(last_response.body).map { |r| r["node_id"] }
expect(ids).to include("!plainnode")
expect(ids).not_to include("!hiddennod")
expect(ids).not_to include("!optoutnod")
end
it "before bypasses the response cache" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
# Start from a clean cache so a prior example's entry cannot mask the
# behaviour under test (the store is process-global with a monotonic TTL).
PotatoMesh::App::ApiCache.invalidate_all
with_db do |db|
db.execute(
"INSERT INTO nodes(node_id, last_heard, first_heard) VALUES(?,?,?)",
["!cacheaaaa", now - 50, now - 50],
)
end
# Prime the newest-page cache with just node A.
get "/api/nodes?limit=3"
expect(last_response).to be_ok
expect(JSON.parse(last_response.body).map { |r| r["node_id"] }).to eq(["!cacheaaaa"])
# Insert node B directly (no POST, so the cache is not invalidated).
with_db do |db|
db.execute(
"INSERT INTO nodes(node_id, last_heard, first_heard) VALUES(?,?,?)",
["!cachebbbb", now - 40, now - 40],
)
end
# A plain request is still served the cached page (proves the cache is live).
get "/api/nodes?limit=3"
expect(JSON.parse(last_response.body).map { |r| r["node_id"] }).to eq(["!cacheaaaa"])
# A `before` request bypasses the cache and sees the fresh node B.
get "/api/nodes?limit=3&before=#{now}"
expect(last_response).to be_ok
before_ids = JSON.parse(last_response.body).map { |r| r["node_id"] }
expect(before_ids).to include("!cacheaaaa", "!cachebbbb")
# Issuing the `before` request did not overwrite the hot newest-page cache
# entry — a subsequent plain request still returns the cached page.
get "/api/nodes?limit=3"
expect(JSON.parse(last_response.body).map { |r| r["node_id"] }).to eq(["!cacheaaaa"])
end
it "returns the stored nodes with derived timestamps" do
import_nodes_fixture
@@ -6968,6 +7199,26 @@ RSpec.describe "Potato Mesh Sinatra app" do
end
describe "GET /api/positions" do
it "exposes every in-window row through before pagination" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
total = PotatoMesh::App::Queries::MAX_QUERY_LIMIT + 5
with_db do |db|
db.transaction do
total.times do |i|
rx = now - 60 - i * 30
db.execute(
"INSERT INTO positions(id, node_id, rx_time, rx_iso, latitude, longitude) VALUES(?,?,?,?,?,?)",
[1000 + i, "!pos00001", rx, Time.at(rx).utc.iso8601, 1.0, 2.0],
)
end
end
end
recovered = walk_before("/api/positions", id_key: "id", sort_key: "rx_time")
expect(recovered.size).to eq(total)
end
it "returns stored positions ordered by receive time" do
node_id = "!specfetch"
rx_times = [reference_time.to_i - 50, reference_time.to_i - 10]
@@ -7122,6 +7373,31 @@ RSpec.describe "Potato Mesh Sinatra app" do
end
describe "GET /api/neighbors" do
it "exposes every in-window row through before pagination" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
total = PotatoMesh::App::Queries::MAX_QUERY_LIMIT + 5
with_db do |db|
db.transaction do
# neighbors carries FK(node_id) and FK(neighbor_id) -> nodes(node_id),
# so both endpoints must exist as nodes before the relationship rows.
db.execute("INSERT INTO nodes(node_id, last_heard, first_heard) VALUES(?,?,?)", ["!nbr00001", now, now])
total.times do |i|
rx = now - 60 - i * 30
nbr = "!%08x" % (0x20000 + i)
db.execute("INSERT INTO nodes(node_id, last_heard, first_heard) VALUES(?,?,?)", [nbr, rx, rx])
db.execute(
"INSERT INTO neighbors(node_id, neighbor_id, rx_time) VALUES(?,?,?)",
["!nbr00001", nbr, rx],
)
end
end
end
recovered = walk_before("/api/neighbors", id_key: "neighbor_id", sort_key: "rx_time")
expect(recovered.size).to eq(total)
end
it "excludes neighbor records older than twenty-eight days from both bulk and per-id queries" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
@@ -7261,6 +7537,26 @@ RSpec.describe "Potato Mesh Sinatra app" do
end
describe "GET /api/telemetry" do
it "exposes every in-window row through before pagination" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
total = PotatoMesh::App::Queries::MAX_QUERY_LIMIT + 5
with_db do |db|
db.transaction do
total.times do |i|
rx = now - 60 - i * 30
db.execute(
"INSERT INTO telemetry(id, node_id, rx_time, rx_iso) VALUES(?,?,?,?)",
[1000 + i, "!tel00001", rx, Time.at(rx).utc.iso8601],
)
end
end
end
recovered = walk_before("/api/telemetry", id_key: "id", sort_key: "rx_time")
expect(recovered.size).to eq(total)
end
it "returns stored telemetry ordered by receive time" do
post "/api/telemetry", telemetry_fixture.to_json, auth_headers
expect(last_response.status).to eq(201)
@@ -7680,6 +7976,26 @@ RSpec.describe "Potato Mesh Sinatra app" do
end
describe "GET /api/traces" do
it "exposes every in-window row through before pagination" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
total = PotatoMesh::App::Queries::MAX_QUERY_LIMIT + 5
with_db do |db|
db.transaction do
total.times do |i|
rx = now - 60 - i * 30
db.execute(
"INSERT INTO traces(id, rx_time, rx_iso) VALUES(?,?,?)",
[1000 + i, rx, Time.at(rx).utc.iso8601],
)
end
end
end
recovered = walk_before("/api/traces", id_key: "id", sort_key: "rx_time")
expect(recovered.size).to eq(total)
end
it "returns stored traces ordered by receive time" do
clear_database
post "/api/traces", trace_fixture.to_json, auth_headers
@@ -7785,6 +8101,26 @@ RSpec.describe "Potato Mesh Sinatra app" do
end
describe "GET /api/ingestors" do
it "exposes every in-window row through before pagination" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)
now = reference_time.to_i
total = PotatoMesh::App::Queries::MAX_QUERY_LIMIT + 5
with_db do |db|
db.transaction do
total.times do |i|
ls = now - 60 - i * 30
db.execute(
"INSERT INTO ingestors(node_id, start_time, last_seen_time, version) VALUES(?,?,?,?)",
["!%08x" % (0x30000 + i), ls, ls, "1.0.0"],
)
end
end
end
recovered = walk_before("/api/ingestors", id_key: "node_id", sort_key: "last_seen_time")
expect(recovered.size).to eq(total)
end
it "uses the twenty-eight-day extended window so slow-tick ingestors stay visible" do
clear_database
allow(Time).to receive(:now).and_return(reference_time)