mirror of
https://github.com/pe1hvh/meshcore-gui.git
synced 2026-11-03 11:02:30 +01:00
fix(repeater): retry failed stats poll up to a configurable limit
WHAT Repeater statistics polls now retry within the same poll instead of recording a failure after a single session. REPEATER_POLL_MAX_ATTEMPTS (default 5) and REPEATER_POLL_RETRY_DELAY (default 5.0) control the budget. The login/status/logout sequence moved into a private _attempt_session(); _poll_one() drives the loop and writes exactly one archive record carrying the new "attempts" field and the last error. A missing password is not retried. Version bumped to 1.24.2; the stale __version__ = "5.0" in __init__.py was aligned with config.VERSION. WHY Polls fail at irregular intervals with no established root cause. This is an explicit workaround: retrying recovers readings now, while the attempt count makes the underlying failure rate measurable so the real cause can be identified from data instead of guessed at. NOTES No API or schema break — add_measurement() gained a keyword argument with a default, and "attempts" is an added field that older records simply lack. A poll blocks the worker loop for its full duration, so worst-case command latency rises from roughly 60 seconds to roughly three minutes for a repeater that never answers. Airtime rises proportionally: up to five login/logout round trips per failing poll.
This commit is contained in:
1 parent
d5e9331d4b
commit
fd3d0d2f35
5 files changed
+200
-26
No files matched your search
@@ -11,6 +11,61 @@ Format follows [Keep a Changelog](https://keepachangelog.com/) and [Semantic Ver
|
||||
---
|
||||
|
||||
|
||||
## [1.24.2] - 2026-09-02 — Repeater poll retries
|
||||
|
||||
### Added
|
||||
- ✨ **`REPEATER_POLL_MAX_ATTEMPTS`** (`config.py`, default `5`): maximum
|
||||
number of attempts per poll of a single repeater. One attempt is a
|
||||
complete session — login, status request, logout. Set to `1` to
|
||||
restore the previous single-shot behaviour.
|
||||
- ✨ **`REPEATER_POLL_RETRY_DELAY`** (`config.py`, default `5.0`): seconds
|
||||
between two attempts within the same poll.
|
||||
- ✨ **`attempts` field in the repeater stats record**
|
||||
(`services/repeater_stats_archive.py`): number of sessions used for
|
||||
this result, so a repeater that only answers on the third try is
|
||||
distinguishable from one that answers immediately. Zero when no
|
||||
session was started, such as when no password is configured.
|
||||
|
||||
### Changed
|
||||
- 🔁 **`RepeaterPoller._poll_one()` now retries a failed session**
|
||||
(`services/repeater_poller.py`). The login/status/logout sequence moved
|
||||
into a new private `_attempt_session()`; `_poll_one()` drives the retry
|
||||
loop and writes exactly one archive record for the poll as a whole,
|
||||
carrying the attempt count and the error of the last attempt. A missing
|
||||
password is not retried — that is a configuration error, not a
|
||||
transient one. The path reset after an unconfirmed login still runs per
|
||||
attempt, so a second attempt goes out as a flood.
|
||||
- 🔢 Version bumped to 1.24.2 in `config.py`. `__init__.py` carried a
|
||||
stale `__version__ = "5.0"` that matched no release and is now aligned.
|
||||
|
||||
### Impact
|
||||
- No API or schema breaks. `add_measurement()` gained a keyword argument
|
||||
with a default, and `attempts` is an added field — records written
|
||||
before 1.24.2 simply lack it and should be read as unknown, not as one.
|
||||
- A poll now blocks the worker loop for longer. `_poll_repeaters()` is
|
||||
awaited inline in the worker main loop, so with the defaults the worst
|
||||
case rises from roughly 60 seconds to roughly 3 minutes for a repeater
|
||||
that never answers. During that window queued commands are not
|
||||
processed. Lower `REPEATER_POLL_MAX_ATTEMPTS` or
|
||||
`REPEATER_LOGIN_TIMEOUT` if that latency matters more than the extra
|
||||
chance of a reading.
|
||||
- Airtime rises proportionally: up to five login/logout round trips per
|
||||
failing poll instead of one.
|
||||
- The manual poll from the REPEATERS panel goes through the same path and
|
||||
inherits the retries. Its notification still clears after 3 seconds, so
|
||||
it disappears before a retrying poll finishes.
|
||||
|
||||
### Rationale
|
||||
Repeater statistics polls fail at irregular intervals and the cause is
|
||||
not established — it could be a stale path, radio congestion, repeater
|
||||
firmware or a timeout that is too tight. This release is an explicit
|
||||
workaround, not a fix: retrying buys readings now, while the `attempts`
|
||||
count makes the underlying failure rate measurable so the real cause can
|
||||
be identified from data rather than guessed at.
|
||||
|
||||
---
|
||||
|
||||
|
||||
## [1.24.1] - 2026-09-01 — Manual poll notification never disappeared
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -5,4 +5,4 @@ A graphical user interface for MeshCore mesh network devices,
|
||||
communicating via USB serial or Bluetooth Low Energy (BLE).
|
||||
"""
|
||||
|
||||
__version__ = "5.0"
|
||||
__version__ = "1.24.2"
|
||||
+14
-1
@@ -25,7 +25,7 @@ from typing import Any, Dict, List
|
||||
# ==============================================================================
|
||||
|
||||
|
||||
VERSION: str = "1.24.1"
|
||||
VERSION: str = "1.24.2"
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
@@ -470,6 +470,19 @@ REPEATER_LOGIN_TIMEOUT: float = 30.0
|
||||
# Minimum seconds to wait for the STATUS_RESPONSE after a status request.
|
||||
REPEATER_STATUS_TIMEOUT: float = 30.0
|
||||
|
||||
# Maximum number of attempts per poll of a single repeater.
|
||||
# One attempt is a complete session: login, status request, logout. A
|
||||
# repeater that fails to answer is retried up to this many times before
|
||||
# the poll is recorded as failed. Set to 1 to disable retries entirely.
|
||||
# Note that a poll blocks the worker loop for its whole duration, so the
|
||||
# worst case is roughly MAX_ATTEMPTS x (LOGIN_TIMEOUT + RETRY_DELAY).
|
||||
REPEATER_POLL_MAX_ATTEMPTS: int = 5
|
||||
|
||||
# Seconds to wait between two attempts within the same poll.
|
||||
# Deliberately short: a longer pause does not make a stale path recover
|
||||
# any faster, and the worker loop is blocked while the poll runs.
|
||||
REPEATER_POLL_RETRY_DELAY: float = 5.0
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# CHANNEL LIST SORT
|
||||
|
||||
@@ -29,6 +29,28 @@ relearns a route from the ACK. This mirrors what the library itself does
|
||||
in ``send_msg_with_retry`` after repeated failures. ``reset_path`` is a
|
||||
device-local command and costs no airtime.
|
||||
|
||||
Retries
|
||||
~~~~~~~
|
||||
A repeater that does not answer is retried within the same poll, up to
|
||||
``REPEATER_POLL_MAX_ATTEMPTS`` times with ``REPEATER_POLL_RETRY_DELAY``
|
||||
seconds in between. An attempt is the complete sequence above, logout
|
||||
included: the login has to be redone because a session that never
|
||||
confirmed cannot be assumed to exist. Combined with the path reset this
|
||||
means the second attempt usually goes out as a flood, which is what
|
||||
recovers a repeater whose stored route went stale.
|
||||
|
||||
Only one record is written per poll, carrying the attempt count and the
|
||||
error of the last attempt. Recording every individual attempt would
|
||||
skew the success ratio in the archive and grow the file several times
|
||||
faster without adding information the count does not already give.
|
||||
|
||||
A missing password is not retried — that is a configuration error, not
|
||||
a transient one.
|
||||
|
||||
This retry is a workaround, not a fix. It masks whatever makes the
|
||||
repeater unreachable rather than addressing it; the attempt count in the
|
||||
archive is there to make that underlying failure rate measurable.
|
||||
|
||||
Spreading
|
||||
~~~~~~~~~
|
||||
At most one repeater is polled per call, and each repeater gets its own
|
||||
@@ -48,11 +70,14 @@ The poller never logs, returns or stores the password.
|
||||
SPDX-License-Identifier: MIT
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from typing import Any, Dict, Optional
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
from meshcore_gui.config import (
|
||||
REPEATER_LOGIN_TIMEOUT,
|
||||
REPEATER_POLL_MAX_ATTEMPTS,
|
||||
REPEATER_POLL_RETRY_DELAY,
|
||||
REPEATER_STATUS_TIMEOUT,
|
||||
debug_print,
|
||||
)
|
||||
@@ -190,7 +215,11 @@ class RepeaterPoller:
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _poll_one(self, mc, info: RepeaterInfo) -> None:
|
||||
"""Run one full session against a single repeater.
|
||||
"""Poll a single repeater, retrying a failed session.
|
||||
|
||||
Runs :meth:`_attempt_session` until it yields a status or the
|
||||
configured attempt budget is exhausted, then writes exactly one
|
||||
archive record for the poll as a whole.
|
||||
|
||||
Args:
|
||||
mc: Connected ``MeshCore`` instance.
|
||||
@@ -201,13 +230,83 @@ class RepeaterPoller:
|
||||
|
||||
if password is None:
|
||||
self._archive.add_measurement(
|
||||
info.pubkey, info.name, error="no_password_configured"
|
||||
info.pubkey,
|
||||
info.name,
|
||||
error="no_password_configured",
|
||||
attempts=0,
|
||||
)
|
||||
return
|
||||
|
||||
# A misconfigured zero or negative budget still buys one attempt;
|
||||
# silently polling nothing at all would be harder to diagnose.
|
||||
max_attempts = max(1, int(REPEATER_POLL_MAX_ATTEMPTS))
|
||||
last_error = "no_attempt_made"
|
||||
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
status, last_error = await self._attempt_session(
|
||||
mc, info, password, label, attempt, max_attempts
|
||||
)
|
||||
|
||||
if status is not None:
|
||||
if attempt > 1:
|
||||
debug_print(
|
||||
f"RepeaterPoller: {label} answered on attempt "
|
||||
f"{attempt}/{max_attempts}"
|
||||
)
|
||||
self._archive.add_measurement(
|
||||
info.pubkey,
|
||||
info.name,
|
||||
status=status,
|
||||
attempts=attempt,
|
||||
)
|
||||
return
|
||||
|
||||
if attempt < max_attempts and REPEATER_POLL_RETRY_DELAY > 0:
|
||||
await asyncio.sleep(REPEATER_POLL_RETRY_DELAY)
|
||||
|
||||
debug_print(
|
||||
f"RepeaterPoller: {label} gave up after {max_attempts} "
|
||||
f"attempt(s) — last error: {last_error}"
|
||||
)
|
||||
self._archive.add_measurement(
|
||||
info.pubkey,
|
||||
info.name,
|
||||
error=last_error,
|
||||
attempts=max_attempts,
|
||||
)
|
||||
|
||||
async def _attempt_session(
|
||||
self,
|
||||
mc,
|
||||
info: RepeaterInfo,
|
||||
password: str,
|
||||
label: str,
|
||||
attempt: int,
|
||||
max_attempts: int,
|
||||
) -> Tuple[Optional[Dict[str, Any]], str]:
|
||||
"""Run one complete session against a repeater.
|
||||
|
||||
Logs in, requests the status and always logs out again. Writes
|
||||
nothing to the archive: the caller decides what a failed attempt
|
||||
means once the whole poll is done.
|
||||
|
||||
Args:
|
||||
mc: Connected ``MeshCore`` instance.
|
||||
info: Repeater to poll.
|
||||
password: Login password for this repeater.
|
||||
label: Display label used in debug output.
|
||||
attempt: Number of this attempt, starting at one.
|
||||
max_attempts: Attempt budget, for debug output only.
|
||||
|
||||
Returns:
|
||||
Tuple of the cleaned status and an empty string on success,
|
||||
or ``None`` and a short failure reason on failure.
|
||||
"""
|
||||
suffix = f" (attempt {attempt}/{max_attempts})"
|
||||
login_attempted = False
|
||||
|
||||
try:
|
||||
debug_print(f"RepeaterPoller: login → {label}")
|
||||
debug_print(f"RepeaterPoller: login → {label}{suffix}")
|
||||
login_attempted = True
|
||||
login_event = await mc.commands.send_login_sync(
|
||||
info.pubkey,
|
||||
@@ -216,34 +315,29 @@ class RepeaterPoller:
|
||||
)
|
||||
|
||||
if login_event is None:
|
||||
debug_print(f"RepeaterPoller: no login confirmation from {label}")
|
||||
self._archive.add_measurement(
|
||||
info.pubkey, info.name, error="login_failed_or_timeout"
|
||||
debug_print(
|
||||
f"RepeaterPoller: no login confirmation from {label}{suffix}"
|
||||
)
|
||||
await self._reset_path(mc, info.pubkey, label)
|
||||
return
|
||||
return None, "login_failed_or_timeout"
|
||||
|
||||
debug_print(f"RepeaterPoller: status request → {label}")
|
||||
debug_print(f"RepeaterPoller: status request → {label}{suffix}")
|
||||
status = await mc.commands.req_status_sync(
|
||||
info.pubkey,
|
||||
min_timeout=REPEATER_STATUS_TIMEOUT,
|
||||
)
|
||||
|
||||
if not status:
|
||||
self._archive.add_measurement(
|
||||
info.pubkey, info.name, error="no_status_response"
|
||||
debug_print(
|
||||
f"RepeaterPoller: no status response from {label}{suffix}"
|
||||
)
|
||||
return
|
||||
return None, "no_status_response"
|
||||
|
||||
self._archive.add_measurement(
|
||||
info.pubkey, info.name, status=_clean_status(status)
|
||||
)
|
||||
return _clean_status(status), ""
|
||||
|
||||
except Exception as exc: # noqa: BLE001 — one repeater must not stop the loop
|
||||
debug_print(f"RepeaterPoller: {label} failed: {exc}")
|
||||
self._archive.add_measurement(
|
||||
info.pubkey, info.name, error=f"exception: {type(exc).__name__}"
|
||||
)
|
||||
debug_print(f"RepeaterPoller: {label} failed{suffix}: {exc}")
|
||||
return None, f"exception: {type(exc).__name__}"
|
||||
|
||||
finally:
|
||||
if login_attempted:
|
||||
|
||||
@@ -19,6 +19,7 @@ Record schema
|
||||
"name": "Repeater display name",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"attempts": 1, # sessions used, since 1.24.2
|
||||
"status": { ... raw fields as reported ... }
|
||||
}
|
||||
|
||||
@@ -27,6 +28,13 @@ an empty ``status``. That is deliberate: a gap in the message archive
|
||||
can then be matched against the poll moments to see whether it coincided
|
||||
with a session.
|
||||
|
||||
``attempts`` counts how many complete sessions the poller needed before
|
||||
it gave up or succeeded, so a repeater that only ever answers on the
|
||||
third try is distinguishable from one that answers immediately. It is
|
||||
zero when no session was started at all, such as when no password is
|
||||
configured. Records written before 1.24.2 do not carry the field;
|
||||
consumers should treat a missing value as unknown rather than as one.
|
||||
|
||||
Values in ``status`` are stored exactly as the repeater reports them —
|
||||
no scaling, no rounding, no smoothing. Interpretation happens elsewhere.
|
||||
|
||||
@@ -103,14 +111,17 @@ class RepeaterStatsArchive:
|
||||
name: str,
|
||||
status: Optional[Dict[str, Any]] = None,
|
||||
error: Optional[str] = None,
|
||||
attempts: int = 1,
|
||||
) -> Dict[str, Any]:
|
||||
"""Append one poll result to the archive.
|
||||
|
||||
Args:
|
||||
pubkey: Full public key of the polled repeater.
|
||||
name: Display name of the repeater.
|
||||
status: Raw status fields as reported, or ``None`` on failure.
|
||||
error: Short failure reason, or ``None`` on success.
|
||||
pubkey: Full public key of the polled repeater.
|
||||
name: Display name of the repeater.
|
||||
status: Raw status fields as reported, or ``None`` on failure.
|
||||
error: Short failure reason, or ``None`` on success.
|
||||
attempts: Number of complete sessions the poller used for this
|
||||
result. Zero when no session was started at all.
|
||||
|
||||
Returns:
|
||||
The record that was written.
|
||||
@@ -121,6 +132,7 @@ class RepeaterStatsArchive:
|
||||
"name": name,
|
||||
"ok": error is None and status is not None,
|
||||
"error": error,
|
||||
"attempts": attempts,
|
||||
"status": status or {},
|
||||
}
|
||||
|
||||
@@ -144,7 +156,7 @@ class RepeaterStatsArchive:
|
||||
|
||||
debug_print(
|
||||
f"RepeaterStatsArchive: recorded {name or pubkey[:16]} — "
|
||||
f"ok={record['ok']}"
|
||||
f"ok={record['ok']}, attempts={attempts}"
|
||||
+ (f", error={error}" if error else "")
|
||||
)
|
||||
return record
|
||||
|
||||
Reference in new issue
Block a user