feat(watchdog): catch sluggish-device failures via soft-pattern counting

The container watchdog only restarted on three legacy "device clearly dead"
log lines, so today's failure mode (firmware briefly stalls and get_stats_*
/ get_battery commands time out with an empty error while passive RX
keeps working) never tripped it — leaving the user with 10-15 s freezes
several times a day and no automatic recovery.

DeviceManager now tracks two liveness signals:
- _last_rx_at, bumped on every RX_LOG_DATA event
- _consecutive_stats_failures, incremented on get_stats_* / get_bat
  exceptions and cleared on success

New /health/strict endpoint exposes these to the watchdog. It returns 503
when the device is connected but has 5+ consecutive stats failures, or
when no RX event has been seen for over 5 minutes on a serial transport.
The cheap /health endpoint keeps its lenient behavior so Docker's
healthcheck doesn't suddenly start tripping.

The watchdog's check_device_unresponsive() gains a "soft" pattern class
with a count threshold of 5 in the last 2 minutes — matching against
get_stats_core/radio/packets failed:, Failed to get battery:, and
Failed to get channel. Hard patterns still trigger on a single hit.

Deploy note: the watchdog runs as a host-level systemd service and is
NOT restarted by mcupdate, so after deploy run:
  sudo systemctl restart mc-webui-watchdog.service

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
MarekWo
2026-06-07 09:43:43 +02:00
parent 63204fe08d
commit 422e7a3b34
3 changed files with 108 additions and 9 deletions
+31 -8
View File
@@ -469,23 +469,46 @@ def handle_unhealthy_container(container_name: str, status: dict):
def check_device_unresponsive(container_name: str) -> bool:
"""Check if the container logs indicate the USB device is unresponsive."""
"""Check if the container logs indicate the USB device is unresponsive.
Two classes of patterns:
- HARD: any single occurrence triggers a restart. These are the
long-standing "device clearly dead" messages.
- SOFT: any of these failing >=5 times in the last 2 minutes triggers
a restart. Catches the "sluggish but not dead" mode (firmware
stalls on get_stats / get_bat commands while still answering
passive RX events).
"""
success, stdout, stderr = run_compose_command([
'logs', '--since', '1m', container_name
'logs', '--since', '2m', container_name
])
if not success:
return False
error_patterns = [
hard_patterns = [
"No response from meshcore node, disconnecting",
"Device connected but self_info is empty",
"Failed to connect after 10 attempts"
"Failed to connect after 10 attempts",
]
for pattern in error_patterns:
soft_patterns = [
"get_stats_core failed:",
"get_stats_radio failed:",
"get_stats_packets failed:",
"Failed to get battery:",
"Failed to get channel",
]
SOFT_THRESHOLD = 5
for pattern in hard_patterns:
if pattern in stdout:
return True
for pattern in soft_patterns:
if stdout.count(pattern) >= SOFT_THRESHOLD:
log(f"Soft-failure threshold tripped: '{pattern}' x{stdout.count(pattern)} in 2m", "WARN")
return True
return False