mirror of
https://github.com/MarekWo/mc-webui.git
synced 2026-07-06 09:51:14 +02:00
feat(watchdog): catch sluggish-device failures via soft-pattern counting
The container watchdog only restarted on three legacy "device clearly dead" log lines, so today's failure mode (firmware briefly stalls and get_stats_* / get_battery commands time out with an empty error while passive RX keeps working) never tripped it — leaving the user with 10-15 s freezes several times a day and no automatic recovery. DeviceManager now tracks two liveness signals: - _last_rx_at, bumped on every RX_LOG_DATA event - _consecutive_stats_failures, incremented on get_stats_* / get_bat exceptions and cleared on success New /health/strict endpoint exposes these to the watchdog. It returns 503 when the device is connected but has 5+ consecutive stats failures, or when no RX event has been seen for over 5 minutes on a serial transport. The cheap /health endpoint keeps its lenient behavior so Docker's healthcheck doesn't suddenly start tripping. The watchdog's check_device_unresponsive() gains a "soft" pattern class with a count threshold of 5 in the last 2 minutes — matching against get_stats_core/radio/packets failed:, Failed to get battery:, and Failed to get channel. Hard patterns still trigger on a single hit. Deploy note: the watchdog runs as a host-level systemd service and is NOT restarted by mcupdate, so after deploy run: sudo systemctl restart mc-webui-watchdog.service Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -469,23 +469,46 @@ def handle_unhealthy_container(container_name: str, status: dict):
|
||||
|
||||
|
||||
def check_device_unresponsive(container_name: str) -> bool:
|
||||
"""Check if the container logs indicate the USB device is unresponsive."""
|
||||
"""Check if the container logs indicate the USB device is unresponsive.
|
||||
|
||||
Two classes of patterns:
|
||||
- HARD: any single occurrence triggers a restart. These are the
|
||||
long-standing "device clearly dead" messages.
|
||||
- SOFT: any of these failing >=5 times in the last 2 minutes triggers
|
||||
a restart. Catches the "sluggish but not dead" mode (firmware
|
||||
stalls on get_stats / get_bat commands while still answering
|
||||
passive RX events).
|
||||
"""
|
||||
success, stdout, stderr = run_compose_command([
|
||||
'logs', '--since', '1m', container_name
|
||||
'logs', '--since', '2m', container_name
|
||||
])
|
||||
if not success:
|
||||
return False
|
||||
|
||||
error_patterns = [
|
||||
|
||||
hard_patterns = [
|
||||
"No response from meshcore node, disconnecting",
|
||||
"Device connected but self_info is empty",
|
||||
"Failed to connect after 10 attempts"
|
||||
"Failed to connect after 10 attempts",
|
||||
]
|
||||
|
||||
for pattern in error_patterns:
|
||||
|
||||
soft_patterns = [
|
||||
"get_stats_core failed:",
|
||||
"get_stats_radio failed:",
|
||||
"get_stats_packets failed:",
|
||||
"Failed to get battery:",
|
||||
"Failed to get channel",
|
||||
]
|
||||
SOFT_THRESHOLD = 5
|
||||
|
||||
for pattern in hard_patterns:
|
||||
if pattern in stdout:
|
||||
return True
|
||||
|
||||
|
||||
for pattern in soft_patterns:
|
||||
if stdout.count(pattern) >= SOFT_THRESHOLD:
|
||||
log(f"Soft-failure threshold tripped: '{pattern}' x{stdout.count(pattern)} in 2m", "WARN")
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user