mc-webui/docs/watchdog.md

Container Watchdog

The Container Watchdog is a systemd service that monitors the mc-webui Docker container and automatically restarts it if it becomes unhealthy or if the LoRa device becomes unresponsive. This is useful for ensuring reliability, especially on resource-constrained systems or when the LoRa hardware hangs.

Features

• Health monitoring - Checks container status every 30 seconds
• Log monitoring - Two pattern classes (see Failure detection)
• Automatic restart - Restarts the container when issues are detected
• Auto-start stopped container - Starts the container if it has stopped (configurable)
• Hardware USB reset - Performs a low-level USB bus reset (unbind/bind or DTR/RTS) if the LoRa device freezes. Note: USB reset is automatically skipped if a TCP connection is used.
• Diagnostic logging - Captures container logs before restart for troubleshooting
• HTTP status endpoint - Query watchdog status via HTTP API
• Restart history - Tracks all automatic restarts with timestamps

Failure detection

check_device_unresponsive() scans the last 2 minutes of container logs against two pattern classes:

• Hard patterns — any single occurrence triggers a restart. These are the long-standing "device clearly dead" messages: No response from meshcore node, disconnecting, Device connected but self_info is empty, Failed to connect after 10 attempts.
• Soft patterns — any of these failing 5 or more times in the last 2 minutes triggers a restart. Catches the "sluggish but not dead" mode where the firmware briefly stalls on get_stats_* / get_battery commands (empty-string concurrent.futures.TimeoutError) while passive RX still works: get_stats_core failed:, get_stats_radio failed:, get_stats_packets failed:, Failed to get battery:, Failed to get channel.

In parallel, the app exposes /health/strict — a stricter device-health check that the watchdog (or any external monitor) can consume to react before the soft-pattern threshold is reached.

Deploy note: the watchdog runs as a host-level systemd service and is not restarted by mcupdate. After deploying changes to scripts/watchdog/, run:

sudo systemctl restart mc-webui-watchdog.service

Installation

From Source installation (a git checkout):

cd ~/mc-webui
sudo ./scripts/watchdog/install.sh

Docker Hub installation (no repository) — run this from the folder holding your docker-compose.yml; it downloads what it needs into /opt/mc-webui-watchdog:

cd ~/mc-webui
curl -fsSL https://raw.githubusercontent.com/MarekWo/mc-webui/main/scripts/watchdog/install.sh | sudo bash

Add MCWEBUI_DIR=/path/to/mc-webui in front of sudo if your instance lives somewhere the installer cannot guess.

Either way the installer will:

• Create a systemd service mc-webui-watchdog
• Start monitoring the container immediately
• Enable automatic startup on boot
• Create a log file at /var/log/mc-webui-watchdog.log

Usage

Check service status

systemctl status mc-webui-watchdog

View watchdog logs

# Real-time logs
tail -f /var/log/mc-webui-watchdog.log

# Or via journalctl
journalctl -u mc-webui-watchdog -f

HTTP Status Endpoints

The watchdog provides HTTP endpoints on port 5051:

# Service health
curl http://localhost:5051/health

# Container status
curl http://localhost:5051/status

# Restart history
curl http://localhost:5051/history

Diagnostic Files

When the container is restarted, diagnostic information is saved to:

/tmp/mc-webui-watchdog-mc-webui-{timestamp}.log

These files contain:

• Container status at the time of failure
• Recent container logs (last 200 lines)
• Timestamp and restart result

Configuration (Optional)

No configuration required - the installer automatically detects paths and sets sensible defaults.

If you need to customize the behavior, the service supports these environment variables:

Variable	Default	Description
MCWEBUI_DIR	(auto-detected)	Path to mc-webui directory
CHECK_INTERVAL	30	Seconds between health checks
LOG_FILE	/var/log/mc-webui-watchdog.log	Path to log file
HTTP_PORT	5051	HTTP status port (0 to disable)
AUTO_START	true	Start stopped container (set to false to disable)
USB_DEVICE_PATH	(auto-detected)	Path to the LoRa device for hardware USB bus reset

To modify defaults, create an override file:

sudo systemctl edit mc-webui-watchdog

Then add your overrides, for example:

[Service]
Environment=CHECK_INTERVAL=60
Environment=AUTO_START=false

Uninstall

# From Source installation
sudo ~/mc-webui/scripts/watchdog/install.sh --uninstall

# Docker Hub installation
curl -fsSL https://raw.githubusercontent.com/MarekWo/mc-webui/main/scripts/watchdog/install.sh | sudo bash -s -- --uninstall

Note: The log file is preserved after uninstall. Remove manually if needed:

sudo rm /var/log/mc-webui-watchdog.log