feat: MQTT retry (clamped) exponential backoff? #167

New Issue

Closed

opened 2026-10-05 02:01:33 +02:00 by adam · 0 comments

adam commented 2026-10-05 02:01:33 +02:00

Owner

Copy link

Copy source

LetsMesh EU seems to be having problems (for me) today, and I see the reconnect notification constantly...

    "2026-04-28 05:33:04 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n  File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n    async with aiomqtt.Client(**client_kwargs) as client:\n               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n    raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded",
    "2026-04-28 05:34:04 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n  File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n    async with aiomqtt.Client(**client_kwargs) as client:\n               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n    raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded",
    "2026-04-28 05:35:04 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n  File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n    async with aiomqtt.Client(**client_kwargs) as client:\n               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n    raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded",
...
    "2026-04-28 16:12:29 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n  File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n    async with aiomqtt.Client(**client_kwargs) as client:\n               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n    raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded",
    "2026-04-28 16:13:29 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n  File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n    async with aiomqtt.Client(**client_kwargs) as client:\n               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n  File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n    raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded"

Retrying a few times with minute intervals sounds fine to handle spurious issues.
But when there's extended downtime, every minute gets rather tiresome.

LetsMesh EU seems to be having problems (for me) today, and I see the reconnect notification constantly... ``` "2026-04-28 05:33:04 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n async with aiomqtt.Client(**client_kwargs) as client:\n ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded", "2026-04-28 05:34:04 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n async with aiomqtt.Client(**client_kwargs) as client:\n ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded", "2026-04-28 05:35:04 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n async with aiomqtt.Client(**client_kwargs) as client:\n ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded", ... "2026-04-28 16:12:29 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n async with aiomqtt.Client(**client_kwargs) as client:\n ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded", "2026-04-28 16:13:29 - app.fanout.mqtt_base - WARNING - Community MQTT [LetsMesh (EU)] connection error. This is usually transient network noise; if it self-resolves, it is generally not a concern: WebSocket handshake error, connection not upgraded (reconnecting in 60s). If this error persists, check your network connection and MQTT broker status.\nTraceback (most recent call last):\n File \"/app/app/fanout/mqtt_base.py\", line 216, in _connection_loop\n async with aiomqtt.Client(**client_kwargs) as client:\n ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File \"/app/.venv/lib/python3.13/site-packages/aiomqtt/client.py\", line 783, in __aenter__\n raise MqttError(str(exc)) from None\naiomqtt.exceptions.MqttError: WebSocket handshake error, connection not upgraded" ``` Retrying a few times with minute intervals sounds fine to handle spurious issues. But when there's extended downtime, every minute gets rather tiresome. <!-- github-sync-id:4344665886 -->

adam closed this issue 2026-10-05 02:01:33 +02:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dependabot/npm_and_yarn/frontend/npm_and_yarn-b3b4accef5

dependabot/uv/uv-10713fdc8d

alternate-repeater-detail-fetch

make-repeater-sortable

fix/cli-response-scoped-subscription

tracked-contact-telemetry

thin-docker

tcp-proxy

repeater-error-count

radio-config-export

channel-mute

extended-contact-fetch-timeout

web-push

ha

migration-updates

runtimes-and-sessions

aur-packaging

arch-publishing

self-reexec-on-windows

migrate-favorites

bugbash-v7

testbranch

move-to-fk-pragma

gnomeadrift/repeater_telemetry_history

use-nginx-docker

settings-scroll-fix

richer-install-script

fix-room-server-ordering

bot-safety

room_server

better-path-management

dm-ingest-unification

keep-track-of-channel-state

notifications

codex-refactor

multibyte-take-2

multibyte

fanout_bus

community-mqtt

loopback

actually-cool-ui-concepts

3.17.1

3.17.0

3.16.2

3.16.1

3.16.0

3.15.2

3.15.1

3.15.0

3.14.1

3.14.0

3.13.0

3.12.3

3.12.2

3.12.1

3.12.0

3.11.3

3.11.2

3.11.1

3.11.0

3.10.0

3.9.0

3.8.0

3.7.1

3.7.0

3.6.7

3.6.3

3.6.2

3.6.1

3.6.0

3.5.0

3.4.1

3.4.0

3.3.0

3.2.0

3.1.1

3.0.0

2.7.9

2.7.8

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

adam (Adam Wojcieszonek)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/Remote-Terminal-for-MeshCore#167