MQTT Protocol Error & Timeout on Windows (NotImplementedError) #116

New Issue

2026-10-05T01:57:48+02:00

adam commented

2026-10-05 01:57:48 +02:00

I run an up-to-date mosquito broker and have a user on windows using the Community MQTT/meshcoretomqtt integration to connect to it using TCP/User/Pass/TLS but being unsuccessful at making a connection.

Connectivity is NOT blocked: Mosquitto logs show a successful TCP connection from the client IP.
Protocol Failure: Immediately after connecting, the broker rejects the client with a Protocol error.
Internal Crash: The application logs reveal a NotImplementedError in the asyncio writer, followed by a timeout.

Environment

OS: Windows 10
Python Version: 3.12.10

1. Broker Logs (Mosquitto)

These logs confirm the connection reaches the broker, ruling out firewall or network issues:

1775361189: New connection from [CLIENT-IP]:17235 on port 8883.
1775361249: Client <unknown> disconnected: Protocol error.
1775361259: New connection from [CLIENT-IP]:48399 on port 8883.
1775361319: Client <unknown> disconnected: Protocol error.

2. Application Logs (`debug.json`)

The application crashes internally when trying to write to the socket because the default Windows event loop is missing required methods:

File "...\aiomqtt\client.py", line 785, in __aenter__
    await self._wait_for(self._connected, timeout=None)
...
  File "C:\Python_3.12\Lib\asyncio\proactor_events.py", line 377, in add_writer
    raise NotImplementedError
NotImplementedError

2026-04-05 10:16:08 - app.fanout.mqtt_base - WARNING - connection error... Operation timed out
aiomqtt.exceptions.MqttError: Operation timed out

3. Analysis

Possible cause is the ProactorEventLoop (the default on Windows since Python 3.8).

aiomqtt and paho-mqtt require loop.add_reader and loop.add_writer.
ProactorEventLoop does not implement these methods, causing the NotImplementedError.
Because the writer fails, the client either sends a malformed CONNECT packet or nothing at all. The broker receives invalid data and closes the connection with a Protocol error.

1. No event loop policy is set anywhere

A search across all .py files for sys.platform, win32, WindowsSelectorEventLoop, ProactorEvent, and set_event_loop_policy returns zero results. There is no Windows-specific event loop handling anywhere in this codebase.

2. The exact failure path in `mqtt_base.py`

In app/fanout/mqtt_base.py line 215, the connection loop does:

async with aiomqtt.Client(**client_kwargs) as client:

aiomqtt.Client.__aenter__ (as seen in the traceback: client.py line 785) calls await self._wait_for(self._connected, ...), but before that resolves, paho-mqtt internally tries to call loop.add_writer() on the socket to register I/O callbacks. On Windows with Python 3.12, the running loop is ProactorEventLoop, which raises NotImplementedError for that method. The exception propagates up and gets caught by the except Exception as e: block at line 250, logging the warning and starting the backoff timer — which explains the "Operation timed out" message.

3. Why the broker logs "Protocol error"

paho-mqtt does establish the TCP connection (hence the broker log "New connection from..."), but since add_writer fails, the MQTT CONNECT packet is never written to the socket. The broker receives nothing valid and closes with Protocol error.

4. Why it only affects MQTT

The rest of the app (uvicorn itself, aiosqlite, radio serial/TCP) works fine on Windows because those libraries either use ProactorEventLoop-compatible APIs or use native async I/O (asyncio.open_connection, asyncio.StreamReader/Writer) that work with both loop types. Only paho-mqtt's internal socket integration is specifically incompatible.

4. The Fix

A single block needs to be added to the very top of app/main.py, before any other code runs:

import sys

# paho-mqtt (used by aiomqtt) requires loop.add_reader() / loop.add_writer(),
# which ProactorEventLoop (the Windows default since Python 3.8) does not implement.
# Force SelectorEventLoop on Windows so MQTT connections work correctly.
if sys.platform == "win32":
    import asyncio
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Why this placement works: When running uvicorn app.main:app, uvicorn imports the app.main module first to get the ASGI app object, then creates the event loop to run it. Module-level code in app/main.py executes during that import step — before uvicorn creates the loop — so the policy is set in time.

Trade-off to be aware of: SelectorEventLoop on Windows has somewhat lower raw I/O throughput than ProactorEventLoop for certain workloads (e.g. subprocess pipes), but for this app's workload (serial/TCP radio + WebSocket + SQLite + MQTT) it makes no practical difference, and this is the officially documented approach for paho-mqtt compatibility.

I run an up-to-date mosquito broker and have a user on windows using the Community MQTT/meshcoretomqtt integration to connect to it using TCP/User/Pass/TLS but being unsuccessful at making a connection. 1. **Connectivity is NOT blocked:** Mosquitto logs show a successful TCP connection from the client IP. 2. **Protocol Failure:** Immediately after connecting, the broker rejects the client with a `Protocol error`. 3. **Internal Crash:** The application logs reveal a `NotImplementedError` in the `asyncio` writer, followed by a timeout. ### Environment - **OS:** Windows 10 - **Python Version:** 3.12.10 ### 1. Broker Logs (Mosquitto) These logs confirm the connection reaches the broker, ruling out firewall or network issues: ```text 1775361189: New connection from [CLIENT-IP]:17235 on port 8883. 1775361249: Client <unknown> disconnected: Protocol error. 1775361259: New connection from [CLIENT-IP]:48399 on port 8883. 1775361319: Client <unknown> disconnected: Protocol error. ``` ### 2. Application Logs (`debug.json`) The application crashes internally when trying to write to the socket because the default Windows event loop is missing required methods: ```text File "...\aiomqtt\client.py", line 785, in __aenter__ await self._wait_for(self._connected, timeout=None) ... File "C:\Python_3.12\Lib\asyncio\proactor_events.py", line 377, in add_writer raise NotImplementedError NotImplementedError 2026-04-05 10:16:08 - app.fanout.mqtt_base - WARNING - connection error... Operation timed out aiomqtt.exceptions.MqttError: Operation timed out ``` ### 3. Analysis Possible cause is the **ProactorEventLoop** (the default on Windows since Python 3.8). * `aiomqtt` and `paho-mqtt` require `loop.add_reader` and `loop.add_writer`. * `ProactorEventLoop` does **not** implement these methods, causing the `NotImplementedError`. * Because the writer fails, the client either sends a malformed `CONNECT` packet or nothing at all. The broker receives invalid data and closes the connection with a `Protocol error`. #### 1. No event loop policy is set anywhere A search across all `.py` files for `sys.platform`, `win32`, `WindowsSelectorEventLoop`, `ProactorEvent`, and `set_event_loop_policy` returns **zero results**. There is no Windows-specific event loop handling anywhere in this codebase. #### 2. The exact failure path in `mqtt_base.py` In [`app/fanout/mqtt_base.py`](app/fanout/mqtt_base.py) line 215, the connection loop does: ```python async with aiomqtt.Client(**client_kwargs) as client: ``` `aiomqtt.Client.__aenter__` (as seen in the traceback: `client.py line 785`) calls `await self._wait_for(self._connected, ...)`, but before that resolves, paho-mqtt internally tries to call `loop.add_writer()` on the socket to register I/O callbacks. On Windows with Python 3.12, the running loop is `ProactorEventLoop`, which **raises `NotImplementedError`** for that method. The exception propagates up and gets caught by the `except Exception as e:` block at line 250, logging the warning and starting the backoff timer — which explains the "Operation timed out" message. #### 3. Why the broker logs "Protocol error" paho-mqtt does establish the TCP connection (hence the broker log "New connection from..."), but since `add_writer` fails, the MQTT `CONNECT` packet is **never written to the socket**. The broker receives nothing valid and closes with `Protocol error`. #### 4. Why it only affects MQTT The rest of the app (uvicorn itself, aiosqlite, radio serial/TCP) works fine on Windows because those libraries either use `ProactorEventLoop`-compatible APIs or use native async I/O (`asyncio.open_connection`, `asyncio.StreamReader/Writer`) that work with both loop types. Only paho-mqtt's internal socket integration is specifically incompatible. ### 4. The Fix A single block needs to be added to the very top of **`app/main.py`**, before any other code runs: ```python import sys # paho-mqtt (used by aiomqtt) requires loop.add_reader() / loop.add_writer(), # which ProactorEventLoop (the Windows default since Python 3.8) does not implement. # Force SelectorEventLoop on Windows so MQTT connections work correctly. if sys.platform == "win32": import asyncio asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) ``` **Why this placement works**: When running `uvicorn app.main:app`, uvicorn imports the `app.main` module first to get the ASGI app object, *then* creates the event loop to run it. Module-level code in `app/main.py` executes during that import step — before uvicorn creates the loop — so the policy is set in time. **Trade-off to be aware of**: `SelectorEventLoop` on Windows has somewhat lower raw I/O throughput than `ProactorEventLoop` for certain workloads (e.g. subprocess pipes), but for this app's workload (serial/TCP radio + WebSocket + SQLite + MQTT) it makes no practical difference, and this is the [officially documented approach](https://docs.python.org/3/library/asyncio-platforms.html#windows) for paho-mqtt compatibility.

adam closed this issue

2026-10-05 01:57:48 +02:00

adam commented

2026-10-05 01:57:49 +02:00

I hope to be able to test the code change this afternoon or evening.

I hope to be able to test the code change this afternoon or evening.

adam commented

2026-10-05 01:57:49 +02:00

I think your fix is pretty spot on; would you prefer to PR or have me merge as suggested?

I think your fix is pretty spot on; would you prefer to PR or have me merge as suggested?

adam commented

2026-10-05 01:57:50 +02:00

The person having the issue tested the above code change, but that didn't work. I have another change to try, but he hasn't yet tested it. Below is what needs testing.

Create run.py in the project root:

"""
Use this instead of `uvicorn app.main:app` when running on Windows, or any time
MQTT fanout (Community MQTT) is configured. On Windows, uvicorn's default event
loop (ProactorEventLoop) does not support the add_reader()/add_writer() calls
that aiomqtt/paho-mqtt require. This script forces SelectorEventLoop via policy
before uvicorn creates the loop, then runs uvicorn with loop="none" so it
honours that policy instead of overriding it.

Usage:
    uv run python run.py
    uv run python run.py --host 0.0.0.0 --port 8000
"""
import sys

# Must happen before uvicorn imports or creates any event loop.
# uvicorn's "auto" loop factory on Windows explicitly returns ProactorEventLoop,
# bypassing set_event_loop_policy. Using loop="none" below makes asyncio.run()
# call asyncio.new_event_loop() via the policy instead.
if sys.platform == "win32":
    import asyncio
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

import uvicorn

if __name__ == "__main__":
    uvicorn.run(
        "app.main:app",
        loop="none",  # "none" → loop_factory=None → asyncio.run uses policy above
    )

The person having the issue tested the above code change, but that didn't work. I have another change to try, but he hasn't yet tested it. Below is what needs testing. Create run.py in the project root: ```python """ Use this instead of `uvicorn app.main:app` when running on Windows, or any time MQTT fanout (Community MQTT) is configured. On Windows, uvicorn's default event loop (ProactorEventLoop) does not support the add_reader()/add_writer() calls that aiomqtt/paho-mqtt require. This script forces SelectorEventLoop via policy before uvicorn creates the loop, then runs uvicorn with loop="none" so it honours that policy instead of overriding it. Usage: uv run python run.py uv run python run.py --host 0.0.0.0 --port 8000 """ import sys # Must happen before uvicorn imports or creates any event loop. # uvicorn's "auto" loop factory on Windows explicitly returns ProactorEventLoop, # bypassing set_event_loop_policy. Using loop="none" below makes asyncio.run() # call asyncio.new_event_loop() via the policy instead. if sys.platform == "win32": import asyncio asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) import uvicorn if __name__ == "__main__": uvicorn.run( "app.main:app", loop="none", # "none" → loop_factory=None → asyncio.run uses policy above ) ```

adam commented

2026-10-05 01:57:51 +02:00

Forgot to mention I hope to have the new code tested today or tomorrow.

Forgot to mention I hope to have the new code tested today or tomorrow.

adam commented

2026-10-05 01:57:51 +02:00

Awesome; no rush!

Well, not really awesome -- I think your proposed fix is well-scoped/the only scope it can be; I just hate the problem. I think probably if we're running on windows we'll probably bubble up an error toast on MQTT configure/enable of any kind and a very loud log warning stating that it won't work until they do an alternate launch invocation (--loop none) and doc that in the advanced readme. Want to balance not disrupting users' established launch flows with what I would assume to be a minority of windows/non-docker users.

Awesome; no rush! Well, not really awesome -- I think your proposed fix is well-scoped/the only scope it can be; I just hate the problem. I think probably if we're running on windows we'll probably bubble up an error toast on MQTT configure/enable of any kind and a very loud log warning stating that it won't work until they do an alternate launch invocation (`--loop none`) and doc that in the advanced readme. Want to balance not disrupting users' established launch flows with what I would assume to be a minority of windows/non-docker users.

adam commented

2026-10-05 01:57:52 +02:00

I can confirm that starting with run.py did work and the windows client could connect to the MQTT broker. I'll let you merge, if that's alright, since you probably have an idea of how you want to document this.

I can confirm that starting with run.py did work and the windows client could connect to the MQTT broker. I'll let you merge, if that's alright, since you probably have an idea of how you want to document this.

adam commented

2026-10-05 01:57:53 +02:00

Okay I'm getting a lil funky with this but I think I've got an in place restart working.

Happy path:

Sad path (worst case scenario, server boots but MQTT errors like it has been):

(I have fixed the text alignment since this screenshot)

I've got some more work to do on it because frankly it's a really gross and brittle piece of code and I want to whack on it a bit more before it goes in the critical path. I'm winding down a bit tonight but might get it merged; if not, probably tomorrow or thursday.

Okay I'm getting a lil funky with this but I think I've got an in place restart working. Happy path: <img width="1718" height="424" alt="Image" src="https://github.com/user-attachments/assets/17a4ce77-e065-47db-9830-3dd7f96004b8" /> Sad path (worst case scenario, server boots but MQTT errors like it has been): <img width="1568" height="808" alt="Image" src="https://github.com/user-attachments/assets/d6d7970b-28eb-4671-814c-5d5b101f374a" /> (I have fixed the text alignment since this screenshot) I've got some more work to do on it because frankly it's a really gross and brittle piece of code and I want to whack on it a bit more before it goes in the critical path. I'm winding down a bit tonight but might get it merged; if not, probably tomorrow or thursday.

adam commented

2026-10-05 01:57:53 +02:00

Okay I shied away from trying to be clever about it. I've added a warning on startup and guidance in the readme for Windows users. It's not perfect, but if someone misses both, it'll pop up a warning in the UI.

Okay I shied away from trying to be clever about it. I've added a warning on startup and guidance in the readme for Windows users. It's not perfect, but if someone misses both, it'll pop up a warning in the UI.

adam commented

2026-10-05 01:57:54 +02:00

Very reasonable for the likely small userbase on windows

Very reasonable for the likely small userbase on windows

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/Remote-Terminal-for-MeshCore#116

MQTT Protocol Error & Timeout on Windows (NotImplementedError) #116

Environment

1. Broker Logs (Mosquitto)

2. Application Logs (debug.json)

3. Analysis

1. No event loop policy is set anywhere

2. The exact failure path in mqtt_base.py

3. Why the broker logs "Protocol error"

4. Why it only affects MQTT

4. The Fix

2. Application Logs (`debug.json`)

2. The exact failure path in `mqtt_base.py`