perf: let the realtime socket upgrade to a WebSocket

Three tabs open made every endpoint answer in ~20s, which two sessions read as
contention on a shared server-side lock. It wasn't the server. With 3 tabs open,
/health - which touches neither the database nor the device - measured a 14.7s
median from inside a tab and 11ms from a client outside the browser at the same
instant. The server was idle throughout.

Socket.IO clients pinned transports: ['polling'], upgrade: false, so each tab
held one HTTP connection open for its whole lifetime. Browsers allow six
concurrent HTTP/1.1 connections per origin, shared across every tab, so three
tabs consumed the pool and everything else queued in the browser waiting for a
free connection. /proc/net/tcp in the container confirmed it: pinned at exactly
6 established connections, unmoving.

The pin dates from 1d47c9c, when werkzeug had no WebSocket support and every
upgrade attempt returned HTTP 500. python-engineio==4.8.1 (pinned five weeks
later, in d3590f9) pulls in simple-websocket, which fixed that; the workaround
had outlived its premise. Drop it and use the default transports, which open on
polling and upgrade. A WebSocket is not part of the HTTP pool, so the pool is
released. Where the upgrade is blocked - a proxy that drops the Upgrade header -
the client stays on polling by itself, which is exactly today's behaviour.

Measured with 3 tabs, in-page medians: /health 14664ms -> 12ms, /api/status
19282ms -> 64ms, both now matching what the same probes read from outside the
browser. All three tabs report transport "websocket", server pushes still
arrive over it, and the log no longer fills with "Session is disconnected"
(0 occurrences, 0 tracebacks across the run).

Also corrects the earlier diagnosis in the docs: the per-endpoint timings that
looked like a lock were measured request->requestfinished in the browser, which
includes connection-queue time, so every endpoint flattened to the same figure
regardless of its own cost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
MarekWo
2026-07-31 09:31:40 +02:00
parent ab64ef72f2
commit fd054a95bd
7 changed files with 21 additions and 11 deletions
+6 -2
View File
@@ -409,7 +409,11 @@ These are top-level routes (not under `/api/`), consumed by Docker's healthcheck
## WebSocket API
All Socket.IO clients (`/chat`, `/console`, `/logs`) are configured with `transports: ['polling']`. The Werkzeug dev server can't upgrade WebSockets, so every `io()` upgrade attempt previously returned HTTP 500 and clients fell into a polling/upgrade reconnect loop — visible as 1015 s freezes on app load. Long-polling keeps real-time pushes working with ~12 s latency.
All Socket.IO clients (`/chat`, `/console`, `/logs`) use the default transports: connect over long-polling, then upgrade to a real WebSocket. Where the upgrade is blocked (a reverse proxy that drops the `Upgrade` header) the client stays on polling by itself, so no configuration is needed either way.
From 2026-06-07 to 2026-07-31 the clients pinned `transports: ['polling'], upgrade: false`, because the Werkzeug server then had no WebSocket support and every `io()` upgrade attempt returned HTTP 500, producing a reconnect loop and 1015 s freezes on app load. That stopped being true when `python-engineio==4.8.1` was pinned (2026-07-14) and pulled in `simple-websocket`, which teaches Werkzeug to serve WebSockets.
**Do not re-pin polling.** Long-polling holds one HTTP connection open per tab for the life of the tab, and browsers allow only six concurrent HTTP/1.1 connections per origin *across all tabs*. Three open tabs therefore consumed the whole pool, and every other request — including ones the server answered in 10 ms — waited tens of seconds in the browser's queue for a free connection. Measured with three tabs open: `/health` took a **14.7 s median** from inside a tab while answering in **11 ms** to a client outside the browser at the same instant; after the upgrade the same probe reads 12 ms. A WebSocket is not part of that HTTP pool, so upgrading is what releases it.
### Console Namespace (`/console`)
@@ -444,7 +448,7 @@ Real-time log streaming via Socket.IO.
**Server → Client:**
- `log_line` - New log line
The `MemoryLogHandler` filters werkzeug access-log records for `/socket.io/` and `/api/logs/` paths before buffering/broadcasting. With `async_mode='threading'` Socket.IO falls back to long-polling; without this filter every poll is logged, the broadcast wakes the pending poll, the client re-polls immediately, and an open System Log tab spins at 10+ requests/sec.
The `MemoryLogHandler` filters werkzeug access-log records for `/socket.io/` and `/api/logs/` paths before buffering/broadcasting. Clients still open on long-polling before upgrading, and stay there wherever the upgrade is blocked; without this filter every poll is logged, the broadcast wakes the pending poll, the client re-polls immediately, and an open System Log tab spins at 10+ requests/sec.
---