From fd054a95bd6f2ea5c5f6559591f7bd00a9abcb69 Mon Sep 17 00:00:00 2001 From: MarekWo Date: Fri, 31 Jul 2026 09:31:40 +0200 Subject: [PATCH] perf: let the realtime socket upgrade to a WebSocket Three tabs open made every endpoint answer in ~20s, which two sessions read as contention on a shared server-side lock. It wasn't the server. With 3 tabs open, /health - which touches neither the database nor the device - measured a 14.7s median from inside a tab and 11ms from a client outside the browser at the same instant. The server was idle throughout. Socket.IO clients pinned transports: ['polling'], upgrade: false, so each tab held one HTTP connection open for its whole lifetime. Browsers allow six concurrent HTTP/1.1 connections per origin, shared across every tab, so three tabs consumed the pool and everything else queued in the browser waiting for a free connection. /proc/net/tcp in the container confirmed it: pinned at exactly 6 established connections, unmoving. The pin dates from 1d47c9c, when werkzeug had no WebSocket support and every upgrade attempt returned HTTP 500. python-engineio==4.8.1 (pinned five weeks later, in d3590f9) pulls in simple-websocket, which fixed that; the workaround had outlived its premise. Drop it and use the default transports, which open on polling and upgrade. A WebSocket is not part of the HTTP pool, so the pool is released. Where the upgrade is blocked - a proxy that drops the Upgrade header - the client stays on polling by itself, which is exactly today's behaviour. Measured with 3 tabs, in-page medians: /health 14664ms -> 12ms, /api/status 19282ms -> 64ms, both now matching what the same probes read from outside the browser. All three tabs report transport "websocket", server pushes still arrive over it, and the log no longer fills with "Session is disconnected" (0 occurrences, 0 tracebacks across the run). Also corrects the earlier diagnosis in the docs: the per-endpoint timings that looked like a lock were measured request->requestfinished in the browser, which includes connection-queue time, so every endpoint flattened to the same figure regardless of its own cost. Co-Authored-By: Claude Opus 5 --- app/static/js/app.js | 9 +++++++-- app/static/js/console.js | 3 +-- app/static/js/dm.js | 3 +-- app/static/js/logs.js | 3 +-- docs/architecture.md | 8 ++++++-- docs/whatsnew.md | 3 ++- requirements.txt | 3 +++ 7 files changed, 21 insertions(+), 11 deletions(-) diff --git a/app/static/js/app.js b/app/static/js/app.js index 3b93f1a..12d0d8c 100644 --- a/app/static/js/app.js +++ b/app/static/js/app.js @@ -482,9 +482,14 @@ function connectChatSocket() { } const wsUrl = window.location.origin; + // Default transports (polling, then upgrade to websocket). Long-polling was + // pinned in 1d47c9c because werkzeug had no websocket support; python-engineio + // 4.8.1 pulled in simple-websocket and it does now. Polling holds an HTTP + // connection open per tab, and browsers only allow six per origin, so three + // tabs starved every other request of a connection for tens of seconds. + // Upgrading moves that connection out of the HTTP pool. Where the upgrade is + // blocked (a proxy that drops Upgrade), the client stays on polling by itself. chatSocket = io(wsUrl + '/chat', { - transports: ['polling'], - upgrade: false, reconnection: true, reconnectionDelay: 2000, reconnectionDelayMax: 10000, diff --git a/app/static/js/console.js b/app/static/js/console.js index cc2e2ee..415b7fe 100644 --- a/app/static/js/console.js +++ b/app/static/js/console.js @@ -37,9 +37,8 @@ function connectWebSocket() { console.log('Connecting to WebSocket:', wsUrl); try { + // Default transports — see the note in app.js connectChatSocket(). socket = io(wsUrl + '/console', { - transports: ['polling'], - upgrade: false, reconnection: true, reconnectionAttempts: Infinity, reconnectionDelay: 1000, diff --git a/app/static/js/dm.js b/app/static/js/dm.js index 3432f1d..b357f7a 100644 --- a/app/static/js/dm.js +++ b/app/static/js/dm.js @@ -149,9 +149,8 @@ function connectChatSocket() { } const wsUrl = window.location.origin; + // Default transports — see the note in app.js connectChatSocket(). chatSocket = io(wsUrl + '/chat', { - transports: ['polling'], - upgrade: false, reconnection: true, reconnectionDelay: 2000, reconnectionDelayMax: 10000, diff --git a/app/static/js/logs.js b/app/static/js/logs.js index ae4050c..8af5d43 100644 --- a/app/static/js/logs.js +++ b/app/static/js/logs.js @@ -32,9 +32,8 @@ const LEVEL_ORDER = { DEBUG: 0, INFO: 1, WARNING: 2, ERROR: 3, CRITICAL: 4 }; // --- WebSocket --- + // Default transports — see the note in app.js connectChatSocket(). const socket = io('/logs', { - transports: ['polling'], - upgrade: false, reconnection: true, reconnectionDelay: 2000, }); diff --git a/docs/architecture.md b/docs/architecture.md index f3ceddd..111b483 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -409,7 +409,11 @@ These are top-level routes (not under `/api/`), consumed by Docker's healthcheck ## WebSocket API -All Socket.IO clients (`/chat`, `/console`, `/logs`) are configured with `transports: ['polling']`. The Werkzeug dev server can't upgrade WebSockets, so every `io()` upgrade attempt previously returned HTTP 500 and clients fell into a polling/upgrade reconnect loop — visible as 10–15 s freezes on app load. Long-polling keeps real-time pushes working with ~1–2 s latency. +All Socket.IO clients (`/chat`, `/console`, `/logs`) use the default transports: connect over long-polling, then upgrade to a real WebSocket. Where the upgrade is blocked (a reverse proxy that drops the `Upgrade` header) the client stays on polling by itself, so no configuration is needed either way. + +From 2026-06-07 to 2026-07-31 the clients pinned `transports: ['polling'], upgrade: false`, because the Werkzeug server then had no WebSocket support and every `io()` upgrade attempt returned HTTP 500, producing a reconnect loop and 10–15 s freezes on app load. That stopped being true when `python-engineio==4.8.1` was pinned (2026-07-14) and pulled in `simple-websocket`, which teaches Werkzeug to serve WebSockets. + +**Do not re-pin polling.** Long-polling holds one HTTP connection open per tab for the life of the tab, and browsers allow only six concurrent HTTP/1.1 connections per origin *across all tabs*. Three open tabs therefore consumed the whole pool, and every other request — including ones the server answered in 10 ms — waited tens of seconds in the browser's queue for a free connection. Measured with three tabs open: `/health` took a **14.7 s median** from inside a tab while answering in **11 ms** to a client outside the browser at the same instant; after the upgrade the same probe reads 12 ms. A WebSocket is not part of that HTTP pool, so upgrading is what releases it. ### Console Namespace (`/console`) @@ -444,7 +448,7 @@ Real-time log streaming via Socket.IO. **Server → Client:** - `log_line` - New log line -The `MemoryLogHandler` filters werkzeug access-log records for `/socket.io/` and `/api/logs/` paths before buffering/broadcasting. With `async_mode='threading'` Socket.IO falls back to long-polling; without this filter every poll is logged, the broadcast wakes the pending poll, the client re-polls immediately, and an open System Log tab spins at 10+ requests/sec. +The `MemoryLogHandler` filters werkzeug access-log records for `/socket.io/` and `/api/logs/` paths before buffering/broadcasting. Clients still open on long-polling before upgrading, and stay there wherever the upgrade is blocked; without this filter every poll is logged, the broadcast wakes the pending poll, the client re-polls immediately, and an open System Log tab spins at 10+ requests/sec. --- diff --git a/docs/whatsnew.md b/docs/whatsnew.md index 0ea9e10..67ed132 100644 --- a/docs/whatsnew.md +++ b/docs/whatsnew.md @@ -12,10 +12,11 @@ For deep technical notes, see [architecture.md](architecture.md). For the full g ### Fixes +- **You can keep mc-webui open in several tabs again.** Two or three open windows used to bring the whole thing to a crawl — the message list crept, buttons took ten or twenty seconds to do anything, and the status could sit on "Connecting…". It looked exactly like an overloaded server, and it wasn't: the server was answering in a few thousandths of a second the entire time. The live connection that pushes new messages to an open page was running in a mode that keeps a browser connection permanently occupied, and a browser only allows six connections to one address **shared across every tab**. Three tabs took the lot, so everything else — loading messages, marking them read, sending — queued in the browser waiting for a free one. That connection now uses a proper WebSocket, which does not come out of that budget. Measured with three tabs open: a request that had been taking around 15 seconds now takes about 12 milliseconds. The advice to keep only one window open no longer applies. If you run mc-webui behind a reverse proxy that isn't set up to pass WebSocket connections through, the page quietly falls back to the old behaviour and works exactly as before. - **Messages no longer go missing after the app has been in the background.** Coming back to a minimised app — or to a phone that had been asleep — could show a chat that quietly stopped at whatever message arrived last before the screen went off, with everything since then missing until the app was force-stopped and reopened. New messages reach an open page over a live connection, and Android tears that connection down while the app sits in the background; nothing then went back to ask the server what had been missed. Now every way back from a gap re-reads the list: the connection coming back, the app returning to the foreground, and a heartbeat that notices when the page has been frozen. The same applies to direct messages, and to a browser tab that lost its network for a while. - **A Refresh item in the menu.** The browser's pull-to-refresh has no equivalent in the Android app, so there is now a **Refresh** entry at the top of the menu that reloads the messages from the server on demand — in the app, and everywhere else too. - **The connection-status check is about five times cheaper.** Every open page asks the server how the mesh device is doing — on load, once a minute after that, and each time you come back to a tab you had left. Answering that took roughly a quarter of a second, almost none of it spent on the device itself: the server was counting the rows of every table in the database to report two numbers, and the row it needed for the "last message" timestamp was found by reading the entire message table and sorting it. The heaviest part, counting the radio-echo records, was thrown away unused — and that table only ever grows, so the check was getting slower the longer an instance had been running. It now asks for exactly the three values it needs, in one go. This is a smaller effect than the bundling below and it does not change what you see on screen; it does take a recurring cost off the server on every open page. -- **Far less load on the server while you have messages on screen.** Whenever new radio traffic came in, the page asked the server about every message on screen separately — hundreds of individual requests at a time, repeated every few seconds, and the same messages over and over. On a busy channel that was thousands of requests a minute from a single tab, which left the server little room to answer anything else; the worst of it looked like the mesh device had dropped off, with the message list stuck on "Loading messages…" and the status on "Connecting…", while the device was connected the whole time. Those requests are now bundled into one. A page that needed hundreds of requests per update now needs a single one. Keeping mc-webui open in several windows at once is still not recommended — that can still slow things down for a different reason — but the app is considerably lighter on the server than it was. +- **Far less load on the server while you have messages on screen.** Whenever new radio traffic came in, the page asked the server about every message on screen separately — hundreds of individual requests at a time, repeated every few seconds, and the same messages over and over. On a busy channel that was thousands of requests a minute from a single tab, which left the server little room to answer anything else; the worst of it looked like the mesh device had dropped off, with the message list stuck on "Loading messages…" and the status on "Connecting…", while the device was connected the whole time. Those requests are now bundled into one. A page that needed hundreds of requests per update now needs a single one. The remaining reason several windows were slow is fixed separately — see the first entry above. --- diff --git a/requirements.txt b/requirements.txt index aaad81f..3495cef 100644 --- a/requirements.txt +++ b/requirements.txt @@ -32,6 +32,9 @@ flask-socketio==5.3.6 # Observer: MQTT packet publishing (meshcore-packet-capture compatible) paho-mqtt==2.1.0 python-socketio==5.10.0 +# Pulls in simple-websocket, which is what lets the werkzeug server serve real +# WebSockets. Clients rely on that upgrade; without it they fall back to +# long-polling and starve the browser's per-origin connection pool. python-engineio==4.8.1 # v2: Direct MeshCore device communication (replaces bridge subprocess)