adam/meshing-around
1
0 Fork 0
mirror of https://github.com/SpudGunMan/meshing-around.git synced 2026-11-03 07:22:23 +01:00
Code Issues 12 Packages Projects Releases Wiki Activity

Connection lost and will not resume #61

New Issue
Closed
opened 2026-10-05 01:18:43 +02:00 by adam · 15 comments
adam commented 2026-10-05 01:18:43 +02:00
Owner
Copy link
Copy source

Running 2.7.0 tech preview (although, has been a problem on 2.6+) on a T-Beam Supreme, connected over wifi to an AP about 20 feet away, unobstructed. The script will lose connection to the T-beam and not be able to reconnect. This is reported in the journal:

Jun 25 23:24:19 gserver python3[43118]: 2025-06-25 23:24:19,194 |    DEBUG | Telemetry:1 numPacketsRx:48086 numPacketsRxErr:0 numPacketsTx:2986 numPacketsTxErr:0 ChUtil%:20.3 AirTx%:1.3 totalNodes:230 On>
Jun 25 23:29:38 gserver python3[43118]: ERROR:root:Unexpected OSError, terminating meshtastic reader... [Errno 104] Connection reset by peer
Jun 25 23:29:38 gserver python3[43118]: 2025-06-25 23:29:38,010 | CRITICAL | System: Lost Connection to Device 192.168.2.241
Jun 25 23:29:40 gserver python3[43118]: 2025-06-25 23:29:40,856 |    ERROR | System: communicating with interface1, trying to reconnect: Timed out waiting for an acknowledgment
Jun 25 23:29:40 gserver python3[43118]: 2025-06-25 23:29:40,856 |    ERROR | System: closing interface1: [Errno 32] Broken pipe

After this happens, the script goes into a retry loop and never reconnects to the node:

Sun 25 23:29:40 gserver python3[43118]: 2025-06-25 23:29:40,856 |    DEBUG | System: Retrying interface1 in 15 seconds
Jun 25 23:29:55 gserver python3[43118]: 2025-06-25 23:29:55,858 |    DEBUG | System: Retrying Interface1
Jun 25 23:29:55 gserver python3[43118]: 2025-06-25 23:29:55,859 |    ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused
Jun 25 23:30:15 gserver python3[43118]: 2025-06-25 23:30:15,860 |    DEBUG | System: Retrying interface1 in 15 seconds
Jun 25 23:30:30 gserver python3[43118]: 2025-06-25 23:30:30,861 |    DEBUG | System: Retrying Interface1
Jun 25 23:30:30 gserver python3[43118]: 2025-06-25 23:30:30,862 |    ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused
Jun 25 23:30:50 gserver python3[43118]: 2025-06-25 23:30:50,864 |    DEBUG | System: Retrying interface1 in 15 seconds
Jun 25 23:31:05 gserver python3[43118]: 2025-06-25 23:31:05,865 |    DEBUG | System: Retrying Interface1
Jun 25 23:31:05 gserver python3[43118]: 2025-06-25 23:31:05,866 |    ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused
Jun 25 23:31:25 gserver python3[43118]: 2025-06-25 23:31:25,867 |    DEBUG | System: Retrying interface1 in 15 seconds

Simply restarting the script gets it going again. Restarting the node while in this loop does not. It just keeps retrying.

Running 2.7.0 tech preview (although, has been a problem on 2.6+) on a T-Beam Supreme, connected over wifi to an AP about 20 feet away, unobstructed. The script will lose connection to the T-beam and not be able to reconnect. This is reported in the journal: ``` Jun 25 23:24:19 gserver python3[43118]: 2025-06-25 23:24:19,194 | DEBUG | Telemetry:1 numPacketsRx:48086 numPacketsRxErr:0 numPacketsTx:2986 numPacketsTxErr:0 ChUtil%:20.3 AirTx%:1.3 totalNodes:230 On> Jun 25 23:29:38 gserver python3[43118]: ERROR:root:Unexpected OSError, terminating meshtastic reader... [Errno 104] Connection reset by peer Jun 25 23:29:38 gserver python3[43118]: 2025-06-25 23:29:38,010 | CRITICAL | System: Lost Connection to Device 192.168.2.241 Jun 25 23:29:40 gserver python3[43118]: 2025-06-25 23:29:40,856 | ERROR | System: communicating with interface1, trying to reconnect: Timed out waiting for an acknowledgment Jun 25 23:29:40 gserver python3[43118]: 2025-06-25 23:29:40,856 | ERROR | System: closing interface1: [Errno 32] Broken pipe ``` After this happens, the script goes into a retry loop and never reconnects to the node: ``` Sun 25 23:29:40 gserver python3[43118]: 2025-06-25 23:29:40,856 | DEBUG | System: Retrying interface1 in 15 seconds Jun 25 23:29:55 gserver python3[43118]: 2025-06-25 23:29:55,858 | DEBUG | System: Retrying Interface1 Jun 25 23:29:55 gserver python3[43118]: 2025-06-25 23:29:55,859 | ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused Jun 25 23:30:15 gserver python3[43118]: 2025-06-25 23:30:15,860 | DEBUG | System: Retrying interface1 in 15 seconds Jun 25 23:30:30 gserver python3[43118]: 2025-06-25 23:30:30,861 | DEBUG | System: Retrying Interface1 Jun 25 23:30:30 gserver python3[43118]: 2025-06-25 23:30:30,862 | ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused Jun 25 23:30:50 gserver python3[43118]: 2025-06-25 23:30:50,864 | DEBUG | System: Retrying interface1 in 15 seconds Jun 25 23:31:05 gserver python3[43118]: 2025-06-25 23:31:05,865 | DEBUG | System: Retrying Interface1 Jun 25 23:31:05 gserver python3[43118]: 2025-06-25 23:31:05,866 | ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused Jun 25 23:31:25 gserver python3[43118]: 2025-06-25 23:31:25,867 | DEBUG | System: Retrying interface1 in 15 seconds ``` Simply restarting the script gets it going again. Restarting the node while in this loop does not. It just keeps retrying. <!-- github-sync-id:3181052789 -->
adam closed this issue 2026-10-05 01:18:43 +02:00
adam commented 2026-10-05 01:18:44 +02:00
Author
Owner
Copy link
Copy source

The API had a loss of pipe by client termination. I can't recover from this well (the API can't handle it I don't think) I know it's stable as my lab is WiFi but I have been tracking a issue with the Linux-native connection I might be able to pass data into meshtastic for.

To help situation try getting a better WiFi signal for the node they have crappy antennas normally.

The API had a loss of pipe by client termination. I can't recover from this well (the API can't handle it I don't think) I know it's stable as my lab is WiFi but I have been tracking a issue with the Linux-native connection I might be able to pass data into meshtastic for. To help situation try getting a better WiFi signal for the node they have crappy antennas normally. <!-- github-comment-id:3013366036 -->
adam commented 2026-10-05 01:18:44 +02:00
Author
Owner
Copy link
Copy source

The signal is strong from the onboard antenna. I will try an external antenna, though. Assuming that doesn't solve the problem, would it be possible to have the daemon reload on connection failure? I'd rather have a user session drop over the whole system just going offline without notice or intervention.

The signal is strong from the onboard antenna. I will try an external antenna, though. Assuming that doesn't solve the problem, would it be possible to have the daemon reload on connection failure? I'd rather have a user session drop over the whole system just going offline without notice or intervention. <!-- github-comment-id:3013802203 -->
adam commented 2026-10-05 01:18:45 +02:00
Author
Owner
Copy link
Copy source

Just to update: I have installed an external antenna. The problem is persisting. I have also replaced the whole unit as I was having other issues with it. The problem persists.

The best I could to do to mitigate this was to edit system.py to exit instead of calling the retry function on connection error. I stuck mesh_bot.py in an infinite loop bash script so it will reload on exit.

Just to update: I have installed an external antenna. The problem is persisting. I have also replaced the whole unit as I was having other issues with it. The problem persists. The best I could to do to mitigate this was to edit system.py to exit instead of calling the retry function on connection error. I stuck mesh_bot.py in an infinite loop bash script so it will reload on exit. <!-- github-comment-id:3017028956 -->
adam commented 2026-10-05 01:18:46 +02:00
Author
Owner
Copy link
Copy source

Yes dumping the API would do it, I might add a config.ini for this procedure to make it optional.

Yes dumping the API would do it, I might add a config.ini for this procedure to make it optional. <!-- github-comment-id:3021642146 -->
adam commented 2026-10-05 01:18:46 +02:00
Author
Owner
Copy link
Copy source

I just started seeing this behaviour, too. I thought it was related to updated code I had pulled in, but don't see anything related. I also updated the node to the latest alpha when trying to resolve the issue, to no avail. So I'm not sure why this started happening for me, but I'm seeing the exact same behaviour. I found it strange that on both Windows and Linux the error string passed through are both os-specific variants of the "Connection refused" message, although it persists when the node has recovered and is definitely listening on :80. I was suspecting it might have been trying to reconnect to the wrong host.
Up until recently it reconnected OK, so I've been digging to find out what changed to no avail.

I just started seeing this behaviour, too. I thought it was related to updated code I had pulled in, but don't see anything related. I also updated the node to the latest alpha when trying to resolve the issue, to no avail. So I'm not sure why this started happening for me, but I'm seeing the exact same behaviour. I found it strange that on both Windows and Linux the error string passed through are both os-specific variants of the "Connection refused" message, although it persists when the node has recovered and is definitely listening on :80. I was suspecting it might have been trying to reconnect to the wrong host. Up until recently it reconnected OK, so I've been digging to find out what changed to no avail. <!-- github-comment-id:3055005779 -->
adam commented 2026-10-05 01:18:47 +02:00
Author
Owner
Copy link
Copy source

The API was never amazing for reconnecting

The API was never amazing for reconnecting <!-- github-comment-id:3055102092 -->
adam commented 2026-10-05 01:18:48 +02:00
Author
Owner
Copy link
Copy source

By API do you mean meshtastic.tcp_interface.TCPInterface ?
I'm really curious to dig into the "Connection refused" message, as it's usually pretty telling that it's connecting to something with a closed port. But I have separate TCP monitoring on the node that shows that the node is listening on the port and shouldn't be sending any RST packets to new connections. Makes me wonder where the socket is reconnecting to.
I just ran a WireShark capture, as well as a listener on localhost:80, and didn't see any connection attempts or connection refuseds. Strange.

By API do you mean `meshtastic.tcp_interface.TCPInterface` ? I'm really curious to dig into the "Connection refused" message, as it's usually pretty telling that it's connecting to something with a closed port. But I have separate TCP monitoring on the node that shows that the node is listening on the port and shouldn't be sending any RST packets to new connections. Makes me wonder where the socket is reconnecting to. I just ran a WireShark capture, as well as a listener on localhost:80, and didn't see any connection attempts or connection refuseds. Strange. <!-- github-comment-id:3057159079 -->
adam commented 2026-10-05 01:18:48 +02:00
Author
Owner
Copy link
Copy source

Corrected a lot of wonky logic to hopefully manage this better

Corrected a lot of wonky logic to hopefully manage this better <!-- github-comment-id:3097102541 -->
adam commented 2026-10-05 01:18:49 +02:00
Author
Owner
Copy link
Copy source

should be resolved

should be resolved <!-- github-comment-id:3100654434 -->
adam commented 2026-10-05 01:18:50 +02:00
Author
Owner
Copy link
Copy source

I'm afraid it has not been resolved:

Jul 24 03:03:32 gserver python3[147338]: 2025-07-24 03:03:32,480 |    DEBUG | System: Retrying interface1 in 15 seconds
Jul 24 03:03:47 gserver python3[147338]: 2025-07-24 03:03:47,482 |    DEBUG | System: Retrying Interface1
Jul 24 03:03:47 gserver python3[147338]: 2025-07-24 03:03:47,482 |    ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused
Jul 24 03:04:07 gserver python3[147338]: 2025-07-24 03:04:07,484 |    DEBUG | System: Retrying interface1 in 15 seconds
Jul 24 03:04:22 gserver python3[147338]: 2025-07-24 03:04:22,485 |    DEBUG | System: Retrying Interface1
Jul 24 03:04:22 gserver python3[147338]: 2025-07-24 03:04:22,485 |    ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused
Jul 24 03:04:42 gserver python3[147338]: 2025-07-24 03:04:42,486 |    DEBUG | System: Retrying interface1 in 15 seconds
Jul 24 03:04:57 gserver python3[147338]: 2025-07-24 03:04:57,487 |    DEBUG | System: Retrying Interface1
Jul 24 03:04:57 gserver python3[147338]: 2025-07-24 03:04:57,488 |    ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused
Jul 24 03:05:17 gserver python3[147338]: 2025-07-24 03:05:17,489 |    DEBUG | System: Retrying interface1 in 15 seconds

It looks like the changes from tag 1.7.8 were not carried to the current release 1.7.9. The tag name for 1.7.8 has an extra . between the v and the 1 (v.1.7.8) which is different than the other tags (v1.7.7, v1.7.9). Not sure if that would cause this. It would be nice if you provided a little more time for testing before closing an issue. It takes time to test intermittant issues like this.

I'm afraid it has not been resolved: ``` Jul 24 03:03:32 gserver python3[147338]: 2025-07-24 03:03:32,480 | DEBUG | System: Retrying interface1 in 15 seconds Jul 24 03:03:47 gserver python3[147338]: 2025-07-24 03:03:47,482 | DEBUG | System: Retrying Interface1 Jul 24 03:03:47 gserver python3[147338]: 2025-07-24 03:03:47,482 | ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused Jul 24 03:04:07 gserver python3[147338]: 2025-07-24 03:04:07,484 | DEBUG | System: Retrying interface1 in 15 seconds Jul 24 03:04:22 gserver python3[147338]: 2025-07-24 03:04:22,485 | DEBUG | System: Retrying Interface1 Jul 24 03:04:22 gserver python3[147338]: 2025-07-24 03:04:22,485 | ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused Jul 24 03:04:42 gserver python3[147338]: 2025-07-24 03:04:42,486 | DEBUG | System: Retrying interface1 in 15 seconds Jul 24 03:04:57 gserver python3[147338]: 2025-07-24 03:04:57,487 | DEBUG | System: Retrying Interface1 Jul 24 03:04:57 gserver python3[147338]: 2025-07-24 03:04:57,488 | ERROR | System: Error Opening interface1 on: [Errno 111] Connection refused Jul 24 03:05:17 gserver python3[147338]: 2025-07-24 03:05:17,489 | DEBUG | System: Retrying interface1 in 15 seconds ``` It looks like the changes from tag 1.7.8 were not carried to the current release 1.7.9. The tag name for 1.7.8 has an extra . between the v and the 1 (v.1.7.8) which is different than the other tags (v1.7.7, v1.7.9). Not sure if that would cause this. It would be nice if you provided a little more time for testing before closing an issue. It takes time to test intermittant issues like this. <!-- github-comment-id:3112871430 -->
adam commented 2026-10-05 01:18:50 +02:00
Author
Owner
Copy link
Copy source

Connection refused I can't handle. Neither can the API

Testing is done by rebooting a node and reconnection.

What did you do to get a connection refused.

Connection refused I can't handle. Neither can the API Testing is done by rebooting a node and reconnection. What did you do to get a connection refused. <!-- github-comment-id:3113434389 -->
adam commented 2026-10-05 01:18:51 +02:00
Author
Owner
Copy link
Copy source

There is a new setting you might like


#Do not retry enabling interface if it fails, just exit to let OS restart the bot
dont_retry_disconnect = False
`

<!-- github-comment-id:3113443515 -->
There is a new setting you might like ```ini #Do not retry enabling interface if it fails, just exit to let OS restart the bot dont_retry_disconnect = False ` <!-- github-comment-id:3113443515 -->
adam commented 2026-10-05 01:18:52 +02:00
Author
Owner
Copy link
Copy source

Connection refused I can't handle. Neither can the API

Testing is done by rebooting a node and reconnection.

What did you do to get a connection refused.

This was the same behavior as before where it happened randomly when just sitting idle.

There is a new setting you might like

#Do not retry enabling interface if it fails, just exit to let OS restart the bot
dont_retry_disconnect = False
`

I didn't realize this was added. I was looking at the code for v1.7.9 and v.1.7.8 and saw the additions for v.1.7.8 were not present so I figured the fixed reboot code was just missing. I have enabled the option and I'll report how it goes.

> Connection refused I can't handle. Neither can the API > > Testing is done by rebooting a node and reconnection. > > What did you do to get a connection refused. This was the same behavior as before where it happened randomly when just sitting idle. > There is a new setting you might like > > #Do not retry enabling interface if it fails, just exit to let OS restart the bot > dont_retry_disconnect = False > ` I didn't realize this was added. I was looking at the code for v1.7.9 and v.1.7.8 and saw the additions for v.1.7.8 were not present so I figured the fixed reboot code was just missing. I have enabled the option and I'll report how it goes. <!-- github-comment-id:3115374769 -->
adam commented 2026-10-05 01:18:52 +02:00
Author
Owner
Copy link
Copy source

The new option appears to be working. Thank you for implementing it.

The new option appears to be working. Thank you for implementing it. <!-- github-comment-id:3123368909 -->
adam commented 2026-10-05 01:18:53 +02:00
Author
Owner
Copy link
Copy source

Welcome glad it's working!

Welcome glad it's working! <!-- github-comment-id:3123454771 -->
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adam/meshing-around#61