adam/meshcore-mqtt
1
0 Fork 0
mirror of https://github.com/ipnet-mesh/meshcore-mqtt.git synced 2026-10-31 03:02:06 +01:00
Code Issues 8 Packages Projects Releases Wiki Activity

Retry logic for DM #2

New Issue
Closed
opened 2026-10-05 06:08:31 +02:00 by adam · 9 comments
adam commented 2026-10-05 06:08:31 +02:00
Owner
Copy link
Copy source

I'm not sure where this fits - whether it should be in the mechcore device or in the application layer, but it seems retry logic is missing?

When monitoring the log, a message send always immediately returns success, however the message is not always received.

Works flawlessly when receiving node is direct, but when travelling (multi-hop) I lose about half the messages.

If this is an app issue, implementing the same logic as for the companion app for more reliable messages.

Note: I was previously using meshcore-cli for sending messages and the success rate was very good.

This may not be a retry issue - it could also be a bridge issue. I'll do more testing with debug on.

I'm not sure where this fits - whether it should be in the mechcore device or in the application layer, but it seems retry logic is missing? When monitoring the log, a message send always immediately returns success, however the message is not always received. Works flawlessly when receiving node is direct, but when travelling (multi-hop) I lose about half the messages. If this is an app issue, implementing the same logic as for the companion app for more reliable messages. Note: I was previously using meshcore-cli for sending messages and the success rate was very good. This may not be a retry issue - it could also be a bridge issue. I'll do more testing with debug on. <!-- github-sync-id:3364670308 -->
adam closed this issue 2026-10-05 06:08:31 +02:00
adam commented 2026-10-05 06:08:32 +02:00
Author
Owner
Copy link
Copy source

DEBUG output for message not received by intended recipient:

2025-08-29 19:16:36,524 - meshcore_mqtt.mqtt_worker - INFO - Received MQTT command: send_msg = {"destination":"af79b3ba6d917a4324299c7c56cd125efa3db175c75399ef117b6e9ae306601f", "message":"Test 1"}
2025-08-29 19:16:36,535 - meshcore_mqtt.message_queue.meshcore - DEBUG - Queued mqtt_command from mqtt to meshcore (id: a0c5bbfb)
2025-08-29 19:16:36,535 - meshcore_mqtt.message_queue.meshcore - DEBUG - Dequeued mqtt_command from mqtt to meshcore (id: a0c5bbfb)
2025-08-29 19:16:36,535 - meshcore_mqtt.meshcore_worker - DEBUG - Processing message: mqtt_command
2025-08-29 19:16:36,535 - meshcore_mqtt.meshcore_worker - INFO - Processing MQTT command: send_msg
2025-08-29 19:16:36,535 - meshcore - DEBUG - Sending message to af79b3ba6d91: Test 1
2025-08-29 19:16:36,535 - meshcore - DEBUG - Sending raw data: 020000d453b168af79b3ba6d91546573742031
2025-08-29 19:16:36,535 - meshcore - DEBUG - sending pkt : b'<\x13\x00\x02\x00\x00\xd4S\xb1h\xafy\xb3\xbam\x91Test 1'
2025-08-29 19:16:36,536 - meshcore - DEBUG - Waiting for events [<EventType.MSG_SENT: 'message_sent'>, <EventType.ERROR: 'command_error'>], timeout=5
2025-08-29 19:16:36,539 - meshcore - DEBUG - Received data: 06004030950a5c1b0000
2025-08-29 19:16:36,539 - meshcore - DEBUG - Dispatching event: EventType.MSG_SENT, {'type': 0, 'expected_ack': b'@0\x95\n', 'suggested_timeout': 7004}, {'type': 0, 'expected_ack': '4030950a'}
2025-08-29 19:16:36,540 - meshcore_mqtt.meshcore_worker - INFO - MeshCore command 'send_msg' successful

It looks like it got the timeout for the ack, and presumably that is where the retry logic should kick in.

DEBUG output for message not received by intended recipient: ``` 2025-08-29 19:16:36,524 - meshcore_mqtt.mqtt_worker - INFO - Received MQTT command: send_msg = {"destination":"af79b3ba6d917a4324299c7c56cd125efa3db175c75399ef117b6e9ae306601f", "message":"Test 1"} 2025-08-29 19:16:36,535 - meshcore_mqtt.message_queue.meshcore - DEBUG - Queued mqtt_command from mqtt to meshcore (id: a0c5bbfb) 2025-08-29 19:16:36,535 - meshcore_mqtt.message_queue.meshcore - DEBUG - Dequeued mqtt_command from mqtt to meshcore (id: a0c5bbfb) 2025-08-29 19:16:36,535 - meshcore_mqtt.meshcore_worker - DEBUG - Processing message: mqtt_command 2025-08-29 19:16:36,535 - meshcore_mqtt.meshcore_worker - INFO - Processing MQTT command: send_msg 2025-08-29 19:16:36,535 - meshcore - DEBUG - Sending message to af79b3ba6d91: Test 1 2025-08-29 19:16:36,535 - meshcore - DEBUG - Sending raw data: 020000d453b168af79b3ba6d91546573742031 2025-08-29 19:16:36,535 - meshcore - DEBUG - sending pkt : b'<\x13\x00\x02\x00\x00\xd4S\xb1h\xafy\xb3\xbam\x91Test 1' 2025-08-29 19:16:36,536 - meshcore - DEBUG - Waiting for events [<EventType.MSG_SENT: 'message_sent'>, <EventType.ERROR: 'command_error'>], timeout=5 2025-08-29 19:16:36,539 - meshcore - DEBUG - Received data: 06004030950a5c1b0000 2025-08-29 19:16:36,539 - meshcore - DEBUG - Dispatching event: EventType.MSG_SENT, {'type': 0, 'expected_ack': b'@0\x95\n', 'suggested_timeout': 7004}, {'type': 0, 'expected_ack': '4030950a'} 2025-08-29 19:16:36,540 - meshcore_mqtt.meshcore_worker - INFO - MeshCore command 'send_msg' successful ``` It looks like it got the timeout for the ack, and presumably that is where the retry logic should kick in. <!-- github-comment-id:3236008549 -->
adam commented 2026-10-05 06:08:33 +02:00
Author
Owner
Copy link
Copy source

It looks like now and again the bridge loses mqtt connection and doesn't re-establish. Then messages do not get through.

It looks like now and again the bridge loses mqtt connection and doesn't re-establish. Then messages do not get through. <!-- github-comment-id:3238760771 -->
adam commented 2026-10-05 06:08:33 +02:00
Author
Owner
Copy link
Copy source

I've renamed this issue. It appears if the recipient node's path changes, then messages will not get through.

Solution would be to retry the message a certain number of times, then reset the path if the maximum retry count is reached. When path is reset, try once more.

I've renamed this issue. It appears if the recipient node's path changes, then messages will not get through. Solution would be to retry the message a certain number of times, then reset the path if the maximum retry count is reached. When path is reset, try once more. <!-- github-comment-id:3240987223 -->
adam commented 2026-10-05 06:08:34 +02:00
Author
Owner
Copy link
Copy source

@claude, I don't believe we currently track acknowledgements of sent messages. Does meshcore_py provide a method of waiting (or handling acks asyncronously), or would this need to be implemented in application logic?

@claude, I don't believe we currently track acknowledgements of sent messages. Does `meshcore_py` provide a method of waiting (or handling acks asyncronously), or would this need to be implemented in application logic? <!-- github-comment-id:3315369951 -->
adam commented 2026-10-05 06:08:35 +02:00
Author
Owner
Copy link
Copy source

Claude finished @jinglemansweep's task —— View job

Analysis of acknowledgement handling in meshcore_py

@jinglemansweep Based on my investigation of the codebase and the debug logs shown in this issue, here's what I found:

Current State

meshcore_py does provide acknowledgement information, but the bridge doesn't currently implement retry logic for failed message sends. From the debug logs in the issue:

Dispatching event: EventType.MSG_SENT, {'type': 0, 'expected_ack': b'@0\x95\n', 'suggested_timeout': 7004}

The MSG_SENT event from meshcore_py includes:

  • expected_ack: The acknowledgement code to expect
  • suggested_timeout: Timeout in milliseconds (7004ms in the example)

How it Currently Works

  1. When send_msg() is called (meshcore_mqtt/meshcore_worker.py:301), it awaits the result from meshcore_py
  2. The result is checked for success/error (meshcore_mqtt/meshcore_worker.py:351-361)
  3. If successful, it logs "MeshCore command 'send_msg' successful"
  4. No retry logic is implemented - if the message fails, it just logs an error

What's Missing

The bridge currently:

  • ✅ Receives the MSG_SENT event with ack/timeout info
  • ❌ Does NOT wait for the actual acknowledgement
  • ❌ Does NOT implement retry logic when acks aren't received
  • ❌ Does NOT reset paths when retries fail (as suggested by @seagull9000)

Recommendation

This needs to be implemented at the application level (in meshcore-mqtt) because:

  1. meshcore_py provides the building blocks (MSG_SENT events with ack info) but doesn't implement the retry logic itself
  2. The retry strategy is application-specific - different applications may want different retry counts, delays, and path reset behaviors
  3. The companion app apparently has this logic, suggesting it's an application responsibility

Implementation Approach

To implement proper retry logic, the bridge would need to:

  1. Track acknowledgements: After sending a message, wait for the expected_ack with the suggested_timeout
  2. Implement retry logic: If no ack is received within timeout, retry X times
  3. Reset path on failure: After max retries, reset the routing path and try once more
  4. Handle multi-hop scenarios: Be more aggressive with retries for indirect routes

This would require subscribing to additional events (like ACK events if meshcore_py provides them) or implementing a timeout-based retry mechanism in the meshcore_worker.

Todo List:

  • Check if retry logic already exists
  • Provide comprehensive answer about acknowledgement tracking options

Job run on branch claude/issue-2-20250920-2349

**Claude finished @jinglemansweep's task** —— [View job](https://github.com/ipnet-mesh/meshcore-mqtt/actions/runs/17886292473) --- ## Analysis of acknowledgement handling in meshcore_py @jinglemansweep Based on my investigation of the codebase and the debug logs shown in this issue, here's what I found: ### Current State **meshcore_py does provide acknowledgement information**, but **the bridge doesn't currently implement retry logic** for failed message sends. From the debug logs in the issue: ``` Dispatching event: EventType.MSG_SENT, {'type': 0, 'expected_ack': b'@0\x95\n', 'suggested_timeout': 7004} ``` The `MSG_SENT` event from meshcore_py includes: - `expected_ack`: The acknowledgement code to expect - `suggested_timeout`: Timeout in milliseconds (7004ms in the example) ### How it Currently Works 1. When `send_msg()` is called (`meshcore_mqtt/meshcore_worker.py:301`), it awaits the result from meshcore_py 2. The result is checked for success/error (`meshcore_mqtt/meshcore_worker.py:351-361`) 3. If successful, it logs "MeshCore command 'send_msg' successful" 4. **No retry logic is implemented** - if the message fails, it just logs an error ### What's Missing The bridge currently: - ✅ Receives the MSG_SENT event with ack/timeout info - ❌ Does NOT wait for the actual acknowledgement - ❌ Does NOT implement retry logic when acks aren't received - ❌ Does NOT reset paths when retries fail (as suggested by @seagull9000) ### Recommendation This needs to be implemented at the **application level** (in meshcore-mqtt) because: 1. **meshcore_py provides the building blocks** (MSG_SENT events with ack info) but doesn't implement the retry logic itself 2. **The retry strategy is application-specific** - different applications may want different retry counts, delays, and path reset behaviors 3. **The companion app apparently has this logic**, suggesting it's an application responsibility ### Implementation Approach To implement proper retry logic, the bridge would need to: 1. **Track acknowledgements**: After sending a message, wait for the expected_ack with the suggested_timeout 2. **Implement retry logic**: If no ack is received within timeout, retry X times 3. **Reset path on failure**: After max retries, reset the routing path and try once more 4. **Handle multi-hop scenarios**: Be more aggressive with retries for indirect routes This would require subscribing to additional events (like ACK events if meshcore_py provides them) or implementing a timeout-based retry mechanism in the meshcore_worker. ### Todo List: - [x] Check if retry logic already exists - [x] Provide comprehensive answer about acknowledgement tracking options --- [Job run](https://github.com/ipnet-mesh/meshcore-mqtt/actions/runs/12720416445) on branch `claude/issue-2-20250920-2349` <!-- github-comment-id:3315370067 -->
adam commented 2026-10-05 06:08:35 +02:00
Author
Owner
Copy link
Copy source

@claude Can you produce a PR with a potential fix to this issue?

@claude Can you produce a PR with a potential fix to this issue? <!-- github-comment-id:3315374776 -->
adam commented 2026-10-05 06:08:36 +02:00
Author
Owner
Copy link
Copy source

Claude finished @jinglemansweep's task —— View job • claude/issue-2-20250921-0000 • Create PR ➔

✅ Completed retry logic implementation for message sending

Successfully added retry logic for direct messages with acknowledgement tracking and path reset functionality.

Implementation Summary:

Configuration System ✅:

  • Added retry parameters to MeshCoreConfig:
    • message_retry_count: Number of retry attempts (0-10, default: 3)
    • message_retry_delay: Base delay between retries with exponential backoff (0.5-30s, default: 2s)
    • reset_path_on_failure: Reset routing path after max retries (default: true)
  • Updated environment variable loading and CLI argument parsing

MeshCore Worker Enhancements ✅:

  • Added acknowledgement tracking with _pending_acks dictionary
  • Created _send_msg_with_retry() and _send_chan_msg_with_retry() methods
  • Implemented exponential backoff between retries
  • Added path reset functionality for direct messages after max retries
  • Subscribed to ACK events for acknowledgement tracking
  • Added helper methods: _wait_for_ack(), _on_ack_received(), _reset_path()

Testing ✅:

  • Created comprehensive test suite in tests/test_retry_logic.py with 13 test cases
  • Tests cover success/failure scenarios, exponential backoff, path reset, and ACK handling

Documentation ✅:

  • Created docs/RETRY_LOGIC.md with detailed feature documentation
  • Includes configuration examples, retry flow explanation, and use cases

How the Solution Works:

  1. When sending a message, the bridge now waits for acknowledgement with the timeout provided by MeshCore
  2. If no ACK is received, it retries with exponential backoff (2s, 4s, 8s, etc.)
  3. After max retries, for direct messages only, it resets the routing path and tries once more
  4. Configuration is flexible via config file, environment variables, or CLI arguments

This implementation addresses the issue where messages fail in multi-hop scenarios by providing automatic retries with intelligent backoff and path reset capabilities.

Job run on branch claude/issue-2-20250921-0000

**Claude finished @jinglemansweep's task** —— [View job](https://github.com/ipnet-mesh/meshcore-mqtt/actions/runs/17886365931) • [`claude/issue-2-20250921-0000`](https://github.com/ipnet-mesh/meshcore-mqtt/tree/claude/issue-2-20250921-0000) • [Create PR ➔](https://github.com/ipnet-mesh/meshcore-mqtt/compare/main...claude/issue-2-20250921-0000?quick_pull=1&title=feat%3A+Add+retry+logic+for+message+sending+with+acknowledgement+tracking&body=%23%23+Summary%0A%0A-+Implements+retry+logic+for+message+sending+to+improve+reliability+in+multi-hop+mesh+networks%0A-+Adds+acknowledgement+tracking+with+configurable+timeout%0A-+Includes+exponential+backoff+and+path+reset+functionality%0A%0AFixes+%232%0A%0A%23%23+Changes%0A%0A%23%23%23+Configuration%0A-+Added+%60message_retry_count%60%2C+%60message_retry_delay%60%2C+and+%60reset_path_on_failure%60+parameters%0A-+Updated+environment+variable+and+CLI+argument+parsing%0A%0A%23%23%23+MeshCore+Worker%0A-+Implemented+%60_send_msg_with_retry%28%29%60+and+%60_send_chan_msg_with_retry%28%29%60+methods%0A-+Added+ACK+tracking+and+waiting+logic%0A-+Implemented+path+reset+for+direct+messages+after+max+retries%0A%0A%23%23%23+Testing%0A-+Added+comprehensive+test+suite+with+13+test+cases%0A-+Tests+cover+various+retry+scenarios+and+edge+cases%0A%0A%23%23%23+Documentation%0A-+Created+detailed+documentation+in+%60docs%2FRETRY_LOGIC.md%60%0A-+Includes+configuration+examples+and+usage+instructions%0A%0A%23%23+Test+Plan%0A%0A-+%5Bx%5D+Unit+tests+pass%0A-+%5B+%5D+Tested+with+actual+MeshCore+hardware%0A-+%5B+%5D+Verified+multi-hop+message+delivery+improvement%0A-+%5B+%5D+Confirmed+backward+compatibility%0A%0AGenerated+with+%5BClaude+Code%5D%28https%3A%2F%2Fclaude.ai%2Fcode%29) --- ### ✅ Completed retry logic implementation for message sending Successfully added retry logic for direct messages with acknowledgement tracking and path reset functionality. ### Implementation Summary: **Configuration System** ✅: - Added retry parameters to `MeshCoreConfig`: - `message_retry_count`: Number of retry attempts (0-10, default: 3) - `message_retry_delay`: Base delay between retries with exponential backoff (0.5-30s, default: 2s) - `reset_path_on_failure`: Reset routing path after max retries (default: true) - Updated environment variable loading and CLI argument parsing **MeshCore Worker Enhancements** ✅: - Added acknowledgement tracking with `_pending_acks` dictionary - Created `_send_msg_with_retry()` and `_send_chan_msg_with_retry()` methods - Implemented exponential backoff between retries - Added path reset functionality for direct messages after max retries - Subscribed to ACK events for acknowledgement tracking - Added helper methods: `_wait_for_ack()`, `_on_ack_received()`, `_reset_path()` **Testing** ✅: - Created comprehensive test suite in `tests/test_retry_logic.py` with 13 test cases - Tests cover success/failure scenarios, exponential backoff, path reset, and ACK handling **Documentation** ✅: - Created `docs/RETRY_LOGIC.md` with detailed feature documentation - Includes configuration examples, retry flow explanation, and use cases ### How the Solution Works: 1. **When sending a message**, the bridge now waits for acknowledgement with the timeout provided by MeshCore 2. **If no ACK is received**, it retries with exponential backoff (2s, 4s, 8s, etc.) 3. **After max retries**, for direct messages only, it resets the routing path and tries once more 4. **Configuration is flexible** via config file, environment variables, or CLI arguments This implementation addresses the issue where messages fail in multi-hop scenarios by providing automatic retries with intelligent backoff and path reset capabilities. --- [Job run](https://github.com/ipnet-mesh/meshcore-mqtt/actions/runs/17886305071) on branch [claude/issue-2-20250921-0000](https://github.com/ipnet-mesh/meshcore-mqtt/tree/claude/issue-2-20250921-0000) <!-- github-comment-id:3315375019 -->
adam commented 2026-10-05 06:08:37 +02:00
Author
Owner
Copy link
Copy source

Thanks so much for this.
I travel a lot and being able to receive status and alert messages while away from home really makes this bridge useful.
I'll test and close once confirmed.

Thanks so much for this. I travel a lot and being able to receive status and alert messages while away from home really makes this bridge useful. I'll test and close once confirmed. <!-- github-comment-id:3315703784 -->
adam commented 2026-10-05 06:08:37 +02:00
Author
Owner
Copy link
Copy source

I need to test it first and merge the branch if it works. Hopefully will get a chance in a couple of days

I need to test it first and merge the branch if it works. Hopefully will get a chance in a couple of days <!-- github-comment-id:3315928135 -->
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adam/meshcore-mqtt#2