Corruption in some packets due to payload being partially overwritten by path bytes #94

New Issue

2026-10-27T09:32:50+01:00

adam commented

2026-10-27 09:32:50 +01:00

A high amount of packet corruption was being detected from several pyMC repeaters using E22P 915 modules. An investigation was started to determine where the corruption was happening which revealed a corruption pattern that seems deterministic.

In advertisements and group text messages, which were chosen for practical evidence collection, it is evident that the path bytes are being written over a portion of the payload up till the last 2 bytes. The corruption is seen in both letsmesh and error messages when these packet are attempted to be ingested into Corescope.

The hashes are different and the signatures are invalid. Sometimes the corruption is happening on retransmit and some nearby observers pass the packets intact to letsmesh. Sometimes the observers on pyMC-repeater send the corrupt packets to MQTT.

It is unclear what preconditions trigger the corruption.

Some example advert packets from letsmesh.

A high amount of packet corruption was being detected from several pyMC repeaters using E22P 915 modules. An investigation was started to determine where the corruption was happening which revealed a corruption pattern that seems deterministic. In advertisements and group text messages, which were chosen for practical evidence collection, it is evident that the path bytes are being written over a portion of the payload up till the last 2 bytes. The corruption is seen in both letsmesh and error messages when these packet are attempted to be ingested into Corescope. The hashes are different and the signatures are invalid. Sometimes the corruption is happening on retransmit and some nearby observers pass the packets intact to letsmesh. Sometimes the observers on pyMC-repeater send the corrupt packets to MQTT. It is unclear what preconditions trigger the corruption. Some example advert packets from letsmesh. <img width="1097" height="1382" alt="Image" src="https://github.com/user-attachments/assets/1241dd33-c45b-4fd9-bcb2-0ae9e3752ca8" /> <img width="1142" height="1399" alt="Image" src="https://github.com/user-attachments/assets/589dba91-ca78-42ab-ab04-8380513958c4" />

adam closed this issue

2026-10-27 09:32:50 +01:00

adam commented

2026-10-27 09:32:51 +01:00

I would like to add one a further example.

Original packet 15 02 a6 49 11 72 e8 5b 81 2e 7a ba 26 7a d7 be ba 63 bb 0a 82 f0 b4 8c 11 67 be 27 78 19 f6 56 7f 79 08 83 fa 7b 1f 76 13 ca d1 73 86 81 9d 28 ac fc c7 d6 c7 5a 77 46 aa b2 1d 76 33 b0 57 f3 90 87 04 15 27 9c 63 95 ae 93 00 28 58 de 91 6c d6 6a 0b 49 76 94 5a 95 f1 d8 70 6a ff 29 24 15 94 59 a2 ec 75 16 39 94 64 90 60 68 93 dc e3 b3 f9 df b8 49 5a 33 24 d4 6c 0d f4 ee a3 74 a2 8b 62 59 5f 9e 62 40 94
This is the packet as heard by the pyMC repeater and sent via mqtt

Corrupted packet 15 03 a6 49 08 11 72 e8 5b 81 2e 7a ba 26 7a d7 be ba 63 bb 0a 82 f0 b4 8c 11 67 be 27 78 19 f6 56 7f 79 08 83 fa 7b 1f 76 13 ca d1 73 86 81 9d 28 ac fc c7 d6 c7 5a 77 46 aa b2 1d 76 33 b0 57 f3 90 87 04 15 27 9c 63 95 ae 93 00 28 58 de 91 6c d6 6a 0b 49 76 94 5a 95 f1 d8 70 6a ff 29 24 15 94 59 a2 ec 75 16 39 94 64 90 60 68 93 dc e3 b3 f9 df b8 49 5a 33 24 d4 6c 0d f4 ee a3 74 a2 15 03 a6 49 94 11 72 e8
as transmitted

End of the payload
Good: 74 A2 8B 62 59 5F 9E 62 40 94
Bad: 74 a2 15 03 a6 49 94 11 72 e8
Note the path a6 49 94 then the beginning of the payload 11 72 e2. The 94 is significant as it's not in the corrupted packet as received/transmitted, but the 94 was seen in paths by other observers, could a second reception of the same hash could be a contributing factor?

I would like to add one a further example. Original packet 15 02 a6 49 11 72 e8 5b 81 2e 7a ba 26 7a d7 be ba 63 bb 0a 82 f0 b4 8c 11 67 be 27 78 19 f6 56 7f 79 08 83 fa 7b 1f 76 13 ca d1 73 86 81 9d 28 ac fc c7 d6 c7 5a 77 46 aa b2 1d 76 33 b0 57 f3 90 87 04 15 27 9c 63 95 ae 93 00 28 58 de 91 6c d6 6a 0b 49 76 94 5a 95 f1 d8 70 6a ff 29 24 15 94 59 a2 ec 75 16 39 94 64 90 60 68 93 dc e3 b3 f9 df b8 49 5a 33 24 d4 6c 0d f4 ee a3 74 a2 8b 62 59 5f 9e 62 40 94 This is the packet as heard by the pyMC repeater and sent via mqtt Corrupted packet 15 03 a6 49 08 11 72 e8 5b 81 2e 7a ba 26 7a d7 be ba 63 bb 0a 82 f0 b4 8c 11 67 be 27 78 19 f6 56 7f 79 08 83 fa 7b 1f 76 13 ca d1 73 86 81 9d 28 ac fc c7 d6 c7 5a 77 46 aa b2 1d 76 33 b0 57 f3 90 87 04 15 27 9c 63 95 ae 93 00 28 58 de 91 6c d6 6a 0b 49 76 94 5a 95 f1 d8 70 6a ff 29 24 15 94 59 a2 ec 75 16 39 94 64 90 60 68 93 dc e3 b3 f9 df b8 49 5a 33 24 d4 6c 0d f4 ee a3 74 a2 15 03 a6 49 94 11 72 e8 as transmitted End of the payload Good: 74 A2 8B 62 59 5F 9E 62 40 94 Bad: 74 a2 15 03 a6 49 94 11 72 e8 Note the path a6 49 94 then the beginning of the payload 11 72 e2. The 94 is significant as it's not in the corrupted packet as received/transmitted, but the 94 was seen in paths by other observers, could a second reception of the same hash could be a contributing factor?

adam commented

2026-10-27 09:32:52 +01:00

The devices in use are Luckfox Pico Ultra with Zindello Industries Luckfox Ultrapeater board https://zindello.com.au/ultrapeater/ with E22P 915M modules. There are many in use around Sydney, but so far we have identified 4 unit exhibiting this behaviour. We can't rule out the others as we are not sure of how they are deployed. The ones identified so far are in high volume sites.

The devices in use are Luckfox Pico Ultra with Zindello Industries Luckfox Ultrapeater board https://zindello.com.au/ultrapeater/ with E22P 915M modules. There are many in use around Sydney, but so far we have identified 4 unit exhibiting this behaviour. We can't rule out the others as we are not sure of how they are deployed. The ones identified so far are in high volume sites.

adam commented

2026-10-27 09:32:52 +01:00

This is potentially caused by https://github.com/pyMC-dev/pyMC_core/pull/78

Waiting on @rightup to review as this is much more his area than mine.

This is potentially caused by https://github.com/pyMC-dev/pyMC_core/pull/78 Waiting on @rightup to review as this is much more his area than mine.

adam commented

2026-10-27 09:32:53 +01:00

not sure if im seeing the same thing here using the mishsmith board, but if you can show me how to get to the packets that have the crc errors, im sure i can get them to you as well.

This is what vscode/claude pulled out of the repeater log - hope this helps

can you look through the repeater log and give me some of the payloads for the CRC packets
2:41 PM


Yep. Here are some of the recent CRC packet payloads from repeater.log, all from today:

#199  13:56:47  RSSI=-97  SNR=-1.8  Len=71
1541bda3813f30b6926c3956770d403a28a30a673a3a49f06f9712a2c4c9dbd51db412188c6f92a9cd2bc2d905b196f80c54e17a6f4e16492a1a513801bdbd8777efb9eb47ed88

#200  13:59:00  RSSI=-75  SNR=-8.5  Len=69
1540812ba005402143af38ef8c5e5e5583c05f7a1af06f9712a2c4c9dbd511d21418bc5f9289e0357efc44fbc3dc6b0f7376fcddd1a32e01eec4ec287748b5519436d2c878

#201  14:00:43  RSSI=-98  SNR=-7.2  Len=75
1543e816243a0a329bdb20970e00b210bb26647b8d7e2b2e3e21aff06f9712a2c4c9dbd511d21418bc5f925fa53f8464a952e21df017fa1184b9f23b362067db8c161b013e05bce65ad1bb

#202  14:01:16  RSSI=-98  SNR=-7.0  Len=75
1543e886bda30a83810b8d7400e4966b345c075c20bc8aa3ab9d18f06c9722a2c4c9dbb274f11f18bc5f92103dbae808a5c48a8c0a55dad67fce66492a1a513801bdbd8777efb9eb47ed88
A few earlier today:

#196  10:52:42  RSSI=-67  SNR=+5.8  Len=69
154381f6dd3f0f4e1b189a7e2d6e83c85835714270f06f9712a2c4c9dbd511d21418bc5f92920e666de3773a10c73621b6f6c0b0d1baf5907a2c0ae9b797822f72b4ed4fa2

#197  10:53:17  RSSI=-98  SNR=-5.8  Len=73
1542e8860a8381d1a284ffb9907e2d9b452cc7f0e6995b0802a06c9712a2c4c9dbd511d21418bc5f925b68b2fc734aecf369e7dce89c02b81f28a2bf2c3f5250695c1b8662d56e7571

#198  10:56:34  RSSI=-76  SNR=-8.2  Len=69
1540818b6abe1df557434f459c7123c76368672603f06f9712a2c4c9dbd511d21418bc5f9276051048657cab804c1955fc609a4a18a32e0177cdb92c7748b55b525682fb7b
The recent ones mostly look like normal MeshCore-looking frames that failed CRC late, not random garbage. Several start with 1541, 1542, or 1543, which lines up with flood/group text style packets with different path lengths.

not sure if im seeing the same thing here using the mishsmith board, but if you can show me how to get to the packets that have the crc errors, im sure i can get them to you as well. This is what vscode/claude pulled out of the repeater log - hope this helps ```` can you look through the repeater log and give me some of the payloads for the CRC packets 2:41 PM Yep. Here are some of the recent CRC packet payloads from repeater.log, all from today: #199 13:56:47 RSSI=-97 SNR=-1.8 Len=71 1541bda3813f30b6926c3956770d403a28a30a673a3a49f06f9712a2c4c9dbd51db412188c6f92a9cd2bc2d905b196f80c54e17a6f4e16492a1a513801bdbd8777efb9eb47ed88 #200 13:59:00 RSSI=-75 SNR=-8.5 Len=69 1540812ba005402143af38ef8c5e5e5583c05f7a1af06f9712a2c4c9dbd511d21418bc5f9289e0357efc44fbc3dc6b0f7376fcddd1a32e01eec4ec287748b5519436d2c878 #201 14:00:43 RSSI=-98 SNR=-7.2 Len=75 1543e816243a0a329bdb20970e00b210bb26647b8d7e2b2e3e21aff06f9712a2c4c9dbd511d21418bc5f925fa53f8464a952e21df017fa1184b9f23b362067db8c161b013e05bce65ad1bb #202 14:01:16 RSSI=-98 SNR=-7.0 Len=75 1543e886bda30a83810b8d7400e4966b345c075c20bc8aa3ab9d18f06c9722a2c4c9dbb274f11f18bc5f92103dbae808a5c48a8c0a55dad67fce66492a1a513801bdbd8777efb9eb47ed88 A few earlier today: #196 10:52:42 RSSI=-67 SNR=+5.8 Len=69 154381f6dd3f0f4e1b189a7e2d6e83c85835714270f06f9712a2c4c9dbd511d21418bc5f92920e666de3773a10c73621b6f6c0b0d1baf5907a2c0ae9b797822f72b4ed4fa2 #197 10:53:17 RSSI=-98 SNR=-5.8 Len=73 1542e8860a8381d1a284ffb9907e2d9b452cc7f0e6995b0802a06c9712a2c4c9dbd511d21418bc5f925b68b2fc734aecf369e7dce89c02b81f28a2bf2c3f5250695c1b8662d56e7571 #198 10:56:34 RSSI=-76 SNR=-8.2 Len=69 1540818b6abe1df557434f459c7123c76368672603f06f9712a2c4c9dbd511d21418bc5f9276051048657cab804c1955fc609a4a18a32e0177cdb92c7748b55b525682fb7b The recent ones mostly look like normal MeshCore-looking frames that failed CRC late, not random garbage. Several start with 1541, 1542, or 1543, which lines up with flood/group text style packets with different path lengths. ````

adam commented

2026-10-27 09:32:53 +01:00

Some changes have been made to tighten the locking process between tx/rx please pull latest dev branch and see if this improves.

Some changes have been made to tighten the locking process between tx/rx please pull latest dev branch and see if this improves.

adam commented

2026-10-27 09:32:54 +01:00

I have had the chance to test pyMC Repeater over the past say or so. Running on the latest dev version I can see my repeater corrupting packets occasionally. If I look at the received packet as reported by in the front end it's fine, the log shows a packet received then re-transmitted. However the re-transmitted packet as heard by my observer and other repeaters has been corrupted and shows a small segment of bytes altered mid payload. Unfortunately I have to switch back to other hardware tomorrow (no AC power where the repeater is located) so won't be able to test further for now.

It seems to still be a frequent occurrence on the Sydney mesh and can be seen on Corescope/letsmesh. Often following either a message in Public or in #test, another un-decodable message will appear for channel 11 or D9, some times more than one, each one flooding the mesh. The corrupted messages path starts out long with the valid packets path to the pyMC repeater, then grows again, and sometimes corrupting and flooding multiple times. Comparing the payloads always a small segment of bytes mid way has been altered.

I have had the chance to test pyMC Repeater over the past say or so. Running on the latest dev version I can see my repeater corrupting packets occasionally. If I look at the received packet as reported by in the front end it's fine, the log shows a packet received then re-transmitted. However the re-transmitted packet as heard by my observer and other repeaters has been corrupted and shows a small segment of bytes altered mid payload. Unfortunately I have to switch back to other hardware tomorrow (no AC power where the repeater is located) so won't be able to test further for now. It seems to still be a frequent occurrence on the Sydney mesh and can be seen on Corescope/letsmesh. Often following either a message in Public or in #test, another un-decodable message will appear for channel 11 or D9, some times more than one, each one flooding the mesh. The corrupted messages path starts out long with the valid packets path to the pyMC repeater, then grows again, and sometimes corrupting and flooding multiple times. Comparing the payloads always a small segment of bytes mid way has been altered.

adam commented

2026-10-27 09:32:54 +01:00

Hi, what branch are you running? Main or Dev?

Hi, what branch are you running? Main or Dev?

adam commented

2026-10-27 09:32:55 +01:00

1.0.11-dev62 at least it think it's current, don't put it past me to make a noob mistake!

1.0.11-dev62 at least it think it's current, don't put it past me to make a noob mistake!

adam commented

2026-10-27 09:32:56 +01:00

1.0.11-dev62 at least it think it's current, don't put it past me to make a noob mistake!

Can you reach out to me on discord. I've been working with wazmac on this specific issue recently.

> 1.0.11-dev62 at least it think it's current, don't put it past me to make a noob mistake! > Can you reach out to me on discord. I've been working with wazmac on this specific issue recently.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/pyMC_Repeater#94