From time to time, UI will freeze when I hit send. It's a Heltec v4 node connected via USB to a Raspberry Pi.
Is this known behavior?
Or is there anything I can do to help debugging?
I have the impression that it may be related to ongoing radiosync in the background.
Maybe related: I also have a simple ping bot running (but it's not very busy and UI may freeze also when bot is 100% idle).
I'm currently on v2.7.9 #463a0c9 but it happened with earlier versions too.
From time to time, UI will freeze when I hit send. It's a Heltec v4 node connected via USB to a Raspberry Pi.
Is this known behavior?
Or is there anything I can do to help debugging?
I have the impression that it may be related to ongoing radiosync in the background.
Maybe related: I also have a simple ping bot running (but it's not very busy and UI may freeze also when bot is 100% idle).
I'm currently on v2.7.9 #463a0c9 but it happened with earlier versions too.
<!-- github-sync-id:4053356444 -->
It could be. Does the message eventually send? And when you say that the front end freezes, do you mean just the message sending section? Or does the whole frontend become unresponsive? Is there anything in the backend logs that correlates with when you see the freezes?
Radio operations will wait for the sink to finish; on my T114 that takes 5 or 6 seconds. I can cannibalize one of my Heltec ESP32 radios and see if I can replicate, but that won't be until later this week
It could be. Does the message eventually send? And when you say that the front end freezes, do you mean just the message sending section? Or does the whole frontend become unresponsive? Is there anything in the backend logs that correlates with when you see the freezes?
Radio operations will wait for the sink to finish; on my T114 that takes 5 or 6 seconds. I can cannibalize one of my Heltec ESP32 radios and see if I can replicate, but that won't be until later this week
<!-- github-comment-id:4033820133 -->
Yes, it does send. Eventually. But may take more than 5 seconds..... 20..30s sometimes.
And yes: it's the send section that freezes, everything else stays responsive.
Nothing special to be found in the log, except for ongoing radiosync. But there's no timestamp, at least not on the console, so it's hard to correlate.
Yes, it does send. Eventually. But may take more than 5 seconds..... 20..30s sometimes.
And yes: it's the send section that freezes, everything else stays responsive.
Nothing special to be found in the log, except for ongoing radiosync. But there's no timestamp, at least not on the console, so it's hard to correlate.
<!-- github-comment-id:4033874765 -->
Oh dang we definitely timestamps on our logs. Tonight or tomorrow I'll add timestamps to the logs, see if I can replicate, and go from there. One of the background tests is a safety mechanism if we can't export the private key from the radio; I've been meaning to gate that task on the key export, so maybe we can cut down on at least one of the periodic poles, but I suspect that it is more likely the one that is loading and unloading contacts. I'll do some digging
Oh dang we definitely timestamps on our logs. Tonight or tomorrow I'll add timestamps to the logs, see if I can replicate, and go from there. One of the background tests is a safety mechanism if we can't export the private key from the radio; I've been meaning to gate that task on the key export, so maybe we can cut down on at least one of the periodic poles, but I suspect that it is more likely the one that is loading and unloading contacts. I'll do some digging
<!-- github-comment-id:4033884119 -->
The format just doesn't match at all what's printed on the console - the format there is more like LEVEL:NAME:MESSAGE
Maybe there are, just not on console.... are there other log sinks available?
On the other hand this looks to me like the timestamp is supposed to be included in each log message: https://github.com/jkingsman/Remote-Terminal-for-MeshCore/blob/9421c10e8f1a9763d2009ea75c33b29623f3debe/app/config.py#L89
The format just doesn't match at all what's printed on the console - the format there is more like LEVEL:NAME:MESSAGE
<!-- github-comment-id:4033921438 -->
Sorry haha I dropped a word there and meant to say we definitely NEED to timestamp on our logs ahhh apologies -- what I said was incorrect and I blame voice to text. That's due to uvicorn eating our log config and it's fixed now.
I have added timestamps on my lunch break, but also throttled and streamlined our background tasks by a LOT (missed some low hanging perf fruit there where we did some reconciliation tasks synchronously) as well as turning off a task that was default but only served to stopgap missed radio events (which is very rare; that flow is enabled now only via MESHCORE_ENABLE_MESSAGE_POLL_FALLBACK=true per https://github.com/jkingsman/Remote-Terminal-for-MeshCore?tab=readme-ov-file#configuration. I suspect the main issue was the synchronous work happening during radio polling for new contacts, which blocks radio operations while it's running (~20s on my system). I've also reduced that full sync frequency back to every ten minutes now; it's an unfortunate reality as we store more contacts than the radio can hold so periodically we need to flush and write back the favorites/recently messages people/adverts/etc. If you still have issues, I'll work on making that be smarter about when it does that.
What transport are you using? Serial, BLE, or TCP?
Let me know how the latest main goes for you.
Sorry haha I dropped a word there and meant to say we definitely NEED to timestamp on our logs ahhh apologies -- what I said was incorrect and I blame voice to text. That's due to uvicorn eating our log config and it's fixed now.
I have added timestamps on my lunch break, but also throttled and streamlined our background tasks by a LOT (missed some low hanging perf fruit there where we did some reconciliation tasks synchronously) as well as turning off a task that was default but only served to stopgap missed radio events (which is very rare; that flow is enabled now only via `MESHCORE_ENABLE_MESSAGE_POLL_FALLBACK=true` per https://github.com/jkingsman/Remote-Terminal-for-MeshCore?tab=readme-ov-file#configuration. I suspect the main issue was the synchronous work happening during radio polling for new contacts, which blocks radio operations while it's running (~20s on my system). I've also reduced that full sync frequency back to every ten minutes now; it's an unfortunate reality as we store more contacts than the radio can hold so periodically we need to flush and write back the favorites/recently messages people/adverts/etc. If you still have issues, I'll work on making that be smarter about when it does that.
What transport are you using? Serial, BLE, or TCP?
Let me know how the latest `main` goes for you.
<!-- github-comment-id:4034542517 -->
Didn't misread your post - my brain has some means of automatic error correction so I even didnt notice 😁
I'm using USB, just installed latest main, lets see....
Didn't misread your post - my brain has some means of automatic error correction so I even didnt notice 😁
I'm using USB, just installed latest main, lets see....
<!-- github-comment-id:4037813249 -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
From time to time, UI will freeze when I hit send. It's a Heltec v4 node connected via USB to a Raspberry Pi.
Is this known behavior?
Or is there anything I can do to help debugging?
I have the impression that it may be related to ongoing radiosync in the background.
Maybe related: I also have a simple ping bot running (but it's not very busy and UI may freeze also when bot is 100% idle).
I'm currently on v2.7.9 #463a0c9 but it happened with earlier versions too.
It could be. Does the message eventually send? And when you say that the front end freezes, do you mean just the message sending section? Or does the whole frontend become unresponsive? Is there anything in the backend logs that correlates with when you see the freezes?
Radio operations will wait for the sink to finish; on my T114 that takes 5 or 6 seconds. I can cannibalize one of my Heltec ESP32 radios and see if I can replicate, but that won't be until later this week
Yes, it does send. Eventually. But may take more than 5 seconds..... 20..30s sometimes.
And yes: it's the send section that freezes, everything else stays responsive.
Nothing special to be found in the log, except for ongoing radiosync. But there's no timestamp, at least not on the console, so it's hard to correlate.
Oh dang we definitely timestamps on our logs. Tonight or tomorrow I'll add timestamps to the logs, see if I can replicate, and go from there. One of the background tests is a safety mechanism if we can't export the private key from the radio; I've been meaning to gate that task on the key export, so maybe we can cut down on at least one of the periodic poles, but I suspect that it is more likely the one that is loading and unloading contacts. I'll do some digging
Maybe there are, just not on console.... are there other log sinks available?
On the other hand this looks to me like the timestamp is supposed to be included in each log message: https://github.com/jkingsman/Remote-Terminal-for-MeshCore/blob/9421c10e8f1a9763d2009ea75c33b29623f3debe/app/config.py#L89
The format just doesn't match at all what's printed on the console - the format there is more like LEVEL:NAME:MESSAGE
Sorry haha I dropped a word there and meant to say we definitely NEED to timestamp on our logs ahhh apologies -- what I said was incorrect and I blame voice to text. That's due to uvicorn eating our log config and it's fixed now.
I have added timestamps on my lunch break, but also throttled and streamlined our background tasks by a LOT (missed some low hanging perf fruit there where we did some reconciliation tasks synchronously) as well as turning off a task that was default but only served to stopgap missed radio events (which is very rare; that flow is enabled now only via
MESHCORE_ENABLE_MESSAGE_POLL_FALLBACK=trueper https://github.com/jkingsman/Remote-Terminal-for-MeshCore?tab=readme-ov-file#configuration. I suspect the main issue was the synchronous work happening during radio polling for new contacts, which blocks radio operations while it's running (~20s on my system). I've also reduced that full sync frequency back to every ten minutes now; it's an unfortunate reality as we store more contacts than the radio can hold so periodically we need to flush and write back the favorites/recently messages people/adverts/etc. If you still have issues, I'll work on making that be smarter about when it does that.What transport are you using? Serial, BLE, or TCP?
Let me know how the latest
maingoes for you.Didn't misread your post - my brain has some means of automatic error correction so I even didnt notice 😁
I'm using USB, just installed latest main, lets see....
Gonna close this as patched -- feel free to reopen or create a new issue if you're still having problems.
None so far, thanks a lot!