mirror of
https://github.com/richonguzman/LoRa_APRS_iGate.git
synced 2026-10-31 02:51:58 +01:00
Public syslog server + signal tracking web page / API #133
Assignees
adam (Adam Wojcieszonek)
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: adam/LoRa_APRS_iGate#133
Reference in new issue
Block a user
Heads up that I've created a public syslog server which can ingest logs from an iGate. It lets the user view their own logs as well as plots data on maps - including signal strength etc. This is purely using the syslog output messages (does not have anything to do with aprs.fi !).
There's a rudimentary API and GeoJSON endpoint in case that's useful.
The website is: https://lora.link9.net/
Maybe you want to set 'lora.link9.net' port '1514' as the default for syslog settings (or the placeholder text?) - I don't mind if it was even enabled by default but that's up to you!
Any questions /concerns / comments please ask!
Thank you so much, i'll give it a try
There is something strange in the reports, i'm used to get long distance contacts, even more than the ones i see in your map, but this time these contacts are not mine, they aren't in the list of "Stations heard directly by" than you can read in aprs.fi website. Your site assign to me some packets digipeated by IQ5VK-11, that is the local club igate-digipeater. Ok, maybe i've igated some packets first heard by IQ5VK-11, but they can't be stated as i've heard directly it. If you want you can have a look at my reports for F5ZVD-4 , F1ZLL-4, 3A2ARM-10. Another strange thing is that in the "Stations who heard directly on radio" list of these callsign, you can't find IU5HKU-10 nor IQ5VK-11. I don't understand this behaviour. For the moment i disconnect my igate from your site, i don't wanna pollute the db with "strange" data, let me know. Anyway, i like the whole things, i find your service very useful.
Thanks for testing it out. I knew there would be a few bugs and you found one!
Should be fixed now if you wouldn't mind enabling it again?
Sure, the syslog is just enabled now, i'll keep it under control in the next hours/days, thank you for this great service :-)
I see your traffic again - shall keep an eye on it.
Of course let me know if you are seeing too few or too many than you expect. I removed the problematic ones and hopefully my logic is right.
It will now ignore Digipeated packets as that isn't really the point of the service (maps etc) so I think this is sensible :-)
this is great for testing !!! nice!
maybe this could be created as a Repository link with info and such and I could link to it in the readme and wiki ?
Please consider the way APRS-IS is working - if one iGates receives a station it sends this info to APRS-IS and this is then displayed on the web - if multible iGates receive a station a filtering starts at the APRS-IS servers and only one (!) receiving iGate is displayed on the web. Usually the "fastest" one is displayed.
Now with syslog you see all (!) iGates which have received a single frame.
So should be pretty ok.
Yes, for sure, but after the fix madpsy did, seems that infos are more suitable to the "real" situation, you can't consider a second-hop packet as directly heard by you...but now it works ok, stats are correct. Well done.
Can you add a button to collapse the left side of the screen (live logs) and make the map full size?
@IU5HKU Glad the fix worked - you are still top for distance :-)
@richonguzman It doesn't have a public code repo but there's an open API (https://lora.link9.net/api/) and a help page.
I was thinking you could make it the default host/port values for syslog settings (but off by default) and maybe a link to the site for convenience. Or... just link to the site in the readme/wiki and people can manually configure it if they want. Whatever you think is best for how you prefer to do things - I appreciate we haven't spoken before!
@S57PNX should be there now if you refresh
Thanks, looks very nice - but I can see the button in Chrome while I don't see it in Safari?!
I've added one of my gates (S58W-13) to your syslog to test, but I find some unexpected results - as if somehow digipeated trackers show as directly heard by my gate.
E.g. S54Z-11 shows as being received by S58W-13, but I am sure it was digipeated by S55YLI-10. All these suspect cases show
while the real direct contacts show this:
Do your spots look better now?
Safari is a funny animal. Does it show if you maximise the window?
Yes, my spots look correct now, thank you. And the button has appeared in Safari too!
One more nitpick: in the pop-up bubble, the Heard: timestamp is off by one hour for me. A station that was just heard for the first time shows as Heard: 1h 0m. I am in GMT+1 observing DST, so effectively GMT+2.
Can you add "ago" at the end of 1h 0m ?
I knew that would happen - initially my intent for this was just for a UK club so didn't take other TZ's into account. Now I'll have to rethink a few things. Timezones are always an 'interesting' problem.
Have added 'ago' again (was there before but removed it - will see how it looks)
Quick update. I've added an unauthenticated MQTT feed which might be useful to some folks.
Demo: https://lora.link9.net/mqtt-demo.html
Hi madspy, there is something strange again in parsing the syslog, look:
HB4LO-9>APLRT1,IQ5VK-4*:!/6tx9P4UXO`GQHB4LO tracker Bat=4.05V (89%)
this is a packet digipeated by IQ5VK-4 and heard by my igate, IU5HKU-10. Your stats assign the contact with HB4LO-9 to my igate. Strange enough i can't find HB4LO-9 into IQ5VK-4 list of "directly heard stations" on aprs.fi, i can't say nothing about IQ5VK-4 syslog, i haven't access to it.
Seems I committed a change and it was behind the main branch somehow so added my changes but reverted my previous commit to fix that!
Should be fixed now, for the second time
(I'm reading all of this, will return from Country Holidays on Monday to be online :P )
Another funny gltch - S58K-10 is plotted in a completely wrong spot, maybe 80km from its true location.
Correct plot from aprs.fi:
Wrong plot on lora.link9.net:
@S57PNX If I look now it seems correct, right? Looking back in the logs its locations hasn't changed at all so how it managed to plot there is a mystery just now!
Let me know if you see it again though as that's a strange one.
I think it was a bug with the
convertCoordinates()function which was needed previously but not now. You may need to hard refresh the page to get the new JavaScript code.If you can still make it break please use the 'Link' button at the top and paste the full link in here for me, thanks!
@madpsy thank you, it is looking correct now. I don't know how it happened to be, I just noticed it one day :)
Out of curiousity, what kind of resources do you have to run this database/web site? Will it handle an influx of new users?
Just now it's running on a small VM (1 CPU / 4GB RAM) but is under no load:
load average: 0.05, 0.01, 0.01If it becomes a problem can just throw more resource at it easily enough 🙂 Just now the most resource intensive page is the 'Top 10' page. If that starts becoming slow I can add some caching in various places so not overly concerned.
The MQTT page (now at http://lora-aprs.live) doesn't create any load as the processing is done on the client side. That is very much just a 'demo' for now as has a few known issues.
Have added SNR / distance / coverage heatmap for iGate owners, it's under the 'Coverage' link on the main page.
Example: https://lora.link9.net/coverage.html?rx_callsign=MB7UWI-10&time_period=48
would it be possible to have the possibility to configure two syslog servers in the configuration, so we can have a shared public one, and a personal local one?
I would also love to have configuration for 2 syslog servers.
73 John GW4BVE
I guess I can add the second syslog posibility
but dont you think this would go on slowing more and more igate process?
can one syslog be linked into another?
@richonguzman
1-)but dont you think this would go on slowing more and more igate process?(
-- I honesty do not know what is the impact of sending log to syslog. You may valuate that based on the various task. If this took time and resources, obviously this function can be omitted. Better give priority to the original tasks.
2-)can one syslog be linked into another?(
-- I couldn't find an easy way to do that, some software are available but need installation and the data forwarding is not their primary function. Standard Syslog servers usually do not have that capability.
usually syslog servers are more powerful than a ESP32 board this is why I asked ;)
If you have a dedicated sw running on a Server, yes, but at home we usually use a light app running on a NAS (in my case) that do not have any features other than recording data.
Just FYI if you're sending syslogs to lora.link9.net there's a 'firehose' MQTT (JSON format) feed to read them back out again, example:
mosquitto_sub -h hydros.link9.net -p 8883 --capath /etc/ssl/certs/ -t lora_aprs/MB7UNH-10/logs | jqThat will show all logs for iGate MB7UNH-10. You can do all the usual MQTT filtering using + and # etc and the structure should be clear from the page below. I wrote a demo of a pure JS implementation using leaflet which only uses this feed to create a live map.
More info here: https://lora-aprs.live/mqtt/
The problem with one syslog server forwarding to another server, or alternatively having a piece of software to split traffic to two syslog servers, is introducing a single point of failure. If this server or splitter fails, you lose both syslogs.
This is why I would prefer to split at the source on ESP32, and have a possiblity to send to two syslogs. It's somewhere on my todo list, but won't have any time to work on my todo for multiple weeks.
@S57PNX Date/times in general should now be fixed throughout
Yes, time works ok now, tnx. And I like the new improvements (Moving etc.) However, one of my use cases for your web page is to track the perfarmance/behaviour/health of my gate(s), so I am curious which stations it is picking up. Can you add a filter to the map view to display only stations heard by a single gate? Like you can select the gate in the Coverage view.
Currently, a tracker will "jump" between two gates that heard it, depending on which gate heard it last. I would rather see that "my" gate heard it 10 minutes ago, with an associated path and SNR etc. displayed, than some other gate heard it just now.
Does this view not do that?
https://lora-aprs.live/igate.html?rx_callsign=S58W-13&time_period=20
or this view?
https://lora-aprs.live/history.html?rx_callsign=S58W-13&tx_callsign=S57PNX-14
or do you mean something else?
Oh yes, the first view ( https://lora-aprs.live/igate.html?rx_callsign=S58W-13&time_period=20 ) does exactly what I wanted, thanks. I just didn't realize that I needed to click on S58W-13 in the logs or on the maps to get this info. :)
Good stuff! Yes, frontend design (UX etc) is not my strong point, as you can tell :-)
Thanks for this madspy, it is a really useful tool. I thought I had a problem, but realised it was a beacon not showing on APRS but was in the log as Direct RX and so deleted that comment :-)
Thanks Paul
MW0PDV
Glad to hear - if you do notice anything wrong obviously let me know though - it's quite a complicated beast now with many opportunities for bugs!
Have made a 'beta' frontend page for tracking elevation vs SNR vs iGate distance. Uses the public 'track' API endpoint so all of this is purely frontend code you can see via 'view source' and feel free to improve! Unsure if it's interesting but I thought it was kind of cool to see. Change tx_callsign to whatever you want. Example:
https://lora-aprs.live/elevation.html?tx_callsign=WA4OSH-1
'points' can be changed via the +/- buttons
@madpsy I noticed that you added Countries to your page - I assume that you mean the country of the tracker (TX) location? It seems that you define the country by coordinates, not by callsign.
S59HIJ-7 is located in Slovenia, not Croatia. Whereas S52ZO-13 indeeds sits in Croatia :)
And when it comes to iGate Unique trackers, S58W-13 is the only gate listed for Croatia, even though it is not located in Croatia.
@S57PNX Yeah it's based on the origin of the packet i.e. the Tracker. Have added a note at the top. I know it's a bit confusing so open to ideas!
@madpsy that's fine, but Tracker S59HIJ-7 is geographically not in Croatia, neither is S58W-13 as an igate.
I suspect there is a glitch in translating coordinates to countries.
Ah ok - there's maybe a bordering issue with the naturalearthdata.com dataset or a bug in my implementation. It's really difficult to get right as I've learned whilst doing this.
This uses the same location service which I wrote using that data - it's a lot more complicated to do than I originally thought!
https://lora-aprs.live/plotter.html
Yeah, the plotter has the same offset. S59HIJ-7 is in reality around 4km away from the Croatian border, yet your service shows it in Croatia. I did some spot checks around the country and there seems to be a random 0-4km offset from the real border. The nearest city feature works reasonably OK, so I simply believe that the border data is bad.
That should be a bit better now, though won't be perfect due to the resolution. Hopefully improved though.
@S57PNX The top lists are now by country of the iGate and hopefully the country accuracy is improved (also a new list for 'countries' on the top 10)
Thanks, the country accuracy is indeed improved. But I can still see up to 1km around the border attributed incorrectly to one or the other country. Doesn't hurt anything right now, but technically it is still not correct.
Out of curiosity, how do you do it? Do you have country borders as geographic coordinates and then you plot polygons with them?
Rather than talk endlessly about how it works (and bore everyone in the process) I'll put the country lookup service on my Github along with a test.html. Ff you want to look at / improve it then feel free. I am by no means an expert on the subject but was a fun learning experience.
@S57PNX https://github.com/madpsy/country-lookup
@madpsy its great page , I'm uploading data for a day and a half buy I see not digirepeated stations or others that are not with my callsign
Example:
CA2RXU-10 (igate) hears XQ3PCN-3 (digi) packet which is repeated by another digi (CA2RXU-11)
on packets listened and uploaded to syslog it shows the XQ3PCN-3 packet, but its not showing on map. Should it appear ?
@richonguzman A digipeated packet doesn't contain anything useful in terms of RF performance I don't think. It may contain its location but it wouldn't have any SNR/RSSI data so I discard it.
In your example I would have RF data for CA2RXU-11 but not XQ3PCN-3 but please do correct me if I'm wrong!
you are right, but why is not (only) position useful ?
My idea wasn't to re-create aprs.fi etc but to focus on the RF side of things, which none of those sites do. I could add purely positional data from digipeated packets if that's useful and make SNR/RSSI unknown or something.
Also, if your igate (CA2RXU-10) heard a direct digipeated packet it would have SNR/RSSI data but probably no positioning info in the packet. I wanted to be careful not to reinvent what existing ARPS sites already do.
well the digirepeater , although not hear directly, was heard being repeated over RF. (it may not have SNR / RSSI data but its still useful as its nearby stations heard by RF)
what do you think?
<165>1 - CA2RXU-10 CA2RXU_LoRa_iGate_1.3 - - - RX / GPS / XQ3PCN-3 / APLRG1,CA2RXU-11* / - / -59dBm / 7.50dB / -3591Hz / -32.97617N / -71.01334E / 52.7km / DIGI LORA CERRO EL ROBLEXQ3PCN-3 -> CA2RXU-11 -> CA2RXU-10 is the RF path
-32.97617N / -71.01334E is XQ3PCN-3's location and 52.7km is the distance between XQ3PCN-3 & CA2RXU-11
-59dBm / 7.50dB is the signal from CA2RXU-11 -> CA2RXU-10
is that right?
the distance es from XQ3PCN-3(digi) and CA2RXU-10 (igate): the distance is calculated from the GPS position of the station, not from the repeater (CA2RXU-11)
Got it, ok, given I have the last known location of CA2RXU-11 I can work out the distance between it and XQ3PCN-3 anyway.
It's a fair amount of changes to make that happen but will take a look this weekend. Have 7 (hopefully easy) bugs to fix first in the list.
Off topic really but how do I go about decoding the new encoded battery data? I couldn't make any sense of it! I have the old 'comment' based battery data working but the ARPS encoded one I gave up trying to decipher:
https://lora-aprs.live/battery.html
Base91 enconding : Link of pdf APRS12B
page 68 at the bottom
From what I remember in the syslog ouput they are wrapped in pipes '||' I presume the data is actually between them and the pipes are something you added
"|" + 2byteA + 2bytesB + "|"
"|" + 2byteA + "|"
"|" + 2bytesB + "|"
2bytesA could be internal battery voltage
2bytesB could be external voltage
watch out as this values depends on the 3 pre-Telemetry packets definitions, UNIT, PARM, EQNS
Thanks very much for the info
@madpsy can you do something about digis? Currently, I see them considered as trackers, whereas they belong more to the gates - or ideally even as a third class of the object. Not sure if syslog contains enough info to recognize digis as such though.
S55YAR-12 is a classic example of a pure digi. Yet under stats and on map it gets considered as a tracker.
and secondly, is there any heuristics to detect and remove impossibly long links? I can spot at least one 1000km+ link that in reality should be more like 10km, based on the geography.
Don't think I know the difference between a true 'tracker' and a digipeter. Regarding the distance issue, it's a strange one. Clearly the tracker has sent those co-ordinates for whatever reason. Just now it's limited to 2000 km. If you check this link you'll see any which that has caught already:
https://lora-aprs.live/logs.html?code=106&page=1
The question is what limit is sensible without excluding long distances from things like balloons etc.
I think gates and digis use Destination callsign APLRG1 while trackers use APLRT1, could that be a distinguisher? And if you want to break it down further, APLRG1, WIDE* is a digi and APRLRG1,TCPIP is a gate.
Yeah, the distance limit is a tricky one. I agree that anything over 2k is almost certainly a fluke, we regularly see valid links in the 600-900km range. I also agree that it's impossible to tell a fake 1000km distance from a real very good QSL. No idea how to fix it, your solution seems the best possible.
This is far more complicated than I thought. Preview with so many bugs:
https://lora-aprs.live/map2.html
It isn't correct but it is progress in terms of something digipeating related!
nice!
iGates appear as the Digi it repeated, not where it entered APRS-is /Syslog
but all looks good in "my place"
Yup, XQ3PCN-3 should be via CA2RXU-11, not -6
This side of things is all HTML/JS so can see the source for the page. I am by no means a frontend dev so is kind of painful if I'm honest!
got cha! I have to learn JS and HTML also for this firmware.... so keep up the good work 🥇
at least XQ3PCN-3 is showing now for you, although not via the correct path. Work in progress :sl
this is turning into a super good tool ;)
this will help to bring better networks of pure RF paths to cover cities
The goal was always to be interesting from an RF point of view rather than just location, hence why your SNR/RSSI syslogs are very useful. Digipeating is different though - I have no plans to go beyond one-hop for this
1 hop is like 99% of the repeated path for LoRa. So it will be fine ;)
In that case I will get the path right for 1-hop and call it done in that regard. Busy with work but hopefully by the weekend. Interest has been gathering pace:
iGates has grown 700 in 100 last days, Trackers 350 in same period.
Once this supports 1-hop digipeaters I was wondering if you wanted to make it the 'placeholder' for syslog text input field (not enabled by default) - depends on reliability etc for sure
sure thing!
My brain now hurts! If anyone fancies taking a quick look at:
https://lora-aprs.live/map2.html
Just to ensure it makes sense and vaguely correct that would be appreciated.
I underestimated how complex it would be supporting digipeaters but I think i'm along the right lines now.
Maybe @richonguzman and @S57PNX just if/when you have a few minutes spare?
Sorry I've been AFK for the last couple of days, I am just checking it now. At first glance it is a definite improvement, presuming color blue is the digi. But I see too much blue now - I suspect it is igates that operate as igate, but send a beacon over RF as well. That beacon gets picked up by another gate, which makes the originating gate blue.
I haven't really thought this over though and I don't see a good solution. @richonguzman any ideas?
@S57PNX Thanks for the feedback - do you have an example callsign which is showing as a digipeater (blue) but is actually an iGate?
It could just be the iGate isn't sending syslog to the site - in which case if it did then it should turn green and show as an iGate (just guessing until you give an example)
You are right, none of the blue gates are currently sending syslog to you - which explains it. I'll reconfigure a few over weekend to send syslogs and see if they become green.
I am curious about S59DEM-11 though. It's green but I am fairly certain it is not sending a syslog to your site (unless somebody went there and changed the config without me knowing).
Can you add a tickbox on top to filter on blue objects?
Found a bug which explains why S59DEM-11 was showing green. It looks good now (note the
object_type). Yeah I'll be adding new filters for digipeatersLooking great!
One more nitpick: ticking on Moving grays out igate and tracker tickboxs, but leaves them enabled (which was probably not the intention). I'd assume that Moving means only moving objects and igate and tracker need to be switched off? Or at least igate (dependoing on the background logic).
It was fetching the wrong endpoint for moving objects so didn't refresh the map - fixed now
@madpsy another feature request :) Can you add a toggle for displaying the station labels? Many stations in a small country make for a very congested map. It would be great if we could toggle off the labels, leave just the position dots and keep the mouseover menu.
@S57PNX Your network is clearly too big! Kidding of course. The main reason for the chaos there is because, unlike APRS-IS, I don't deduplicate the digipeated packets so you see every node involved. That's on purpose though.
Getting back to your actual request - have added a Minimal view. Looks a bit crap but hopefully should do for now.
I perfectly understand why we get to see every node involved, and we thank you for that - that's exactly the purpose of parsing the syslogs in comparison to APRS-IS based web pages. However, you ain't seen nothing yet, there's another 50 or so igates in our network yet to start sending data to you!
Thanks for the minimal view, looks much better.
Lightweight Python terminal application to view iGate logs in real time. Nothing fancy for a proof of concept, feel free to fork and do whatever you want with it :-)
https://github.com/madpsy/lora-aprs-python-client
Also a web equivalent:
https://lora-aprs.live/mqtt/stream.html?igate=S58W-13
Making them look/feel the same was more fun rather than a promise :-)
That's pretty impressive, thank you!
Linux and Windows binaries available:
https://github.com/madpsy/lora-aprs-python-client/releases
Hi @madpsy , I got a funny one for you... My friends were all excited this morning about an amazing performance of the gate S55DLM-10, doing 200+ km through a mountain range. Too good to be true. So I investigated and this is what I believe happend.
S57C-9 (the tracker on the other side of the mountain) has most llikely two trackers with the same SSID. One is at his home beaconing via RF and is picked by S55DLM-10 some 18km away. The other tracker is AprsDroid via TCPIP, 200+ km away. Clearly this creates confusion and should not be done. However, your map also gets confused and seems to take the RF report of the near tracker with the location of the far tracker, showing an impossibly long RF path.
Ah, I think I might have found the smoking gun:
This is the RF tracker at home that the gate is receiveng via RF:
And this is the AprsDroid tracker 200km away - but why is it logged by the gate and considered a Direct RX by your server?
I think the gate gets the aprsdroid report from aprs-is but because it was also seen locally via RF within the last 30 minutes, it will be syslogged and reported back to APRS-IS as "heard". @richonguzman , does this make sense?
A quick fix would be to filter out any packets with TCPIP* in the path from Direct RX category.
@S57PNX Nice find - indeed those should not be direct RX - have fixed that now!
Thank you, it looks better now and some other less obvious duplicates are gone too.
when syslog sends an aprsdroid packet (with APRSdroid TOCALL) it is received by LoRa through RF. TOCALLs are not changed when sended by tracker as tracker or TNC station.
User has same ssid on RF tracker at home and on aprsdroid 200km away. In this case the gate received an aprsdroid packet through APRS-IS and it was logged in syslog as received.
Why is the gate logging a packet with path TCPIP* in his syslog since it came from APRS-IS?
I think this happened because the same SSID was in RF range less than 30 minutes ago, so the gate had this call (with same SSID) in history and accepted it.
so it the "same callsign+ssid" error?
Yes, but as a result of two trackers with same callsign+ssid we get this in the gate log:
Should gate really log a packet that came from APRS-IS ? And why is there a RSSI and S/N on a packet that originated from aprsdroid and was sent directly to APRS-IS? I think the gate was confused because it saw an earlier packet from the duplicate tracker via RF and mixed both together.
iGate does not log (to syslog) packets listened from APRS-IS unless is a query-answer.
this may be a third party packet over RF.
It is almost impossible that this aprsdroid packet came over RF. Aprsdroid is behind a big mountain 200km away. But a real tracker with same call+ssid is 18km away and received normally in the syslog.
Hi, Great work. I have an issue thogh: Packets with timestamp are not accepted. Marked as Invalid GPS Type. The APRS Packet Decoder gets it correctly though.
Hi
This timestamp is faulty on my igate firmware step, right?
I totally missed to decode those packets with timestamp
Ah. Thanks. That may explain it :)
this was long ago solved, thanks @madpsy for the help