docs update

README.md

@@ -1,38 +1,277 @@
 # Meshtastic Bridge
 
-Connect two distinct radio [Meshtastic](https://meshtastic.org) networks using TCP.
+Connect [Meshtastic](https://meshtastic.org) radio networks using MQTT and HTTP.
 
-WARNING: Traffic is sent insecure using TCP. Use a VPN to secure traffic between the nodes
+WARNING: Work in progress
 
 ## Requirements
 
-- Python3
+- Command-line install
-- Two Meshtastic devices:   
+  - Python 3.8
-   - Local `LOCAL_NODE_ADDR` The IP address or Serial devPath (micro USB cable needed for serial access) of a local Meshtastic device
+  - git
-   - Remote `REMOTE_NODE_ADDR` The IP address of a remote Meshtastic device
+  - Outbound HTTPS (TCP/443) or SSH (TCP/22) access to github.com
+- Docker-based install
+  - Docker
+- Meshtastic radio device:
+  - The _IP address_ or Serial _devPath_ (micro USB cable needed for serial access) of a Meshtastic device
+- MQTT server:
+  - The _domain name_ of the server
+  - The _port_ (e.g. 1883)
 
-Refer to <https://meshtastic.org/docs/settings/config/wifi#wifi-client> to configure a Meshtastic device to use wifi and expose a TCP address
+Refer to <https://meshtastic.org/docs/settings/config/wifi#wifi-client> for details on how to configure a Meshtastic device to use wifi and expose a TCP address.
 
-## Setup
+## Command-line installation
 
-Run the following steps to download and install the software locally
+Download the code and install it onto a system:
 
 ```
 $ git clone https://github.com/geoffwhittington/meshtastic-bridge.git
+```
+
+Create a Python virtual environment
+
+```
 $ python3 -m venv meshtastic-bridge
+```
+
+Install the bridge dependencies
+
+```
 $ cd meshtastic-bridge
 $ source bin/activate
 $ pip install -r requirements.txt
 ```
 
-## Turn on the Bridge
+## Docker installation
 
-In the `meshtastic-bridge` directory run the following - replacing `BRIDGE_DISTANCE_KM`, `LOCAL_NODE_ADDR` and `REMOTE_NODE_ADDR` with the proper values:
+There is nothing to install with Docker, the bridge is downloaded at the time it is run
+
+## Configuration
+
+The bridge is configured using a YAML file `config.yaml`. It is composed of three sections, `devices`, `mqtt_servers` and `pipelines`.
+
+An example `config.yaml` is provided below:
 
 ```
-BRIDGE_DISTANCE_KM=0 LOCAL_NODE_ADDR=/dev/ttyUSB0 REMOTE_NODE_ADDR=182.168.86.123 python main.py
+devices:
+   - name: remote
+     tcp: 192.168.86.39
+     active: true
+mqtt_servers:
+   - name: external
+     server: broker.hivemq.com
+     port: 1883
+     topic: meshtastic/radio-network1
+     pipelines:
+       mqtt-to-radio:
+         - decrypt_filter:
+            key: '/home/user/keys/key.pem'
+         - send_plugin:
+            device: remote
+pipelines:
+ radio-to-mqtt:
+   - encrypt_filter:
+       key: '/home/user/keys/cert.pem'
+   - mqtt_plugin:
+       name: external
+       topic: mesh/tastic
 ```
 
-## Bridge Options
+`devices` is a list of radios the bridge listens for packets or to where it can send packets.
 
-* `BRIDGE_DISTANCE_KM` Do not bridge messages from nodes that more than BRIDGE_DISTANCE_KM kilometers from the local Meshtastic device. Default `0` (no limit)
+- **name** Reference given to a radio that is used elsewhere in the `pipelines` configuration. For example, `my_radio`
+- **tcp** The IP address of the radio. For example, `192.168.0.1` (Optional)
+- **serial** The name of the serial device attached to the radio. For example, `/dev/ttyUSB0` (Optional)
+- **active** Indicator whether this configuration is active. Values: `true` or `false`. Default = `true`.
+
+NOTE: If `tcp` or `serial` are not given the bridge will attempt to detect a radio attached to the serial port. Additional configuration may be needed to use the serial port with the Docker option.
+
+`mqtt_servers` is a list of MQTT servers the bridge listens for shared network traffic.
+
+- **name** Reference given to the MQTT server. For example, `my_mqtt_server`
+- **server** The IP address or hostname of a MQTT server. For example, `server.mqttserver.com`
+- **port** The port the MQTT server listens on
+- **topic** The topic name associated with the network traffic. For example, `mesh/network`
+- **insecure** Use a secure connection but do not validate the server certificate
+- **pipelines** A set of plugins (filters/actions) that run when a new message emerges for _topic_. Each pipeline is given a name; such as `mqtt-to-radio` (as in the example above)
+
+`pipelines` is a list of ordered plugins (filters/actions) that run when a packet is detected by any connected radio. Each set is given a name; such as `radio-to-mqtt` (as in the example above). Pipelines can run in any order, however plugins run in the order they are defined.
+
+## Plugins
+
+The following plugins can be used in the `pipelines` section of `config.yaml`:
+
+| Plugin            | Description                                                          |
+| ----------------- | -------------------------------------------------------------------- |
+| `debugger`        | Log the packet to the system console                                 |
+| `message_filter`  | Filters out packets from the bridge that match a specific criteria   |
+| `distance_filter` | Filters out packets that originate too far from a specified `device` |
+| `webhook`         | Send HTTP requests with custom payloads using packet information     |
+| `mqtt_plugin`     | Send packets to a MQTT server                                        |
+| `encrypt_filter`  | Encrypt a packet for a desired MQTT recipient                        |
+| `decrypt_filter`  | Decrypt a packet originating from MQTT                               |
+| `send_plugin`     | Send a packet to a specified `device`                                |
+
+### debugger - Output the contents of a packet
+
+- **log_level** `debug` or `info`. Default `info`
+
+For example:
+
+```
+debugger:
+  log_level: debug
+```
+
+Useful for troubleshooting.
+
+### message_filter - Allow or block packets based on criteria
+
+- **log_level** `debug` or `info`. Default `info`
+- **app** Name of meshtastic application to allow or disallow
+- **from** The packet `fromId` values to allow or disallow
+- **to** The packet `toId` values to allow or disallow
+
+For example:
+
+```
+message_filter:
+  app:
+     allow:
+        - !bd5ba0ec
+        - !f85bc0bc
+     disallow:
+        - !c15ba2ec
+```
+
+### distance_filter - Allow or block packets based on distance from origin to radio
+
+- **log_level** `debug` or `info`. Default `info`
+- **max_distance_km** Number of kilometers
+
+For example:
+
+```
+distance_filter:
+  max_distance_km: 1000
+```
+
+### webhook - Send a HTTP request
+
+- **log_level** `debug` or `info`. Default `info`
+- **active** Plugin is active. Values: `true` or `false`. Default = `true`.
+- **body** The JSON payload to send
+- **url** The target URL
+- **headers** HTTP headers to include in the request. Secrets can be passed using ENV variables
+- **message** Override the packet message
+
+Placeholders can be used with the **body** value:
+
+- `{LAT}` - Latitude associated with the POSITION packet. Empty if no value available.
+- `{LNG}` - Latitude associated with the POSITION packet. Empty if no value available.
+- `{MSG}` - Packet text or `message` from the configuration (above)
+- `{FID}` - The `fromId` associated with the packet.
+- `{TID}` - The `toId` associated with the packet.
+
+For example:
+
+```
+webhook:
+  active: true
+  body: '{"lat": "{LAT}", "lng": "{LNG}", "text_message": "{MSG}"}'
+  url: 'https://localhost:8000/message'
+  headers:
+     Authorization: Token {AUTH_TOKEN}
+     Content-type: application/json
+```
+
+### mqtt_plugin - Send a packet to a MQTT server
+
+- **log_level** `debug` or `info`. Default `info`
+- **active** Plugin is active. Values: `true` or `false`. Default = `true`.
+- **name** Reference of an existing MQTT server configured in the top-level `mqtt_servers` configuration
+- **message** Override the packet message with a custom value
+- **topic** The message topic
+
+For example:
+
+```
+mqtt_plugin:
+  name: external
+  topic: meshtastic/topic
+```
+
+### encrypt_filter - Encrypt a packet before sending it to a MQTT server
+
+- **log_level** `debug` or `info`. Default `info`
+- **key** The PEM filename of the public key used to encrypt the message.
+
+For example:
+
+```
+encrypt_filter:
+  key: '/home/user/keys/cert.pem'
+```
+
+### decrypt_filter - Decrypt message from a MQTT server
+
+- **log_level** `debug` or `info`. Default `info`
+- **key** The PEM filename of the key used to decrypt the message.
+
+For example:
+
+```
+decrypt_filter:
+  key: '/home/user/keys/key.pem'
+```
+
+### send_plugin - Send a packet to a radio
+
+- **log_level** `debug` or `info`. Default `info`
+- **active** Plugin is active. Values: `true` or `false`. Default `true`.
+- **device** Required. Send the packet to a Radio with this name. It should be configured in the top-level `devices` configuration
+- **message** Send a text message
+- **lat** Send a position message having this latitude
+- **lng** Send a position message having this longitude
+- **node_mapping** Map the packet `to` value to another value
+- **to** Target node reference
+- **toId** Target node reference. Ignored if `to` is used.
+
+For example:
+
+Broadcasts all packets to the "remote" radio network that are destined to the node `12354345`.
+
+```
+send_plugin:
+  device: remote
+  node_mapping:
+    12354345: ^all
+```
+
+## Run the bridge
+
+### Command-line
+
+Create a `config.yaml` in the `meshtastic-bridge` directory. Run:
+
+```
+$ source bin/activate
+```
+
+And:
+
+```
+python main.py
+```
+
+### Docker
+
+Create a `config.yaml` with the desired settings and run the following Docker command:
+
+```
+docker run -v $(pwd)/config.yaml:/code/config.yaml gwhittington/meshtastic-bridge:latest
+```
+
+## Resources
+
+- Example guidance for creating [PEM](https://www.suse.com/support/kb/doc/?id=000018152) key files.

main.py

@@ -1,3 +1,4 @@
+import json
 import logging
 import meshtastic
 import meshtastic.serial_interface
@@ -11,16 +12,64 @@ from plugins import plugins
 from pubsub import pub
 import yaml
 from yaml.loader import SafeLoader
+import paho.mqtt.client as mqtt
+
+logging.basicConfig()
 
 logger = logging.getLogger(name="meshtastic.bridge")
 logger.setLevel(logging.DEBUG)
 
+
+def onReceive(packet, interface):  # called when a packet arrives
+
+    for pipeline, pipeline_plugins in bridge_config["pipelines"].items():
+        logger.debug(f"Pipeline {pipeline} initiated")
+
+        p = plugins["packet_filter"]
+        pipeline_packet = p.do_action(packet)
+
+        for plugin in pipeline_plugins:
+            for plugin_key, plugin_config in plugin.items():
+
+                logger.debug(f"Processing plugin: {pipeline}/{plugin_key}")
+                if not pipeline_packet:
+                    logger.debug("Skipping since the packet is null")
+                    continue
+
+                if plugin_key not in plugins:
+                    logger.error(f"No such plugin: {plugin_key}. Skipping")
+                    continue
+
+                p = plugins[plugin_key]
+                p.configure(devices, mqtt_servers, plugin_config)
+
+                pipeline_packet = p.do_action(pipeline_packet)
+
+        logger.debug(f"Pipeline {pipeline} completed")
+
+
+def onConnection(
+    interface, topic=pub.AUTO_TOPIC
+):  # called when we (re)connect to the radio
+    nodeInfo = interface.getMyNodeInfo()
+    logger.info(
+        f"Connected to node: userId={nodeInfo['user']['id']} hwModel={nodeInfo['user']['hwModel']}"
+    )
+
+
+pub.subscribe(onReceive, "meshtastic.receive")
+pub.subscribe(onConnection, "meshtastic.connection.established")
+
 with open("config.yaml") as f:
     bridge_config = yaml.load(f, Loader=SafeLoader)
 
 devices = {}
+mqtt_servers = {}
 
 for device in bridge_config["devices"]:
+    if "active" in device and not device["active"]:
+        continue
+
     if "serial" in device:
         devices[device["name"]] = meshtastic.serial_interface.SerialInterface(
             devPath=device["serial"]
@@ -32,42 +81,93 @@ for device in bridge_config["devices"]:
     else:
         devices[device["name"]] = meshtastic.serial_interface.SerialInterface()
 
+for config in bridge_config["mqtt_servers"]:
+    required_options = [
+        "name",
+        "server",
+        "port",
+    ]
 
-def onReceive(packet, interface):  # called when a packet arrives
+    for option in required_options:
-    for pipeline in bridge_config["pipelines"]:
+        if option not in config:
+            logger.warning("Missing config: {option}")
 
-        pipeline_packet = packet
+    client_id = config["client_id"] if "client_id" in config else None
+    username = config["username"] if "username" in config else None
+    password = config["password"] if "password" in config else None
 
-        for key, config in pipeline.items():
+    if client_id:
+        mqttc = mqtt.Client(client_id)
+    else:
+        mqttc = mqtt.Client()
 
-            if not pipeline_packet:
+    if username and password:
+        mqttc.username_pw_set(username, password)
+
+    mqtt_servers[config["name"]] = mqttc
+
+    def on_connect(mqttc, obj, flags, rc):
+        logger.debug(f"Connected to MQTT {config['name']}")
+
+    def on_message(mqttc, obj, msg):
+        packet = msg.payload.decode()
+
+        logger.debug(f"MQTT {config['name']}: on_message")
+
+        if "pipelines" not in config:
+            logger.warning(f"MQTT {config['name']}: no pipeline")
+            return
+
+        for pipeline, pipeline_plugins in config["pipelines"].items():
+
+            logger.debug(f"MQTT {config['name']} pipeline {pipeline} started")
+            if not packet:
                 continue
 
-            if key not in plugins:
+            for plugin in pipeline_plugins:
-                logger.error(f"No such plugin: {key}. Skipping")
+                for plugin_key, plugin_config in plugin.items():
-                continue
+                    if plugin_key not in plugins:
+                        logger.error(f"No such plugin: {plugin_key}. Skipping")
+                        continue
 
-            p = plugins[key]
+                    p = plugins[plugin_key]
-            p.configure(devices, config)
+                    p.configure(devices, mqtt_servers, plugin_config)
 
-            pipeline_packet = p.do_action(pipeline_packet)
+                    try:
+                        packet = p.do_action(packet)
+                    except Exception as e:
+                        logger.error(f"Hit an error: {e}", exc_info=True)
+            logger.debug(f"MQTT {config['name']} pipeline {pipeline} finished")
 
+    def on_publish(mqttc, obj, mid):
+        logger.debug(f"MQTT {config['name']}: on_publish: {mid}")
 
-def onConnection(
+    def on_subscribe(mqttc, obj, mid, granted_qos):
-    interface, topic=pub.AUTO_TOPIC
+        logger.debug(f"MQTT {config['name']}: on_subscribe: {mid}")
-):  # called when we (re)connect to the radio
-    nodeInfo = interface.getMyNodeInfo()
 
-    logger.info(
+    mqttc.on_message = on_message
-        f"Connected to node: userId={nodeInfo['user']['id']} hwModel={nodeInfo['user']['hwModel']}"
+    mqttc.on_connect = on_connect
-    )
+    mqttc.on_publish = on_publish
+    mqttc.on_subscribe = on_subscribe
 
+    import ssl
 
-pub.subscribe(onReceive, "meshtastic.receive")
+    if "insecure" in config and config["insecure"]:
-pub.subscribe(onConnection, "meshtastic.connection.established")
+        mqttc.tls_set(cert_reqs=ssl.CERT_NONE)
+        mqttc.tls_insecure_set(True)
+
+    mqttc.connect(config["server"], config["port"], 60)
+
+    if "topic" in config:
+        mqttc.subscribe(config["topic"], 0)
+
+    mqttc.loop_start()
 
 while True:
     time.sleep(1000)
 
 for device, instance in devices.items():
     instance.close()
+
+for server, instance in mqtt_servers.items():
+    instance.disconnect()

plugins.py

@@ -1,18 +1,22 @@
 from haversine import haversine
 from meshtastic import mesh_pb2
 from meshtastic.__init__ import BROADCAST_ADDR
+import base64
+import json
 import logging
 import os
 
+
 plugins = {}
 
 
 class Plugin:
-    def configure(self, devices, config):
+    def configure(self, devices, mqtt_servers, config):
         self.config = config
         self.devices = devices
+        self.mqtt_servers = mqtt_servers
 
-        if "log_level" in config:
+        if config and "log_level" in config:
             if config["log_level"] == "debug":
                 self.logger.setLevel(logging.DEBUG)
             elif config["log_level"] == "info":
@@ -22,13 +26,39 @@ class Plugin:
         pass
 
 
+class PacketFilter(Plugin):
+    logger = logging.getLogger(name="meshtastic.bridge.filter.packet")
+
+    def strip_raw(self, dict_obj):
+        if type(dict_obj) is not dict:
+            return dict_obj
+
+        if "raw" in dict_obj:
+            del dict_obj["raw"]
+
+        for k, v in dict_obj.items():
+            dict_obj[k] = self.strip_raw(v)
+
+        return dict_obj
+
+    def do_action(self, packet):
+        packet = self.strip_raw(packet)
+
+        if "decoded" in packet and "payload" in packet["decoded"]:
+            packet["decoded"]["payload"] = base64.b64encode(
+                packet["decoded"]["payload"]
+            ).decode("utf-8")
+
+        return packet
+
+
+plugins["packet_filter"] = PacketFilter()
+
+
 class DebugFilter(Plugin):
     logger = logging.getLogger(name="meshtastic.bridge.plugin.logging")
 
     def do_action(self, packet):
-        self.logger.info(
-            f"{packet['id']} | {packet['fromId']}=>{packet['toId']} | {packet['decoded']['portnum']}"
-        )
         self.logger.debug(packet)
         return packet
 
@@ -80,7 +110,7 @@ class DistanceFilter(Plugin):
     logger = logging.getLogger(name="meshtastic.bridge.filter.distance")
 
     def do_action(self, packet):
-        if 'device' not in self.config:
+        if "device" not in self.config:
             return packet
 
         if "position" not in packet["decoded"]:
@@ -129,19 +159,31 @@ class WebhookPlugin(Plugin):
     logger = logging.getLogger(name="meshtastic.bridge.plugin.webhook")
 
     def do_action(self, packet):
+        if type(packet) is not dict:
+            try:
+                packet = json.loads(packet)
+            except:
+                self.logger.warning("Packet is not dict")
+                return packet
+
         if "active" in self.config and not self.config["active"]:
             return packet
 
-        if "position" not in packet["decoded"]:
+        if "body" not in self.config:
+            self.logger.warning("Missing config: body")
             return packet
 
-        import json
         import requests
 
+        position = (
+            packet["decoded"]["position"] if "position" in packet["decoded"] else None
+        )
+        text = packet["decoded"]["text"] if "text" in packet["decoded"] else None
+
         macros = {
-            "{LAT}": packet["decoded"]["position"]["latitude"],
+            "{LAT}": position["latitude"] if position else None,
-            "{LNG}": packet["decoded"]["position"]["longitude"],
+            "{LNG}": position["longitude"] if position else None,
-            "{MSG}": self.config["message"] if "message" in self.config else "",
+            "{MSG}": self.config["message"] if "message" in self.config else text,
             "{FID}": packet["fromId"],
             "{TID}": packet["toId"],
         }
@@ -176,16 +218,116 @@ class WebhookPlugin(Plugin):
 plugins["webhook"] = WebhookPlugin()
 
 
+class MQTTPlugin(Plugin):
+    logger = logging.getLogger(name="meshtastic.bridge.plugin.mqtt")
+
+    def do_action(self, packet):
+        required_options = ["name", "topic"]
+
+        for option in required_options:
+            if option not in self.config:
+                self.logger.warning(f"Missing config: {option}")
+                return packet
+
+        if self.config["name"] not in self.mqtt_servers:
+            self.logger.warning(f"No server established: {self.config['name']}")
+            return packet
+
+        mqtt_server = self.mqtt_servers[self.config["name"]]
+
+        packet_payload = packet if type(packet) is str else json.dumps(packet)
+
+        message = self.config["message"] if "message" in self.config else packet_payload
+
+        info = mqtt_server.publish(self.config["topic"], message)
+        info.wait_for_publish()
+
+        self.logger.debug("Message sent")
+
+
+plugins["mqtt_plugin"] = MQTTPlugin()
+
+
+class EncryptFilter(Plugin):
+    logger = logging.getLogger(name="meshtastic.bridge.filter.encrypt")
+
+    def do_action(self, packet):
+
+        if "key" not in self.config:
+            return None
+
+        from jwcrypto import jwk, jwe
+        from jwcrypto.common import json_encode, json_decode
+
+        with open(self.config["key"], "rb") as pemfile:
+            encrypt_key = jwk.JWK.from_pem(pemfile.read())
+
+        public_key = jwk.JWK()
+        public_key.import_key(**json_decode(encrypt_key.export_public()))
+        protected_header = {
+            "alg": "RSA-OAEP-256",
+            "enc": "A256CBC-HS512",
+            "typ": "JWE",
+            "kid": public_key.thumbprint(),
+        }
+
+        message = json.dumps(packet)
+
+        jwetoken = jwe.JWE(
+            message.encode("utf-8"), recipient=public_key, protected=protected_header
+        )
+
+        self.logger.debug(f"Encrypted message: {packet['id']}")
+        return jwetoken.serialize()
+
+
+plugins["encrypt_filter"] = EncryptFilter()
+
+
+class DecryptFilter(Plugin):
+    logger = logging.getLogger(name="meshtastic.bridge.filter.decrypt")
+
+    def do_action(self, packet):
+        if "key" not in self.config:
+            return packet
+
+        if type(packet) is not str:
+            self.logger.warning(f"Packet is not string")
+            return packet
+
+        from jwcrypto import jwk, jwe
+
+        with open(self.config["key"], "rb") as pemfile:
+            private_key = jwk.JWK.from_pem(pemfile.read())
+
+        jwetoken = jwe.JWE()
+        jwetoken.deserialize(packet, key=private_key)
+        payload = jwetoken.payload
+        packet = json.loads(payload)
+        self.logger.debug(f"Decrypted message: {packet['id']}")
+        return packet
+
+
+plugins["decrypt_filter"] = DecryptFilter()
+
+
 class SendPlugin(Plugin):
     logger = logging.getLogger(name="meshtastic.bridge.plugin.send")
 
     def do_action(self, packet):
 
+        if type(packet) is not dict:
+            try:
+                packet = json.loads(packet)
+            except:
+                self.logger.error("Packet is not a dict")
+                return packet
+
         if self.config["device"] not in self.devices:
             self.logger.error(f"Missing interface for device {self.config['device']}")
             return packet
 
-        if "to" not in packet:
+        if "to" not in packet and "toId" not in packet:
             self.logger.debug("Not a message")
             return packet
 
@@ -196,15 +338,22 @@ class SendPlugin(Plugin):
         ):
             destinationId = self.config["node_mapping"][packet["to"]]
         else:
-            destinationId = packet["to"]
+            destinationId = packet["to"] if "to" in packet else packet["toId"]
 
         if "to" in self.config:
             destinationId = self.config["to"]
+        elif "toId" in self.config:
+            destinationId = self.config["toId"]
 
         device_name = self.config["device"]
+
+        if device_name not in self.devices:
+            self.logger.warning(f"No such radio device: {device_name}")
+            return packet
+
         device = self.devices[device_name]
 
-        self.logger.debug(f"Sending packet to {device_name}")
+        self.logger.debug(f"Sending packet to Radio {device_name}")
 
         if "message" in self.config and self.config["message"]:
             device.sendText(text=self.config["message"], destinationId=destinationId)
@@ -227,14 +376,11 @@ class SendPlugin(Plugin):
         else:
             meshPacket = mesh_pb2.MeshPacket()
             meshPacket.channel = 0
-            meshPacket.decoded.payload = packet["decoded"]["payload"]
+            meshPacket.decoded.payload = base64.b64decode(packet["decoded"]["payload"])
             meshPacket.decoded.portnum = packet["decoded"]["portnum"]
             meshPacket.decoded.want_response = False
             meshPacket.id = device._generatePacketId()
 
-            self.logger.debug(
-                f"Sending packet {meshPacket.id} to {self.config['device']}"
-            )
             device._sendPacket(meshPacket=meshPacket, destinationId=destinationId)
 
         return packet

requirements.txt

@@ -2,3 +2,5 @@ haversine
 meshtastic
 requests
 pyyaml
+paho-mqtt
+jwcrypto