docs: Restructure and expand project documentation, add contribution guidelines and license, and update reporting dependencies.

.gitignore

@@ -3,6 +3,12 @@ __pycache__/
 *.py[cod]
 *$py.class
 
+# Distribution / Packaging
+build/
+dist/
+*.egg-info/
+.eggs/
+
 # Environments
 .env
 .venv

CONTRIBUTING.md

@@ -0,0 +1,40 @@
+# Contributing to LoRa Mesh Analyzer
+
+Thank you for your interest in contributing to the LoRa Mesh Analyzer! We welcome contributions from the community.
+
+## How to Contribute
+
+1.  **Fork the repository**: Click the "Fork" button on the top right of the repository page.
+2.  **Clone your fork**:
+    ```bash
+    git clone https://github.com/YOUR_USERNAME/LoRa-Mesh-Analyzer.git
+    cd LoRa-Mesh-Analyzer
+    ```
+3.  **Create a branch**:
+    ```bash
+    git checkout -b feature/my-new-feature
+    ```
+4.  **Make your changes**: Implement your feature or fix.
+5.  **Run tests**: Ensure all tests pass.
+    ```bash
+    pytest
+    ```
+6.  **Commit your changes**:
+    ```bash
+    git commit -am 'Add some feature'
+    ```
+7.  **Push to the branch**:
+    ```bash
+    git push origin feature/my-new-feature
+    ```
+8.  **Submit a Pull Request**: Open a PR on the main repository.
+
+## Coding Standards
+
+-   Follow PEP 8 style guidelines for Python code.
+-   Ensure code is well-documented.
+-   Add tests for new features.
+
+## Reporting Issues
+
+If you find a bug or have a feature request, please open an issue on the GitHub repository.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 EddieOz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -2,169 +2,47 @@
 
 An autonomous Python application designed to monitor, test, and diagnose the health of a Meshtastic mesh network. It identifies "toxic" behaviors, congestion, and configuration issues that can degrade network performance.
 
-## Features
+## Documentation
 
-The monitor runs a continuous loop (every 60 seconds) and performs the following checks:
+Full documentation is available in the `docs/` directory:
 
-### 1. Passive Health Checks
+-   **[Usage Guide](docs/usage.md)**: How to run the monitor (USB/TCP) and command-line options.
-*   **Congestion Detection**: Flags nodes reporting a Channel Utilization (`ChUtil`) > **25%**. High utilization leads to packet collisions and mesh instability.
+-   **[Configuration Guide](docs/configuration.md)**: Detailed explanation of `config.yaml` settings.
-*   **Spam Detection**: 
+-   **[Report Generation](docs/report_generation.md)**: How to use the `report_generate.py` tool.
-    *   **Airtime**: Flags nodes with an Airtime Transmit Duty Cycle (`AirUtilTx`) > **10%**.
+-   **[Architecture](docs/architecture.md)**: Overview of the codebase structure and components.
-    *   **Duplication**: Flags nodes causing excessive message duplication (>3 copies of the same packet).
-*   **Topology Checks**:
-    *   **Hop Count**: Flags nodes that are >3 hops away, indicating a potentially inefficient topology.
-*   **Role Audit**:
-    *   **Deprecated Roles**: Flags any node using the deprecated `ROUTER_CLIENT` role.
-    *   **Placement Verification**: Flags `ROUTER` or `REPEATER` nodes that do not have a valid GPS position.
 
-    *   **Placement Verification**: Flags `ROUTER` or `REPEATER` nodes that do not have a valid GPS position.
+## Quick Start
-    *   **Router Density**: Flags `ROUTER` nodes that are physically too close (default < 2km) to each other, indicating redundancy.
-*   **Network Size**: Warns if the network size exceeds the recommendation for the current preset (e.g. > 60 nodes for LONG_FAST).
 
-### 2. Auto-Discovery of Targets
+1.  **Install Dependencies**:
-If `priority_nodes` is empty in `config.yaml`, the monitor will automatically select targets based on:
-- **Roles**: Prioritizes `ROUTER`, `ROUTER_CLIENT`, `REPEATER`, then `CLIENT` (configurable).
-- **Geolocation**: Selects a mix of the nearest and furthest nodes to test both neighborhood and long-range connectivity.
-- **Limit**: Configurable limit (default 5) to keep the test cycle manageable.
-
-### 3. Geospatial Analysis
-*   **Signal vs Distance**: Flags nodes that are close (< 1km) but have poor SNR (< -5dB), indicating potential hardware issues or obstructions.
-*   **Distance Calculation**: Uses GPS coordinates to calculate distances between nodes for topology analysis.
-
-### 3. Route Analysis (New!)
-*   **Relay Usage Statistics**: Identifies which nodes are acting as relays most frequently (your network's "backbone").
-*   **Bottleneck Detection**: Flags nodes that are critical for reaching multiple destinations (single points of failure).
-*   **Common Paths**: Analyzes path stability to identify fluctuating routes.
-*   **Link Quality**: Aggregates SNR data to visualize link quality between nodes.
-
-### 4. Local Configuration Analysis (On Boot)
-*   **Role Check**: Warns if the monitoring node itself is set to `ROUTER` or `ROUTER_CLIENT` (Monitoring is best done as `CLIENT`).
-*   **Hop Limit**: Warns if the default hop limit is > 3, which can cause network congestion.
-
-### 5. Data Persistence & Regeneration
-*   **JSON Data**: Saves all raw data (nodes, test results, analysis) to a JSON file alongside the Markdown report.
-*   **Regeneration Tool**: Includes `report_generate.py` to regenerate reports from JSON files, allowing for format updates or re-analysis without re-running tests.
-
-### 6. Comprehensive Reporting
-*   Generates a detailed **Markdown Report** (`report-YYYYMMDD-HHMMSS.md`) after each test cycle.
-*   Includes:
-    *   Executive Summary
-    *   Network Health Findings
-    *   Route Analysis (Relays, Bottlenecks)
-    *   Detailed Traceroute Results Table
-
-## Installation
-
-1.  **Clone the repository** (if applicable) or navigate to the project folder.
-2.  **Set up a Virtual Environment** (Recommended):
-    ```bash
-    python3 -m venv venv
-    source venv/bin/activate
-    ```
-3.  **Install Dependencies**:
     ```bash
     pip install -r requirements.txt
     ```
 
-## Usage
+2.  **Configure**:
-
+    Copy `sample-config.yaml` to `config.yaml` and edit it:
-### Basic Run (USB/Serial)
-Connect your Meshtastic device via USB and run:
-```bash
-python3 main.py
-```
-
-### Network Connection (TCP)
-If your node is on the network (e.g., WiFi):
-```bash
-python3 main.py --tcp 192.168.1.10
-```
-
-### Options
-*   `--ignore-no-position`: Suppress warnings about routers without a position (useful for portable routers or privacy).
     ```bash
-    python3 main.py --ignore-no-position
+    cp sample-config.yaml config.yaml
+    nano config.yaml
     ```
 
-### Regenerating Reports
+3.  **Run**:
-To regenerate a report from a saved JSON file (e.g., to apply new analysis logic):
+    ```bash
-```bash
+    python3 main.py
-python3 report_generate.py reports/report-YYYYMMDD-HHMMSS.json
+    ```
-```
-You can also specify a custom output filename:
-```bash
-python3 report_generate.py reports/report-YYYYMMDD-HHMMSS.json --output my_custom_report.md
-```
 
-## Configuration (Priority Testing)
+## Features at a Glance
 
-To prioritize testing specific nodes (e.g., to check if a router is reachable), add their IDs to `config.yaml`:
+-   **Passive Health Checks**: Detects congestion (>25% ChUtil), spam (>10% AirUtil), and bad topology.
-
+-   **Auto-Discovery**: Automatically finds and tests important nodes (Routers, Repeaters).
-```yaml
+-   **Active Testing**: Performs traceroutes to map the network and find dead zones.
-priority_nodes:
+-   **Route Analysis**: Identifies critical relays and bottlenecks.
-  - "!12345678" 
+-   **Reporting**: Generates detailed Markdown and HTML reports with network insights.
-  - "!87654321"
+-   **Data Persistence**: Saves all data to JSON for future analysis.
-
-# Auto-Discovery Settings
-analysis_mode: router_clusters # 'distance' or 'router_clusters'
-cluster_radius: 3000 # Meters
-
-# Generate report after N full testing cycles
-report_cycles: 1
-
-# Active Testing Settings
-traceroute_timeout: 90
-active_test_interval: 30
-
-# Manual Geolocation Overrides
-manual_positions:
-  "!12345678":
-    lat: 59.12345
-    lon: 24.12345
-
-# Thresholds for Analysis
-thresholds:
-  channel_utilization: 25.0 # Percent
-  air_util_tx: 7.0 # Percent
-  router_density_threshold: 2000 # Meters (Minimum distance between routers)
-
-# Network Size Settings
-max_nodes_for_long_fast: 60
-```
-
-The monitor will cycle through these nodes and send traceroute requests to them.
-
-## Interpreting Logs
-
-The monitor outputs logs to the console. Here is how to interpret common messages:
-
-### Health Warnings
-```text
-WARNING - Found 2 potential issues:
-WARNING -   - Congestion: Node 'MountainRepeater' reports ChUtil 45.0% (Threshold: 25.0%)
-```
-*   **Meaning**: The node 'MountainRepeater' is seeing very high traffic. It might be in a noisy area or hearing too many nodes.
-*   **Action**: Investigate the node. If it's a router, consider moving it or changing its settings.
-
-```text
-WARNING -   - Config: Node 'OldUnit' is using deprecated role 'ROUTER_CLIENT'.
-```
-*   **Meaning**: 'OldUnit' is configured with a role that is known to cause routing loops.
-*   **Action**: Change the role to `CLIENT`, `ROUTER`, or `CLIENT_MUTE`.
-
-### Active Test Logs
-```text
-INFO - Sending traceroute to priority node !12345678...
-...
-INFO - Received Traceroute Packet: {...}
-```
-*   **Meaning**: The monitor sent a test packet and received a response.
-*   **Action**: Check the hop count in the response (if visible/parsed) to verify the path.
 
 ## Project Structure
-*   `mesh_analyzer/`: Source code.
+
-    *   `monitor.py`: Main application loop.
+-   `mesh_analyzer/`: Core application logic.
-    *   `analyzer.py`: Health check logic.
+-   `scripts/`: Utilities like `report_generate.py`.
-    *   `active_tests.py`: Traceroute logic.
+-   `reports/`: Output directory for reports and data.
-*   `tests/`: Unit tests.
+-   `docs/`: Detailed documentation.
-*   `config.yaml`: Configuration file.
+

docs/architecture.md

@@ -0,0 +1,68 @@
+# Architecture Overview
+
+The LoRa Mesh Analyzer is structured as a modular Python application. This document outlines the key components and their responsibilities.
+
+## Directory Structure
+
+```
+LoRa-Mesh-Analyzer/
+├── mesh_analyzer/       # Core package
+│   ├── monitor.py       # Main application loop and orchestration
+│   ├── analyzer.py      # Passive health check logic
+│   ├── active_tests.py  # Active testing logic (Traceroute, etc.)
+│   ├── reporter.py      # Report generation (Markdown/HTML)
+│   ├── route_analyzer.py# Route analysis and topology mapping
+│   ├── config_validator.py # Configuration validation
+│   └── utils.py         # Shared utility functions
+├── scripts/             # Standalone scripts and tools
+│   ├── report_generate.py # Tool to regenerate reports from JSON
+│   └── ...
+├── reports/             # Generated reports and data
+├── tests/               # Unit tests
+├── config.yaml          # User configuration
+└── main.py              # Entry point
+```
+
+## Core Components
+
+### `monitor.py`
+The central coordinator. It:
+1.  Initializes the Meshtastic interface.
+2.  Loads configuration.
+3.  Runs the main loop:
+    - Collects node data.
+    - Triggers auto-discovery or priority node selection.
+    - Orchestrates active tests.
+    - Invokes the analyzer and reporter.
+
+### `analyzer.py`
+Responsible for passive analysis of the mesh. It checks for:
+-   **Congestion**: High Channel Utilization.
+-   **Spam**: High Airtime usage.
+-   **Placement**: Routers without GPS, redundant routers.
+-   **Configuration**: Deprecated roles, bad hop limits.
+
+### `active_tests.py`
+Handles active network probing. It:
+-   Sends traceroute requests.
+-   Parses responses.
+-   Manages timeouts and rate limiting.
+
+### `reporter.py`
+Generates human-readable reports. It:
+-   Takes analysis results and test data.
+-   Formats them into Markdown or HTML.
+-   Saves raw data to JSON for persistence.
+
+### `route_analyzer.py`
+Analyzes the topology based on traceroute data. It:
+-   Identifies common relays (backbone nodes).
+-   Detects bottlenecks (single points of failure).
+-   Calculates link quality metrics.
+
+## Data Flow
+
+1.  **Collection**: `monitor.py` collects raw node data from the Meshtastic interface.
+2.  **Testing**: `active_tests.py` probes specific nodes and adds results to the dataset.
+3.  **Analysis**: `analyzer.py` and `route_analyzer.py` process the raw data and test results to identify issues and patterns.
+4.  **Reporting**: `reporter.py` formats the findings into a report and saves the state to JSON.

docs/configuration.md

@@ -0,0 +1,90 @@
+# Configuration Guide
+
+The `config.yaml` file controls the behavior of the Meshtastic Network Monitor. This guide explains each configuration option.
+
+## Core Settings
+
+### `log_level`
+- **Description**: Sets the verbosity of the logging output.
+- **Values**: `debug`, `info`, `warn`, `error`.
+- **Default**: `info`.
+
+## Auto-Discovery Settings
+
+These settings control how the monitor automatically finds nodes to test when `priority_nodes` is empty.
+
+### `analysis_mode`
+- **Description**: Determines the strategy for selecting target nodes.
+- **Values**:
+    - `distance`: Selects a mix of nearest and furthest nodes.
+    - `router_clusters`: Selects nodes that are within a certain radius of identified routers.
+- **Default**: `distance`.
+
+### `cluster_radius`
+- **Description**: The radius (in meters) around a router to search for nodes when `analysis_mode` is set to `router_clusters`.
+- **Default**: `2000`.
+
+### `auto_discovery_roles`
+- **Description**: A list of node roles to prioritize for testing. The monitor will look for nodes with these roles in the specified order.
+- **Values**: `ROUTER`, `ROUTER_LATE`, `REPEATER`, `CLIENT`, `CLIENT_MUTE`, `TRACKER`, etc.
+
+### `auto_discovery_limit`
+- **Description**: The maximum number of nodes to select for active testing in each cycle.
+- **Default**: `5`.
+
+## Reporting Settings
+
+### `report_cycles`
+- **Description**: The number of full testing cycles to complete before generating a report.
+- **Default**: `1`.
+
+### `report_output_formats`
+- **Description**: The formats in which to generate the report.
+- **Values**: `markdown`, `html`.
+- **Default**: `['markdown']`.
+
+## Active Testing Settings
+
+### `traceroute_timeout`
+- **Description**: The time (in seconds) to wait for a traceroute response before giving up.
+- **Default**: `90`.
+
+### `active_test_interval`
+- **Description**: The minimum time (in seconds) to wait between sending test packets to different nodes. This prevents flooding the network.
+- **Default**: `30`.
+
+### `hop_limit`
+- **Description**: The maximum number of hops for traceroute packets.
+- **Default**: `7`.
+
+### `priority_nodes`
+- **Description**: A list of specific Node IDs to test. If this list is populated, auto-discovery is disabled, and only these nodes are tested.
+- **Format**: `"!<NodeID>"` (e.g., `"!12345678"`).
+
+## Manual Geolocation Overrides
+
+### `manual_positions`
+- **Description**: Allows you to manually specify the latitude and longitude for nodes that do not report their position (e.g., fixed routers without GPS).
+- **Format**:
+  ```yaml
+  manual_positions:
+    "!nodeid":
+      lat: 59.12345
+      lon: 24.12345
+  ```
+
+## Analysis Thresholds
+
+These thresholds determine when the monitor flags a node or network condition as an issue.
+
+### `thresholds`
+- **`channel_utilization`**: The percentage of channel utilization above which a node is flagged for congestion (Default: `25.0`).
+- **`air_util_tx`**: The percentage of transmit airtime above which a node is flagged for spamming (Default: `7.0`).
+- **`router_density_threshold`**: The minimum distance (in meters) between routers. Routers closer than this are flagged as redundant (Default: `2000`).
+- **`active_threshold_seconds`**: Nodes seen within this time window are considered "active" (Default: `7200` i.e., 2 hours).
+
+## Network Size Settings
+
+### `max_nodes_for_long_fast`
+- **Description**: The recommended maximum number of nodes for the `LONG_FAST` preset. If the network size exceeds this, a warning is generated.
+- **Default**: `60`.

docs/report_generation.md

@@ -0,0 +1,38 @@
+# Report Generation Tool
+
+The `report_generate.py` script allows you to regenerate Markdown reports from existing JSON data files. This is useful for:
+- Re-applying analysis logic after updating the code or configuration.
+- Generating reports in different formats (e.g., if you forgot to enable HTML).
+- Debugging report generation issues without re-running long tests.
+
+## Usage
+
+```bash
+python3 scripts/report_generate.py <json_file_path> [--output <output_path>]
+```
+
+### Arguments
+
+- `json_file_path`: Path to the JSON data file (e.g., `reports/report-20251128-145548.json`).
+- `--output`, `-o`: (Optional) Custom output path for the Markdown report. If not specified, the report is generated in the `reports/` directory with a new timestamp.
+
+## Examples
+
+**Regenerate a report from a JSON file:**
+
+```bash
+python3 scripts/report_generate.py reports/report-20251128-145548.json
+```
+
+**Regenerate a report and save it to a specific file:**
+
+```bash
+python3 scripts/report_generate.py reports/report-20251128-145548.json --output my_custom_report.md
+```
+
+## How it Works
+
+1.  **Loads Data**: Reads the raw node data, test results, and session metadata from the JSON file.
+2.  **Applies Configuration**: Uses the configuration embedded in the JSON file, but applies any manual positions from the *current* `config.yaml` to ensure up-to-date geolocation.
+3.  **Re-runs Analysis**: Re-initializes the `NetworkHealthAnalyzer` and re-runs the analysis on the loaded data. This means any improvements to the analysis logic in the code will be reflected in the new report.
+4.  **Generates Report**: Uses the `NetworkReporter` to generate the Markdown report, incorporating the new analysis results.

docs/usage.md

@@ -0,0 +1,70 @@
+# Usage Guide
+
+This guide covers how to run the Meshtastic Network Monitor in various modes.
+
+## Prerequisites
+
+Ensure you have installed the dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+## Basic Execution
+
+### USB / Serial Connection
+If your Meshtastic device is connected via USB:
+
+```bash
+python3 main.py
+```
+
+The monitor will automatically detect the serial port.
+
+### TCP / Network Connection
+If your Meshtastic device is on the network (e.g., WiFi):
+
+```bash
+python3 main.py --tcp <IP_ADDRESS>
+```
+
+Example:
+```bash
+python3 main.py --tcp 192.168.1.10
+```
+
+## Command Line Options
+
+| Option | Description |
+| :--- | :--- |
+| `--tcp <IP>` | Connect to a device via TCP/IP instead of Serial. |
+| `--ignore-no-position` | Suppress warnings about routers without a valid GPS position. Useful for portable routers. |
+| `--help` | Show the help message and exit. |
+
+## Running in the Background
+
+To run the monitor continuously, you might want to use `nohup` or a systemd service.
+
+**Using nohup:**
+```bash
+nohup python3 main.py > monitor.log 2>&1 &
+```
+
+## Interpreting Output
+
+The monitor outputs logs to the console (and `monitor.log` if redirected).
+
+### Common Log Messages
+
+-   **`INFO - Connected to radio...`**: Successful connection to the Meshtastic device.
+-   **`INFO - Starting analysis cycle...`**: The monitor is beginning a new round of checks.
+-   **`WARNING - Congestion: Node X reports ChUtil Y%`**: The specified node is experiencing high channel utilization.
+-   **`INFO - Sending traceroute to...`**: The monitor is actively testing a node.
+
+## Reports
+
+Reports are generated in the `reports/` directory.
+-   **Format**: `report-YYYYMMDD-HHMMSS.md` (and `.html` if enabled).
+-   **Data**: `report-YYYYMMDD-HHMMSS.json` contains the raw data.
+
+See [Report Generation](report_generation.md) for details on how to regenerate reports.

mesh_analyzer/reporter.py

@@ -104,7 +104,7 @@ class NetworkReporter:
             # 1. Markdown Output
             if 'markdown' in output_formats:
                 md_filepath = os.path.join(self.report_dir, f"{base_name}.md")
-                with open(md_filepath, "w") as md_file:
+                with open(md_filepath, "w", encoding='utf-8') as md_file:
                     md_file.write(markdown_content)
                 generated_files.append(md_filepath)
                 logger.info(f"Report generated: {md_filepath}")
@@ -133,7 +133,7 @@ class NetworkReporter:
                 html_content = markdown.markdown(markdown_content, extensions=['tables', 'fenced_code'])
                 full_html = f"<!DOCTYPE html>\n<html>\n<head>\n<meta charset='utf-8'>\n<title>Meshtastic Network Report - {report_date}</title>\n{css}\n</head>\n<body>\n{html_content}\n</body>\n</html>"
                 
-                with open(html_filepath, "w") as html_file:
+                with open(html_filepath, "w", encoding='utf-8') as html_file:
                     html_file.write(full_html)
                 generated_files.append(html_filepath)
                 logger.info(f"Report generated: {html_filepath}")
@@ -237,7 +237,7 @@ class NetworkReporter:
         }
         
         # Write to file with pretty formatting
-        with open(filepath, 'w') as f:
+        with open(filepath, 'w', encoding='utf-8') as f:
             json.dump(data, f, indent=2, default=str)
 
     def _get_location_string(self, nodes, local_node):

sample-config.yaml

@@ -44,6 +44,9 @@ traceroute_timeout: 90
 # Minimum interval between tests (in seconds)
 active_test_interval: 30
 
+# Maximum hops for traceroute
+hop_limit: 7
+
 # Manual Geolocation Overrides
 # Useful for nodes that don't report position
 # Format: "!nodeid": {lat: 0.0, lon: 0.0}

scripts/report_generate.py

@@ -34,7 +34,7 @@ def load_json_data(json_filepath):
         sys.exit(1)
     
     try:
-        with open(json_filepath, 'r') as f:
+        with open(json_filepath, 'r', encoding='utf-8') as f:
             data = json.load(f)
         return data
     except json.JSONDecodeError as e:

setup.py

@@ -9,6 +9,7 @@ setup(
         "meshtastic",
         "pypubsub",
         "PyYAML",
+        "markdown",
     ],
     entry_points={
         "console_scripts": [