mirror of
https://github.com/jorijn/meshcore-stats.git
synced 2026-05-18 07:16:01 +02:00
a9f6926104
* test: add comprehensive pytest test suite with 95% coverage Add full unit and integration test coverage for the meshcore-stats project: - 1020 tests covering all modules (db, charts, html, reports, client, etc.) - 95.95% code coverage with pytest-cov (95% threshold enforced) - GitHub Actions CI workflow for automated testing on push/PR - Proper mocking of external dependencies (meshcore, serial, filesystem) - SVG snapshot infrastructure for chart regression testing - Integration tests for collection and rendering pipelines Test organization: - tests/charts/: Chart rendering and statistics - tests/client/: MeshCore client and connection handling - tests/config/: Environment and configuration parsing - tests/database/: SQLite operations and migrations - tests/html/: HTML generation and Jinja templates - tests/reports/: Report generation and formatting - tests/retry/: Circuit breaker and retry logic - tests/unit/: Pure unit tests for utilities - tests/integration/: End-to-end pipeline tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: add test-engineer agent configuration Add project-local test-engineer agent for pytest test development, coverage analysis, and test review tasks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: comprehensive test suite review with 956 tests analyzed Conducted thorough review of all 956 test cases across 47 test files: - Unit Tests: 338 tests (battery, metrics, log, telemetry, env, charts, html, reports, formatters) - Config Tests: 53 tests (env loading, config file parsing) - Database Tests: 115 tests (init, insert, queries, migrations, maintenance, validation) - Retry Tests: 59 tests (circuit breaker, async retries, factory) - Charts Tests: 76 tests (transforms, statistics, timeseries, rendering, I/O) - HTML Tests: 81 tests (site generation, Jinja2, metrics builders, reports index) - Reports Tests: 149 tests (location, JSON/TXT formatting, aggregation, counter totals) - Client Tests: 63 tests (contacts, connection, meshcore availability, commands) - Integration Tests: 22 tests (reports, collection, rendering pipelines) Results: - Overall Pass Rate: 99.7% (953/956) - 3 tests marked for improvement (empty test bodies in client tests) - 0 tests requiring fixes Key findings documented in test_review/tests.md including quality observations, F.I.R.S.T. principle adherence, and recommendations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: implement snapshot testing for charts and reports Add comprehensive snapshot testing infrastructure: SVG Chart Snapshots: - Deterministic fixtures with fixed timestamps (2024-01-15 12:00:00) - Tests for gauge/counter metrics in light/dark themes - Empty chart and single-point edge cases - Extended normalize_svg_for_snapshot_full() for reproducible comparisons TXT Report Snapshots: - Monthly/yearly report snapshots for repeater and companion - Empty report handling tests - Tests in tests/reports/test_snapshots.py Infrastructure: - tests/snapshots/conftest.py with shared fixtures - UPDATE_SNAPSHOTS=1 environment variable for regeneration - scripts/generate_snapshots.py for batch snapshot generation Run `UPDATE_SNAPSHOTS=1 pytest tests/charts/test_chart_render.py::TestSvgSnapshots` to generate initial snapshots. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: fix SVG normalization and generate initial snapshots Fix normalize_svg_for_snapshot() to handle: - clipPath IDs like id="p47c77a2a6e" - url(#p...) references - xlink:href="#p..." references - <dc:date> timestamps Generated initial snapshot files: - 7 SVG chart snapshots (gauge, counter, empty, single-point in light/dark) - 6 TXT report snapshots (monthly/yearly for repeater/companion + empty) All 13 snapshot tests now pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: fix SVG normalization to preserve axis rendering The SVG normalization was replacing all matplotlib-generated IDs with the same value, causing duplicate IDs that broke SVG rendering: - Font glyphs, clipPaths, and tick marks all got id="normalized" - References couldn't resolve to the correct elements - X and Y axes failed to render in normalized snapshots Fix uses type-specific prefixes with sequential numbering: - glyph_N for font glyphs (DejaVuSans-XX patterns) - clip_N for clipPath definitions (p[0-9a-f]{8,} patterns) - tick_N for tick marks (m[0-9a-f]{8,} patterns) This ensures all IDs remain unique while still being deterministic for snapshot comparison. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: add coverage and pytest artifacts to gitignore Add .coverage, .coverage.*, htmlcov/, and .pytest_cache/ to prevent test artifacts from being committed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * style: fix all ruff lint errors across codebase - Sort and organize imports (I001) - Use modern type annotations (X | Y instead of Union, collections.abc) - Remove unused imports (F401) - Combine nested if statements (SIM102) - Use ternary operators where appropriate (SIM108) - Combine nested with statements (SIM117) - Use contextlib.suppress instead of try-except-pass (SIM105) - Add noqa comments for intentional SIM115 violations (file locks) - Add TYPE_CHECKING import for forward references - Fix exception chaining (B904) All 1033 tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: add TDD workflow and pre-commit requirements to CLAUDE.md - Add mandatory test-driven development workflow (write tests first) - Add pre-commit requirements (must run lint and tests before committing) - Document test organization and running commands - Document 95% coverage requirement 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: resolve mypy type checking errors with proper structural fixes - charts.py: Create PeriodConfig dataclass for type-safe period configuration, use mdates.date2num() for matplotlib datetime handling, fix x-axis limits for single-point charts - db.py: Add explicit int() conversion with None handling for SQLite returns - env.py: Add class-level type annotations to Config class - html.py: Add MetricDisplay TypedDict, fix import order, add proper type annotations for table data functions - meshcore_client.py: Add return type annotation Update tests to use new dataclass attribute access and regenerate SVG snapshots. Add mypy step to CLAUDE.md pre-commit requirements. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: cast Jinja2 template.render() to str for mypy Jinja2's type stubs declare render() as returning Any, but it actually returns str. Wrap with str() to satisfy mypy's no-any-return check. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * ci: improve workflow security and reliability - test.yml: Pin all actions by SHA, add concurrency control to cancel in-progress runs on rapid pushes - release-please.yml: Pin action by SHA, add 10-minute timeout - conftest.py: Fix snapshot_base_time to use explicit UTC timezone for consistent behavior across CI and local environments Regenerate SVG snapshots with UTC-aware timestamps. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: add mypy command to permissions in settings.local.json * test: add comprehensive script tests with coroutine warning fixes - Add tests/scripts/ with tests for collect_companion, collect_repeater, and render scripts (1135 tests total, 96% coverage) - Fix unawaited coroutine warnings by using AsyncMock properly for async functions and async_context_manager_factory fixture for context managers - Add --cov=scripts to CI workflow and pyproject.toml coverage config - Omit scripts/generate_snapshots.py from coverage (dev utility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: migrate claude setup to codex skills * feat: migrate dependencies to uv (#31) * fix: run tests through uv * test: fix ruff lint issues in tests Consolidate patch context managers and clean unused imports/variables Use datetime.UTC in snapshot fixtures * test: avoid unawaited async mocks in entrypoint tests * ci: replace codecov with github coverage artifacts Add junit XML output and coverage summary in job output Upload HTML and XML coverage artifacts (3.12 only) on every run --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
338 lines
10 KiB
Python
338 lines
10 KiB
Python
"""MeshCore client wrapper with safe command execution and contact lookup."""
|
|
|
|
import asyncio
|
|
import fcntl
|
|
from collections.abc import AsyncIterator, Coroutine
|
|
from contextlib import asynccontextmanager
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
from . import log
|
|
from .env import get_config
|
|
|
|
# Try to import meshcore - will fail gracefully if not installed
|
|
try:
|
|
from meshcore import EventType, MeshCore
|
|
MESHCORE_AVAILABLE = True
|
|
except ImportError:
|
|
MESHCORE_AVAILABLE = False
|
|
MeshCore = None
|
|
EventType = None
|
|
|
|
|
|
def auto_detect_serial_port() -> str | None:
|
|
"""
|
|
Auto-detect a suitable serial port for MeshCore device.
|
|
Prefers /dev/ttyACM* or /dev/ttyUSB* devices.
|
|
"""
|
|
try:
|
|
import serial.tools.list_ports
|
|
except ImportError:
|
|
log.error("pyserial not installed, cannot auto-detect serial port")
|
|
return None
|
|
|
|
ports = list(serial.tools.list_ports.comports())
|
|
if not ports:
|
|
log.error("No serial ports found")
|
|
return None
|
|
|
|
# Prefer ACM devices (CDC/ACM USB), then USB serial
|
|
for port in ports:
|
|
if "ttyACM" in port.device:
|
|
log.info(f"Auto-detected serial port: {port.device} ({port.description})")
|
|
return str(port.device)
|
|
|
|
for port in ports:
|
|
if "ttyUSB" in port.device:
|
|
log.info(f"Auto-detected serial port: {port.device} ({port.description})")
|
|
return str(port.device)
|
|
|
|
# Fall back to first available
|
|
port = ports[0]
|
|
log.info(f"Using first available port: {port.device} ({port.description})")
|
|
return str(port.device)
|
|
|
|
|
|
async def connect_from_env() -> Any | None:
|
|
"""
|
|
Connect to MeshCore device using environment configuration.
|
|
|
|
Returns:
|
|
MeshCore instance or None on failure
|
|
"""
|
|
if not MESHCORE_AVAILABLE:
|
|
log.error("meshcore library not available")
|
|
return None
|
|
|
|
cfg = get_config()
|
|
transport = cfg.mesh_transport.lower()
|
|
|
|
try:
|
|
if transport == "serial":
|
|
port = cfg.mesh_serial_port
|
|
if not port:
|
|
port = auto_detect_serial_port()
|
|
if not port:
|
|
log.error("No serial port configured or detected")
|
|
return None
|
|
|
|
log.debug(f"Connecting via serial: {port} @ {cfg.mesh_serial_baud}")
|
|
mc = await MeshCore.create_serial(
|
|
port, cfg.mesh_serial_baud, debug=cfg.mesh_debug
|
|
)
|
|
return mc
|
|
|
|
elif transport == "tcp":
|
|
log.debug(f"Connecting via TCP: {cfg.mesh_tcp_host}:{cfg.mesh_tcp_port}")
|
|
mc = await MeshCore.create_tcp(cfg.mesh_tcp_host, cfg.mesh_tcp_port)
|
|
return mc
|
|
|
|
elif transport == "ble":
|
|
if not cfg.mesh_ble_addr:
|
|
log.error("MESH_BLE_ADDR required for BLE transport")
|
|
return None
|
|
log.debug(f"Connecting via BLE: {cfg.mesh_ble_addr}")
|
|
mc = await MeshCore.create_ble(cfg.mesh_ble_addr, pin=cfg.mesh_ble_pin)
|
|
return mc
|
|
|
|
else:
|
|
log.error(f"Unknown transport: {transport}")
|
|
return None
|
|
|
|
except Exception as e:
|
|
log.error(f"Failed to connect: {e}")
|
|
return None
|
|
|
|
|
|
async def _acquire_lock_async(
|
|
lock_file,
|
|
timeout: float = 60.0,
|
|
poll_interval: float = 0.1,
|
|
) -> None:
|
|
"""Acquire exclusive file lock without blocking the event loop.
|
|
|
|
Uses non-blocking LOCK_NB with async polling to avoid freezing the event loop.
|
|
|
|
Args:
|
|
lock_file: Open file handle to lock
|
|
timeout: Maximum seconds to wait for lock
|
|
poll_interval: Seconds between lock attempts
|
|
|
|
Raises:
|
|
TimeoutError: If lock cannot be acquired within timeout
|
|
"""
|
|
loop = asyncio.get_running_loop()
|
|
deadline = loop.time() + timeout
|
|
|
|
while True:
|
|
try:
|
|
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
|
|
return
|
|
except BlockingIOError as err:
|
|
if loop.time() >= deadline:
|
|
raise TimeoutError(
|
|
f"Could not acquire serial lock within {timeout}s. "
|
|
"Another process may be using the serial port."
|
|
) from err
|
|
await asyncio.sleep(poll_interval)
|
|
|
|
|
|
@asynccontextmanager
|
|
async def connect_with_lock(
|
|
lock_timeout: float = 60.0,
|
|
) -> AsyncIterator[Any | None]:
|
|
"""Connect to MeshCore with serial port locking to prevent concurrent access.
|
|
|
|
For serial transport: Acquires exclusive file lock before connecting.
|
|
For TCP/BLE: No locking needed (protocol handles multiple connections).
|
|
|
|
Args:
|
|
lock_timeout: Maximum seconds to wait for serial lock
|
|
|
|
Yields:
|
|
MeshCore client instance, or None if connection failed
|
|
"""
|
|
cfg = get_config()
|
|
lock_file = None
|
|
mc = None
|
|
needs_lock = cfg.mesh_transport.lower() == "serial"
|
|
|
|
try:
|
|
if needs_lock:
|
|
lock_path: Path = cfg.state_dir / "serial.lock"
|
|
lock_path.parent.mkdir(parents=True, exist_ok=True)
|
|
|
|
# Use 'a' mode: doesn't truncate, creates if missing
|
|
lock_file = open(lock_path, "a") # noqa: SIM115 - must stay open for lock
|
|
try:
|
|
await _acquire_lock_async(lock_file, timeout=lock_timeout)
|
|
log.debug(f"Acquired serial lock: {lock_path}")
|
|
except Exception:
|
|
# If lock acquisition fails, close file before re-raising
|
|
lock_file.close()
|
|
lock_file = None
|
|
raise
|
|
|
|
mc = await connect_from_env()
|
|
yield mc
|
|
|
|
finally:
|
|
# Disconnect first (while we still hold the lock)
|
|
if mc is not None and hasattr(mc, "disconnect"):
|
|
try:
|
|
await mc.disconnect()
|
|
except Exception as e:
|
|
log.debug(f"Error during disconnect (ignored): {e}")
|
|
|
|
# Release lock by closing the file (close() auto-releases flock)
|
|
if lock_file is not None:
|
|
lock_file.close()
|
|
log.debug("Released serial lock")
|
|
|
|
|
|
async def run_command(
|
|
mc: Any,
|
|
cmd_coro: Coroutine,
|
|
name: str,
|
|
) -> tuple[bool, str | None, dict | None, str | None]:
|
|
"""
|
|
Run a MeshCore command and capture result.
|
|
|
|
Args:
|
|
mc: MeshCore instance
|
|
cmd_coro: The command coroutine to execute
|
|
name: Human-readable command name for logging
|
|
|
|
Returns:
|
|
(success, event_type_name, payload_dict, error_message)
|
|
"""
|
|
if not MESHCORE_AVAILABLE:
|
|
return (False, None, None, "meshcore not available")
|
|
|
|
try:
|
|
log.debug(f"Running command: {name}")
|
|
event = await cmd_coro
|
|
|
|
if event is None:
|
|
return (False, None, None, "No response received")
|
|
|
|
# Extract event type name
|
|
event_type_name = None
|
|
if hasattr(event, "type"):
|
|
event_type_name = event.type.name if hasattr(event.type, "name") else str(event.type)
|
|
|
|
# Check for error
|
|
if EventType and hasattr(event, "type") and event.type == EventType.ERROR:
|
|
error_msg = None
|
|
if hasattr(event, "payload"):
|
|
error_msg = str(event.payload)
|
|
return (False, event_type_name, None, error_msg or "Command returned error")
|
|
|
|
# Extract payload
|
|
payload = None
|
|
if hasattr(event, "payload"):
|
|
payload = event.payload
|
|
# Try to convert to dict if it's a custom object
|
|
if payload is not None and not isinstance(payload, dict):
|
|
if hasattr(payload, "__dict__"):
|
|
payload = vars(payload)
|
|
elif hasattr(payload, "_asdict"):
|
|
payload = payload._asdict()
|
|
else:
|
|
payload = {"raw": payload}
|
|
|
|
log.debug(f"Command {name} returned: {event_type_name}")
|
|
return (True, event_type_name, payload, None)
|
|
|
|
except TimeoutError:
|
|
return (False, None, None, "Timeout")
|
|
except Exception as e:
|
|
return (False, None, None, str(e))
|
|
|
|
|
|
def get_contact_by_name(mc: Any, name: str) -> Any | None:
|
|
"""
|
|
Find a contact by advertised name.
|
|
|
|
Note: This is a synchronous method on MeshCore.
|
|
|
|
Args:
|
|
mc: MeshCore instance
|
|
name: The advertised name to search for
|
|
|
|
Returns:
|
|
Contact object or None
|
|
"""
|
|
if not hasattr(mc, "get_contact_by_name"):
|
|
log.warn("get_contact_by_name not available in meshcore")
|
|
return None
|
|
|
|
try:
|
|
return mc.get_contact_by_name(name)
|
|
except Exception as e:
|
|
log.debug(f"get_contact_by_name failed: {e}")
|
|
return None
|
|
|
|
|
|
def get_contact_by_key_prefix(mc: Any, prefix: str) -> Any | None:
|
|
"""
|
|
Find a contact by public key prefix.
|
|
|
|
Note: This is a synchronous method on MeshCore.
|
|
|
|
Args:
|
|
mc: MeshCore instance
|
|
prefix: Hex prefix of the public key
|
|
|
|
Returns:
|
|
Contact object or None
|
|
"""
|
|
if not hasattr(mc, "get_contact_by_key_prefix"):
|
|
log.warn("get_contact_by_key_prefix not available in meshcore")
|
|
return None
|
|
|
|
try:
|
|
return mc.get_contact_by_key_prefix(prefix)
|
|
except Exception as e:
|
|
log.debug(f"get_contact_by_key_prefix failed: {e}")
|
|
return None
|
|
|
|
|
|
def extract_contact_info(contact: Any) -> dict[str, Any]:
|
|
"""Extract useful info from a contact object or dict."""
|
|
info = {}
|
|
|
|
attrs = ["adv_name", "name", "pubkey_prefix", "public_key", "type", "flags"]
|
|
|
|
# Handle dict contacts
|
|
if isinstance(contact, dict):
|
|
for attr in attrs:
|
|
if attr in contact:
|
|
val = contact[attr]
|
|
if val is not None:
|
|
if isinstance(val, bytes):
|
|
info[attr] = val.hex()
|
|
else:
|
|
info[attr] = val
|
|
else:
|
|
# Handle object contacts
|
|
for attr in attrs:
|
|
if hasattr(contact, attr):
|
|
val = getattr(contact, attr)
|
|
if val is not None:
|
|
if isinstance(val, bytes):
|
|
info[attr] = val.hex()
|
|
else:
|
|
info[attr] = val
|
|
|
|
return info
|
|
|
|
|
|
def list_contacts_summary(contacts: list) -> list[dict[str, Any]]:
|
|
"""Get summary of all contacts for debugging."""
|
|
result = []
|
|
for c in contacts:
|
|
info = extract_contact_info(c)
|
|
result.append(info)
|
|
return result
|