Files
meshcore-stats/src/meshmon/meshcore_client.py
T
Jorijn Schrijvershof a9f6926104 test: add comprehensive pytest test suite with 95% coverage (#29)
* test: add comprehensive pytest test suite with 95% coverage

Add full unit and integration test coverage for the meshcore-stats project:

- 1020 tests covering all modules (db, charts, html, reports, client, etc.)
- 95.95% code coverage with pytest-cov (95% threshold enforced)
- GitHub Actions CI workflow for automated testing on push/PR
- Proper mocking of external dependencies (meshcore, serial, filesystem)
- SVG snapshot infrastructure for chart regression testing
- Integration tests for collection and rendering pipelines

Test organization:
- tests/charts/: Chart rendering and statistics
- tests/client/: MeshCore client and connection handling
- tests/config/: Environment and configuration parsing
- tests/database/: SQLite operations and migrations
- tests/html/: HTML generation and Jinja templates
- tests/reports/: Report generation and formatting
- tests/retry/: Circuit breaker and retry logic
- tests/unit/: Pure unit tests for utilities
- tests/integration/: End-to-end pipeline tests

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: add test-engineer agent configuration

Add project-local test-engineer agent for pytest test development,
coverage analysis, and test review tasks.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: comprehensive test suite review with 956 tests analyzed

Conducted thorough review of all 956 test cases across 47 test files:

- Unit Tests: 338 tests (battery, metrics, log, telemetry, env, charts, html, reports, formatters)
- Config Tests: 53 tests (env loading, config file parsing)
- Database Tests: 115 tests (init, insert, queries, migrations, maintenance, validation)
- Retry Tests: 59 tests (circuit breaker, async retries, factory)
- Charts Tests: 76 tests (transforms, statistics, timeseries, rendering, I/O)
- HTML Tests: 81 tests (site generation, Jinja2, metrics builders, reports index)
- Reports Tests: 149 tests (location, JSON/TXT formatting, aggregation, counter totals)
- Client Tests: 63 tests (contacts, connection, meshcore availability, commands)
- Integration Tests: 22 tests (reports, collection, rendering pipelines)

Results:
- Overall Pass Rate: 99.7% (953/956)
- 3 tests marked for improvement (empty test bodies in client tests)
- 0 tests requiring fixes

Key findings documented in test_review/tests.md including quality
observations, F.I.R.S.T. principle adherence, and recommendations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: implement snapshot testing for charts and reports

Add comprehensive snapshot testing infrastructure:

SVG Chart Snapshots:
- Deterministic fixtures with fixed timestamps (2024-01-15 12:00:00)
- Tests for gauge/counter metrics in light/dark themes
- Empty chart and single-point edge cases
- Extended normalize_svg_for_snapshot_full() for reproducible comparisons

TXT Report Snapshots:
- Monthly/yearly report snapshots for repeater and companion
- Empty report handling tests
- Tests in tests/reports/test_snapshots.py

Infrastructure:
- tests/snapshots/conftest.py with shared fixtures
- UPDATE_SNAPSHOTS=1 environment variable for regeneration
- scripts/generate_snapshots.py for batch snapshot generation

Run `UPDATE_SNAPSHOTS=1 pytest tests/charts/test_chart_render.py::TestSvgSnapshots`
to generate initial snapshots.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: fix SVG normalization and generate initial snapshots

Fix normalize_svg_for_snapshot() to handle:
- clipPath IDs like id="p47c77a2a6e"
- url(#p...) references
- xlink:href="#p..." references
- <dc:date> timestamps

Generated initial snapshot files:
- 7 SVG chart snapshots (gauge, counter, empty, single-point in light/dark)
- 6 TXT report snapshots (monthly/yearly for repeater/companion + empty)

All 13 snapshot tests now pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: fix SVG normalization to preserve axis rendering

The SVG normalization was replacing all matplotlib-generated IDs with
the same value, causing duplicate IDs that broke SVG rendering:
- Font glyphs, clipPaths, and tick marks all got id="normalized"
- References couldn't resolve to the correct elements
- X and Y axes failed to render in normalized snapshots

Fix uses type-specific prefixes with sequential numbering:
- glyph_N for font glyphs (DejaVuSans-XX patterns)
- clip_N for clipPath definitions (p[0-9a-f]{8,} patterns)
- tick_N for tick marks (m[0-9a-f]{8,} patterns)

This ensures all IDs remain unique while still being deterministic
for snapshot comparison.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: add coverage and pytest artifacts to gitignore

Add .coverage, .coverage.*, htmlcov/, and .pytest_cache/ to prevent
test artifacts from being committed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* style: fix all ruff lint errors across codebase

- Sort and organize imports (I001)
- Use modern type annotations (X | Y instead of Union, collections.abc)
- Remove unused imports (F401)
- Combine nested if statements (SIM102)
- Use ternary operators where appropriate (SIM108)
- Combine nested with statements (SIM117)
- Use contextlib.suppress instead of try-except-pass (SIM105)
- Add noqa comments for intentional SIM115 violations (file locks)
- Add TYPE_CHECKING import for forward references
- Fix exception chaining (B904)

All 1033 tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: add TDD workflow and pre-commit requirements to CLAUDE.md

- Add mandatory test-driven development workflow (write tests first)
- Add pre-commit requirements (must run lint and tests before committing)
- Document test organization and running commands
- Document 95% coverage requirement

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: resolve mypy type checking errors with proper structural fixes

- charts.py: Create PeriodConfig dataclass for type-safe period configuration,
  use mdates.date2num() for matplotlib datetime handling, fix x-axis limits
  for single-point charts
- db.py: Add explicit int() conversion with None handling for SQLite returns
- env.py: Add class-level type annotations to Config class
- html.py: Add MetricDisplay TypedDict, fix import order, add proper type
  annotations for table data functions
- meshcore_client.py: Add return type annotation

Update tests to use new dataclass attribute access and regenerate SVG
snapshots. Add mypy step to CLAUDE.md pre-commit requirements.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: cast Jinja2 template.render() to str for mypy

Jinja2's type stubs declare render() as returning Any, but it actually
returns str. Wrap with str() to satisfy mypy's no-any-return check.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ci: improve workflow security and reliability

- test.yml: Pin all actions by SHA, add concurrency control to cancel
  in-progress runs on rapid pushes
- release-please.yml: Pin action by SHA, add 10-minute timeout
- conftest.py: Fix snapshot_base_time to use explicit UTC timezone for
  consistent behavior across CI and local environments

Regenerate SVG snapshots with UTC-aware timestamps.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add mypy command to permissions in settings.local.json

* test: add comprehensive script tests with coroutine warning fixes

- Add tests/scripts/ with tests for collect_companion, collect_repeater,
  and render scripts (1135 tests total, 96% coverage)
- Fix unawaited coroutine warnings by using AsyncMock properly for async
  functions and async_context_manager_factory fixture for context managers
- Add --cov=scripts to CI workflow and pyproject.toml coverage config
- Omit scripts/generate_snapshots.py from coverage (dev utility)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: migrate claude setup to codex skills

* feat: migrate dependencies to uv (#31)

* fix: run tests through uv

* test: fix ruff lint issues in tests

Consolidate patch context managers and clean unused imports/variables

Use datetime.UTC in snapshot fixtures

* test: avoid unawaited async mocks in entrypoint tests

* ci: replace codecov with github coverage artifacts

Add junit XML output and coverage summary in job output

Upload HTML and XML coverage artifacts (3.12 only) on every run

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 17:16:53 +01:00

338 lines
10 KiB
Python

"""MeshCore client wrapper with safe command execution and contact lookup."""
import asyncio
import fcntl
from collections.abc import AsyncIterator, Coroutine
from contextlib import asynccontextmanager
from pathlib import Path
from typing import Any
from . import log
from .env import get_config
# Try to import meshcore - will fail gracefully if not installed
try:
from meshcore import EventType, MeshCore
MESHCORE_AVAILABLE = True
except ImportError:
MESHCORE_AVAILABLE = False
MeshCore = None
EventType = None
def auto_detect_serial_port() -> str | None:
"""
Auto-detect a suitable serial port for MeshCore device.
Prefers /dev/ttyACM* or /dev/ttyUSB* devices.
"""
try:
import serial.tools.list_ports
except ImportError:
log.error("pyserial not installed, cannot auto-detect serial port")
return None
ports = list(serial.tools.list_ports.comports())
if not ports:
log.error("No serial ports found")
return None
# Prefer ACM devices (CDC/ACM USB), then USB serial
for port in ports:
if "ttyACM" in port.device:
log.info(f"Auto-detected serial port: {port.device} ({port.description})")
return str(port.device)
for port in ports:
if "ttyUSB" in port.device:
log.info(f"Auto-detected serial port: {port.device} ({port.description})")
return str(port.device)
# Fall back to first available
port = ports[0]
log.info(f"Using first available port: {port.device} ({port.description})")
return str(port.device)
async def connect_from_env() -> Any | None:
"""
Connect to MeshCore device using environment configuration.
Returns:
MeshCore instance or None on failure
"""
if not MESHCORE_AVAILABLE:
log.error("meshcore library not available")
return None
cfg = get_config()
transport = cfg.mesh_transport.lower()
try:
if transport == "serial":
port = cfg.mesh_serial_port
if not port:
port = auto_detect_serial_port()
if not port:
log.error("No serial port configured or detected")
return None
log.debug(f"Connecting via serial: {port} @ {cfg.mesh_serial_baud}")
mc = await MeshCore.create_serial(
port, cfg.mesh_serial_baud, debug=cfg.mesh_debug
)
return mc
elif transport == "tcp":
log.debug(f"Connecting via TCP: {cfg.mesh_tcp_host}:{cfg.mesh_tcp_port}")
mc = await MeshCore.create_tcp(cfg.mesh_tcp_host, cfg.mesh_tcp_port)
return mc
elif transport == "ble":
if not cfg.mesh_ble_addr:
log.error("MESH_BLE_ADDR required for BLE transport")
return None
log.debug(f"Connecting via BLE: {cfg.mesh_ble_addr}")
mc = await MeshCore.create_ble(cfg.mesh_ble_addr, pin=cfg.mesh_ble_pin)
return mc
else:
log.error(f"Unknown transport: {transport}")
return None
except Exception as e:
log.error(f"Failed to connect: {e}")
return None
async def _acquire_lock_async(
lock_file,
timeout: float = 60.0,
poll_interval: float = 0.1,
) -> None:
"""Acquire exclusive file lock without blocking the event loop.
Uses non-blocking LOCK_NB with async polling to avoid freezing the event loop.
Args:
lock_file: Open file handle to lock
timeout: Maximum seconds to wait for lock
poll_interval: Seconds between lock attempts
Raises:
TimeoutError: If lock cannot be acquired within timeout
"""
loop = asyncio.get_running_loop()
deadline = loop.time() + timeout
while True:
try:
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
return
except BlockingIOError as err:
if loop.time() >= deadline:
raise TimeoutError(
f"Could not acquire serial lock within {timeout}s. "
"Another process may be using the serial port."
) from err
await asyncio.sleep(poll_interval)
@asynccontextmanager
async def connect_with_lock(
lock_timeout: float = 60.0,
) -> AsyncIterator[Any | None]:
"""Connect to MeshCore with serial port locking to prevent concurrent access.
For serial transport: Acquires exclusive file lock before connecting.
For TCP/BLE: No locking needed (protocol handles multiple connections).
Args:
lock_timeout: Maximum seconds to wait for serial lock
Yields:
MeshCore client instance, or None if connection failed
"""
cfg = get_config()
lock_file = None
mc = None
needs_lock = cfg.mesh_transport.lower() == "serial"
try:
if needs_lock:
lock_path: Path = cfg.state_dir / "serial.lock"
lock_path.parent.mkdir(parents=True, exist_ok=True)
# Use 'a' mode: doesn't truncate, creates if missing
lock_file = open(lock_path, "a") # noqa: SIM115 - must stay open for lock
try:
await _acquire_lock_async(lock_file, timeout=lock_timeout)
log.debug(f"Acquired serial lock: {lock_path}")
except Exception:
# If lock acquisition fails, close file before re-raising
lock_file.close()
lock_file = None
raise
mc = await connect_from_env()
yield mc
finally:
# Disconnect first (while we still hold the lock)
if mc is not None and hasattr(mc, "disconnect"):
try:
await mc.disconnect()
except Exception as e:
log.debug(f"Error during disconnect (ignored): {e}")
# Release lock by closing the file (close() auto-releases flock)
if lock_file is not None:
lock_file.close()
log.debug("Released serial lock")
async def run_command(
mc: Any,
cmd_coro: Coroutine,
name: str,
) -> tuple[bool, str | None, dict | None, str | None]:
"""
Run a MeshCore command and capture result.
Args:
mc: MeshCore instance
cmd_coro: The command coroutine to execute
name: Human-readable command name for logging
Returns:
(success, event_type_name, payload_dict, error_message)
"""
if not MESHCORE_AVAILABLE:
return (False, None, None, "meshcore not available")
try:
log.debug(f"Running command: {name}")
event = await cmd_coro
if event is None:
return (False, None, None, "No response received")
# Extract event type name
event_type_name = None
if hasattr(event, "type"):
event_type_name = event.type.name if hasattr(event.type, "name") else str(event.type)
# Check for error
if EventType and hasattr(event, "type") and event.type == EventType.ERROR:
error_msg = None
if hasattr(event, "payload"):
error_msg = str(event.payload)
return (False, event_type_name, None, error_msg or "Command returned error")
# Extract payload
payload = None
if hasattr(event, "payload"):
payload = event.payload
# Try to convert to dict if it's a custom object
if payload is not None and not isinstance(payload, dict):
if hasattr(payload, "__dict__"):
payload = vars(payload)
elif hasattr(payload, "_asdict"):
payload = payload._asdict()
else:
payload = {"raw": payload}
log.debug(f"Command {name} returned: {event_type_name}")
return (True, event_type_name, payload, None)
except TimeoutError:
return (False, None, None, "Timeout")
except Exception as e:
return (False, None, None, str(e))
def get_contact_by_name(mc: Any, name: str) -> Any | None:
"""
Find a contact by advertised name.
Note: This is a synchronous method on MeshCore.
Args:
mc: MeshCore instance
name: The advertised name to search for
Returns:
Contact object or None
"""
if not hasattr(mc, "get_contact_by_name"):
log.warn("get_contact_by_name not available in meshcore")
return None
try:
return mc.get_contact_by_name(name)
except Exception as e:
log.debug(f"get_contact_by_name failed: {e}")
return None
def get_contact_by_key_prefix(mc: Any, prefix: str) -> Any | None:
"""
Find a contact by public key prefix.
Note: This is a synchronous method on MeshCore.
Args:
mc: MeshCore instance
prefix: Hex prefix of the public key
Returns:
Contact object or None
"""
if not hasattr(mc, "get_contact_by_key_prefix"):
log.warn("get_contact_by_key_prefix not available in meshcore")
return None
try:
return mc.get_contact_by_key_prefix(prefix)
except Exception as e:
log.debug(f"get_contact_by_key_prefix failed: {e}")
return None
def extract_contact_info(contact: Any) -> dict[str, Any]:
"""Extract useful info from a contact object or dict."""
info = {}
attrs = ["adv_name", "name", "pubkey_prefix", "public_key", "type", "flags"]
# Handle dict contacts
if isinstance(contact, dict):
for attr in attrs:
if attr in contact:
val = contact[attr]
if val is not None:
if isinstance(val, bytes):
info[attr] = val.hex()
else:
info[attr] = val
else:
# Handle object contacts
for attr in attrs:
if hasattr(contact, attr):
val = getattr(contact, attr)
if val is not None:
if isinstance(val, bytes):
info[attr] = val.hex()
else:
info[attr] = val
return info
def list_contacts_summary(contacts: list) -> list[dict[str, Any]]:
"""Get summary of all contacts for debugging."""
result = []
for c in contacts:
info = extract_contact_info(c)
result.append(info)
return result