42 Commits

Author SHA1 Message Date
Louis King fb435a53c4 test: add coverage for Redis caching layer
Add 29 new tests covering previously uncovered code paths:
- redis.py: delete(), close(), string value decoding
- cache.py: serialization branches (pydantic, dict, list), set error fallback
- app.py: lifespan Redis init (enabled/disabled), cache close on shutdown
- app.py: X-Cache middleware (HIT/MISS/no-status)
- app.py: /health/ready Redis status (connected/unreachable/omitted)
- cli.py: Redis banner output, create_app param passthrough
- Route key builders: dashboard/stats, dashboard/message-activity,
  channels, messages
2026-06-09 23:30:29 +01:00
Louis King 385d1ab141 feat: add optional Redis caching layer for API endpoints
Add Redis-backed response caching for read-heavy API endpoints (nodes,
advertisements, messages, channels, dashboard, profiles) with configurable
TTL, key prefix isolation, and graceful fallback when Redis is unavailable.

New files:
- common/redis.py: CacheBackend, NullCache, RedisCacheBackend
- api/cache.py: @cached decorator, sorted_query_string helper
- tests/test_api/test_cache.py: 23 unit tests

Changes:
- pyproject.toml: add redis[hiredis] dependency
- common/config.py: 8 Redis settings on APISettings
- api/cli.py: Redis Click options + startup banner
- api/app.py: Redis lifespan init/cleanup, X-Cache middleware, health check
- 6 route files: apply @cached decorator to list endpoints
- docker-compose.yml: Redis service (cache profile), env vars
- docker-compose.dev.yml: Redis port exposure
- .env.example, README.md, AGENTS.md, docs/upgrading.md: documentation

Redis is disabled by default (REDIS_ENABLED=false). Enable with
--profile cache and REDIS_ENABLED=true.
2026-06-09 23:08:49 +01:00
Louis King 9f79ceac14 Add test coverage for channels feature and fix CLI ResourceWarning
Add 39 new tests across 7 files to improve patch coverage:

- test_messages: sort desc/asc branches, channel visibility edge cases
- test_channels: operator role visibility filtering
- test_dashboard: tag name resolution, sender names, operator visibility
- test_config: feature dependency auto-disable rules (dashboard, map, members)
- test_letsmesh_decoder: reload_keys, _enrich_payload_decoded, guards
- test_cli: channel list/add/remove/enable/disable, _import_channels,
  seed command with channels.yaml

Fix ResourceWarning in channel CLI commands by moving db.dispose()
into try/finally blocks to ensure sessions close before engine disposal.
2026-06-04 14:37:26 +01:00
Louis King f8c2a7bb40 Rename channel visibility 'public' to 'community'
- Rename ChannelVisibility.PUBLIC to ChannelVisibility.COMMUNITY
- Update stored value from 'public' to 'community' across model, schema, API, CLI, and frontend
- Add Alembic migration to update existing database rows
- Consolidate upgrade docs: merge v0.11.0, v0.12.0, v0.13.0 into single v0.11.0 section
- Add i18n visibility level translation keys (en, nl)
- Update section headings on channels page to use t() for i18n
- Keep visibility badges lowercase per UI design
2026-06-04 14:07:12 +01:00
Louis King 5f6d44c7b8 Add database-backed channels with role-based visibility and web dashboard
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
2026-05-20 00:37:05 +01:00
Louis King 9afff5bc70 feat: add route type tracking and flood-only defaults for advertisements
Track advertisement route type (flood/transport_flood/direct/transport_direct)
and node advert timestamp to distinguish zero-hop from flood adverts, improve
deduplication with 300s buckets, and default all dashboard/ad-API queries to
flood-only (including NULL for historical records).
2026-05-15 20:55:48 +01:00
Louis King f716e34bf5 fix: add public_key filter to advertisements API endpoint
The node detail page sends a public_key query param when fetching
recent advertisements, but the API silently ignored it (FastAPI
ignores unknown query params). This returned the 10 most recent ads
network-wide instead of for the specific node, making all dates
appear as today.
2026-05-15 19:18:35 +01:00
Louis King 78d54b76e0 fix: enable async SQLite FK enforcement and clean up orphaned node relations
The async SQLAlchemy engine was missing PRAGMA foreign_keys=ON, causing
ondelete="CASCADE" constraints to be silently ignored when the collector
deleted inactive nodes. This left orphaned rows in user_profile_nodes,
event_observers, and node_tags, which crashed the API with AttributeError
when accessing assoc.node.public_key on null relationships.

- Add FK PRAGMA listener to async engine (database.py)
- Add null-guard in _build_adopted_nodes() and refactor list_profiles()
- Add cleanup_orphaned_node_relations() covering all 3 dependent tables
- Integrate orphan cleanup into scheduled retention cycle (subscriber.py)
- Add --node-cleanup/--node-cleanup-days flags to CLI cleanup command
- Fix truncate cascade warning to include user_profile_nodes/event_observers
- Add FK PRAGMA to test fixtures for cascade verification
- Add upgrade note to docs/upgrading.md
2026-05-15 16:43:38 +01:00
Louis King cee487ef42 feat: hide users with test OIDC role from public views
Add OIDC_ROLE_TEST config var (default: 'test') to exclude test users
from dashboard stats, member counts, and the Members page. Uses
server-side filtering with exclude_test query param (default: true) and
client-side defense-in-depth filter in members.js.

- Add oidc_role_test to WebSettings in config.py
- Exclude test users from operator/member count queries in dashboard.py
- Add exclude_test param to GET /api/v1/user/profiles in user_profiles.py
- Filter test users client-side in members.js via role_names.test config
- Wire oidc_role_test into app.state and frontend config in web/app.py
- Document OIDC_ROLE_TEST in AGENTS.md and .env.example
2026-05-09 00:31:03 +01:00
Louis King 829971c174 fix: allow role-less OIDC users to save their own profile
The web proxy's endpoint access mapping previously required one of
(admin/operator/member) roles for PUT /api/v1/user/profile. This
blocked OIDC users with no assigned roles from saving their own profile.

Add an _AUTHENTICATED sentinel access level that grants access to any
logged-in user regardless of roles, and apply it to the profile PUT
endpoint. The API layer already enforces owner-only checks via
RequireUserOwner, so the proxy role gate was redundant.
2026-05-08 23:49:21 +01:00
Louis King 8dc6ccdad0 feat: change default nodes sort to last_seen DESC and add mobile sort controls
- Change API and frontend default sort from name/asc to last_seen/desc
- Add mobileSortSelect() shared component for native select dropdown
- Add mobile sort select to nodes, advertisements, and messages pages
- Add i18n sort labels for all three list pages
- Update sort tests for new default with staggered timestamps
2026-05-05 19:28:28 +01:00
Louis King 160ba6d5bf feat: add clickable sort controls to list pages with alpha default for nodes
Add sort/order query parameters to Nodes, Advertisements, and Messages
API endpoints. Nodes default to alpha-by-name (via COALESCE of name tag,
node name, public key). Ads and Messages default to newest-first.

Frontend adds sortableTableHeader() component with asc/desc toggle
indicators. Sort state is preserved in URL params, surviving
auto-refresh and pagination.
2026-05-05 17:05:54 +01:00
Louis King 9af90efee4 feat: add observer multi-select and collapsible filters to list pages
- Add observer multi-select (<select multiple size=2>) to Advertisements and
  Messages filter bars, populated from /api/v1/nodes?observer=true
- Make all filter sections collapsible via <details> on Nodes, Advertisements,
  and Messages pages; collapsed by default, auto-expands when active filters
  exist, preserves open state across auto-refresh ticks
- Add backend observer=true|false query param to GET /api/v1/nodes for
  observer-only or non-observer-only node filtering via subquery
- Change observed_by in Advertisements/Messages API from single public_key
  to list[str] with .in_() for multi-select support
- Fix router.js and api.js to handle array query params (duplicate keys
  promoted to arrays, .append() per element)
- Fix createFilterHandler to use FormData.getAll() for multi-value support
- Replace DaisyUI form-control/label/label-text classes with Tailwind-native
  equivalents (flex flex-col gap-1, flex items-center py-1, opacity-80 text-sm)
  since DaisyUI CSS is tree-shaken from the build output
- Thicker collapsible border (border-2 border-base-content/25) visible in
  both light and dark themes
- Bottom-align Filter/Clear buttons via two-row form layout
- Move Observer filter to last position on Advertisements page
- Add filter_observer_label i18n key
- Add tests for observer=true node filtering and multi observer params
2026-05-05 12:22:52 +01:00
Louis King 27b9ec21f2 feat: replace admin tag page with inline editor on node detail
Replace the dedicated admin tag management page with inline tag editing
on the node detail page. Operators can now edit tags directly on nodes
they've adopted; admins retain unrestricted access.

Key changes:
- Remove admin SPA page (admin/index.js, admin/node-tags.js)
- Add inline tag editor to node-detail.js with add/edit/delete modals
- Replace RequireAdmin with RequireOperatorOrAdmin for tag API routes
- Add ownership check: operators restricted to adopted nodes only
- Add validate_and_coerce_tag_value for number/boolean coercion
- Remove unused bulk endpoints (copy, move, replace all)
- Use AbortController for event listeners to prevent accumulation
  on lit-html DOM reuse across re-renders
- Track Leaflet map instance at module scope for defensive cleanup
- Fix checkAuthResponse to only redirect on 401 (not 403)
- Update tests for new OIDC-based auth model
- Update en.json locale, i18n.md, upgrading.md, AGENTS.md
2026-05-03 21:25:53 +01:00
Louis King 560eb0796a feat: replace role=infra tag with adoption-based infrastructure detection
Replace the role=infra NodeTag convention with UserProfileNode adoption
as the canonical infrastructure indicator across map, Prometheus metrics,
and alerting. Renames is_infra to is_adopted, infra_center to
adopted_center. Map icons change to blue (adopted) / green (normal),
with all adoption UI gated on OIDC_ENABLED. Adds meshcore_nodes_adopted
gauge and Alembic migration to clean up obsolete tags.
2026-05-03 19:02:05 +01:00
Louis King f2ea530c0f feat: add description and url fields to user profiles, fix nullable field clearing
- Add description (Text) and url (String 2048) columns to user_profiles
- Expose in all API schemas (Read, Public, Update, ListItem) and list/get/profile endpoints
- Update profile.js form: add description/url inputs, render on view page
- Update members.js: render description and URL link in member tiles
- Fix update handler: use model_dump(exclude_unset=True) for nullable fields
  while protecting name (set by IdP) from being cleared
- AnyUrl validation on update, converted to str for SQLite compatibility
- Add i18n keys (description_label/placeholder, url_label/placeholder)
- 7 new API tests covering description/url CRUD, URL validation, null-clearing,
  and name non-nullability
2026-05-02 23:33:25 +01:00
Louis King d37b30a05b Replace Member model with UserProfile-backed data
Remove the static Member model/table, CRUD API, YAML seed files, and
admin UI. Replace with UserProfile-driven members page that reads roles
from OIDC identity provider. Key changes:

- Drop members table, add roles column to user_profiles (Alembic migration)
- Add GET /api/v1/user/profiles (paginated, no user_id exposed)
- Add GET /api/v1/user/profile/me (auto-creates profile for current user)
- Replace member_id node tag filter with adopted_by (profile UUID)
- Members page now shows profiles grouped by operator/member roles
- Profile page supports public view (/profile/:id) and owner edit (/profile)
- Node detail page shows adoption card side-by-side with public key card
- Auto-create user profile during OIDC login callback
- Hide Adopted Nodes section for non-operator/admin users
- Add member since date to profile cards
- Add role badges and adopted node badges to member tiles
- Add antenna/users icons to Members page group headers
2026-04-30 20:57:26 +01:00
Louis King 38c792196f Auto-populate user profile name from IdP on first access
Proxy now injects X-User-Name header from session. Profile auto-creation
uses it as the initial name value. Existing profile names are never
overwritten.
2026-04-30 00:23:16 +01:00
Louis King 31418e6847 Add user profiles with node adoption via /v1/adoptions endpoint
Move adopt/release from profile routes to dedicated /v1/adoptions endpoint.
Node API now returns adopted_by field. Profile page shows read-only adopted
nodes. Node detail page has adopt/release buttons (operator adopts, admin
can release any). Admin release bypasses ownership check.
2026-04-30 00:07:49 +01:00
Louis King e867f0074a Add observer detail rows with SNR/path data and UI polish
- Add path_len column to event_observers table with Alembic migration
- Create shared observer_utils.py for building observer responses
- Extract SNR and path_len from LetsMesh uploads via normalizer
- Pass snr/path_len from all 4 handlers to add_event_observer()
- Add expandable observer detail sub-tables to messages and ads pages
- Rename Receivers column to Observers with updated i18n key
- Add satellite dish icon to observer badges and detail rows
- Add mobile card observer detail toggle
- Show full datetime tooltip on hover for relative times
- Hide Path column from advertisement observer tables (not populated)
2026-04-26 13:02:39 +01:00
Louis King 179e3bd39b Add tests to improve patch coverage for PR #148
Covers duplicate event dedup/observer paths, IntegrityError handling,
sender name resolution, observer lists in API responses, subscriber
dispatch lifecycle, MQTT retry, and CLI group/run/seed commands.
2026-04-17 17:27:06 +01:00
Louis King 58499c420b Replace native interface with external packet capture and rename receiver to observer
Remove the meshcore_interface component in favor of external
meshcore-packet-capture for data ingestion. Rename receiver_node_id
to observer_node_id across all models, schemas, handlers, and API
routes. Add Alembic migration for the column/table renames. Fix
frontend JS property name mismatch that prevented the Receiver column
from displaying observer data.
2026-04-12 14:07:14 +01:00
Louis King 29b5820ed1 feat: support multibyte path hashes for MeshCore firmware v1.14+
Update path hash handling to accept variable-length hex-encoded hashes
(e.g. "4a" for single-byte, "b3fa" for multibyte) instead of requiring
exactly 2-character hashes. Bump meshcore dependency to >=2.3.0.

- Update normalizer to accept even-length hex strings >= 2 chars
- Update schemas and model docstrings for variable-length hashes
- Add tests for multibyte and mixed-length path hash round-trips
- Fix web test flakiness from local .env datetime locale leaking
2026-03-17 22:59:29 +00:00
Louis King 4b58160f31 fix: harden security across auth, XSS, and proxy trust
- Use hmac.compare_digest for constant-time API key comparison in auth
  and metrics endpoints to prevent timing attacks
- Escape user-controlled data in admin JS templates (members, node-tags)
  to prevent XSS via innerHTML
- Escape </script> sequences in embedded JSON config to prevent XSS
  breakout from <script> blocks
- Add configurable WEB_TRUSTED_PROXY_HOSTS setting instead of trusting
  all proxy headers unconditionally
- Warn on startup when admin is enabled with default trust-all proxy
- Remove legacy HTML dashboard endpoint (unused, superseded by SPA)
- Add comprehensive auth and dashboard test coverage
2026-03-09 22:53:53 +00:00
yellowcooln 2a380f88b4 Fix review items 001/003/005 for decoder, channel labels, and node filters 2026-03-04 20:07:37 -05:00
yellowcooln 2f40b4a730 Add LetsMesh compatibility ingest, decoder integration, and admin auth updates 2026-03-03 16:18:54 -05:00
Louis King 2a153a5239 Add role label to node last seen metric and filter alerts by role
Joins NodeTag (key='role') to the node last seen Prometheus metric so
alert rules can target infrastructure nodes only (role="infra").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 00:01:20 +00:00
Louis King 5a20da3afa Add Prometheus metrics endpoint, Alertmanager, and 1h stats window
Add /metrics endpoint with Prometheus gauges for nodes, messages,
advertisements, telemetry, trace paths, events, and members. Include
per-node last_seen timestamps for alerting. Add Alertmanager service
to Docker Compose metrics profile with default blackhole receiver.
Add NodeNotSeen alert rule (48h threshold). Add 1h time window to
all windowed metrics alongside existing 24h/7d/30d windows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 23:06:07 +00:00
Louis King 345ffd219b Separate API prefix search from exact match endpoint
- Add /api/v1/nodes/prefix/{prefix} for prefix-based node lookup
- Change /api/v1/nodes/{public_key} to exact match only
- /n/{prefix} now simply redirects to /nodes/{prefix}
- /nodes/{key} resolves prefixes via API and redirects to full key
2026-01-26 22:27:15 +00:00
Louis King 0b3ac64845 Add prefix matching support to node API endpoint
Allow users to navigate to a node using any prefix of its public key
instead of requiring the full 64-character key. If multiple nodes match
the prefix, the first one alphabetically is returned.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 21:27:36 +00:00
Louis King f7b4df13a7 Added more test coverage 2026-01-12 21:00:02 +00:00
Claude d8a0f2abb8 Fix security vulnerabilities and add validation
- Fix XSS vulnerability by using data attributes instead of inline
  onclick handlers in node_tags.html template
- Fix URL injection by using urlencode for all redirect URL parameters
- Add validation to reject moves where source and destination nodes
  are the same (returns 400 Bad Request)
- Add error handling for response.json() calls that may fail
- Add missing test coverage for update endpoint error scenarios
2026-01-11 11:51:57 +00:00
Claude 367f838371 Add admin interface for managing node tags
Implement CRUD operations for NodeTags in the admin interface:

- Add NodeTagMove schema for moving tags between nodes
- Add PUT /nodes/{public_key}/tags/{key}/move API endpoint
- Add web routes at /a/node-tags for tag management
- Create admin templates with node selector and tag management UI
- Support editing, adding, moving, and deleting tags via API calls
- Add comprehensive tests for new functionality

The interface allows selecting a node from a dropdown, viewing its
tags, and performing all CRUD operations including moving a tag
to a different node without having to delete and recreate it.
2026-01-11 01:34:07 +00:00
Louis King a290db0491 Updated chart stats 2025-12-08 19:37:45 +00:00
Louis King 0b8fc6e707 Charts 2025-12-05 19:50:22 +00:00
Claude 796e303665 Remove internal UUID fields from API responses
Internal database UUIDs (id, node_id, receiver_node_id) were being
exposed in API responses. These are implementation details that should
not be visible to API consumers. The canonical identifier for nodes
is the 64-char hex public_key.

Changes:
- Remove id, node_id from NodeTagRead, NodeRead schemas
- Remove id from MemberRead schema
- Remove id, receiver_node_id, node_id from MessageRead, AdvertisementRead,
  TracePathRead, TelemetryRead schemas
- Update web map component to use public_key instead of member.id
  for owner filtering
- Update tests to not assert on removed fields
2025-12-05 16:50:21 +00:00
Claude cbaf4f451c Add tests to verify Node API returns tags
- Add test_get_node_with_tags to verify GET /nodes/{pk} includes tags
- Add test_list_nodes_includes_tags to verify GET /nodes includes tags
- Update existing tests to assert tags field is present

The Node API was already correctly returning tags via the
lazy="selectin" relationship loading strategy. These tests
document and verify that behavior.
2025-12-03 18:26:32 +00:00
Claude 3ac5452a08 Fix dashboard route path from /dashboard/dashboard to /dashboard
The dashboard router was mounted with prefix /dashboard and the HTML
route was also /dashboard, making the full path /api/v1/dashboard/dashboard.
Changed the route to / so it's accessible at /api/v1/dashboard.
2025-12-03 14:49:14 +00:00
Claude 0ac5ba567c Fix flake8 and mypy linting errors
- Update .flake8 and pre-commit config to properly use flake8 config
- Add B008 to ignored errors (FastAPI Depends pattern)
- Add E402 to ignored errors (intentional module-level imports)
- Remove unused imports from test files and source files
- Fix f-strings without placeholders
- Add type annotations to inner async functions
- Fix SQLAlchemy execute() to use text() wrapper
- Add type: ignore comments for alembic.command imports
- Exclude alembic/ directory from mypy in pre-commit
- Update mypy overrides for test files to not require type annotations
- Fix type annotations for params dicts in web routes
- Fix generator return type in test fixtures
2025-12-03 01:24:42 +00:00
Louis King c88ee99e55 Updates 2025-12-03 01:07:22 +00:00
Claude aefa9b735f Phase 4: Implement REST API component
- Add FastAPI application with lifespan management
- Implement bearer token authentication (read/admin levels)
- Create comprehensive REST API routes:
  - Nodes: list, get by public key
  - Node tags: CRUD operations
  - Messages: list with filters, get by ID
  - Advertisements: list with filters, get by ID
  - Telemetry: list with filters, get by ID
  - Trace paths: list with filters, get by ID
  - Commands: send message, channel message, advertisement
  - Dashboard: stats API and HTML dashboard
- Add API CLI command for running the server
- Create API test suite with 44 passing tests

Routes use proper RESTful status codes (201 Created, 204 No Content).
Authentication is optional - when keys not configured, endpoints are open.
2025-12-02 23:41:32 +00:00
Claude 3c1625d4c9 Implement Phase 1: Foundation for MeshCore Hub
This commit establishes the complete foundation for the MeshCore Hub project:

- Project setup with pyproject.toml (Python 3.11+, all dependencies)
- Development tools: black, flake8, mypy, pytest configuration
- Pre-commit hooks for code quality
- Package structure with all components (interface, collector, api, web)

Common package includes:
- Pydantic settings for all component configurations
- SQLAlchemy models for nodes, messages, advertisements, traces, telemetry
- Pydantic schemas for events, API requests/responses, commands
- MQTT client utilities with topic builder
- Logging configuration

Database infrastructure:
- Alembic setup with initial migration for all tables
- Database manager with session handling

CLI entry point:
- Click-based CLI with subcommands for all components
- Database migration commands (upgrade, downgrade, revision)

Tests:
- Basic test suite for config and models
- pytest fixtures for in-memory database testing
2025-12-02 23:10:53 +00:00