Commit Graph

48 Commits

Author SHA1 Message Date
Louis King 385d1ab141 feat: add optional Redis caching layer for API endpoints
Add Redis-backed response caching for read-heavy API endpoints (nodes,
advertisements, messages, channels, dashboard, profiles) with configurable
TTL, key prefix isolation, and graceful fallback when Redis is unavailable.

New files:
- common/redis.py: CacheBackend, NullCache, RedisCacheBackend
- api/cache.py: @cached decorator, sorted_query_string helper
- tests/test_api/test_cache.py: 23 unit tests

Changes:
- pyproject.toml: add redis[hiredis] dependency
- common/config.py: 8 Redis settings on APISettings
- api/cli.py: Redis Click options + startup banner
- api/app.py: Redis lifespan init/cleanup, X-Cache middleware, health check
- 6 route files: apply @cached decorator to list endpoints
- docker-compose.yml: Redis service (cache profile), env vars
- docker-compose.dev.yml: Redis port exposure
- .env.example, README.md, AGENTS.md, docs/upgrading.md: documentation

Redis is disabled by default (REDIS_ENABLED=false). Enable with
--profile cache and REDIS_ENABLED=true.
2026-06-09 23:08:49 +01:00
Louis King f7d9901c9b Split NETWORK_RADIO_CONFIG into individual env vars and add FEATURE_RADIO_CONFIG flag
- Replace single NETWORK_RADIO_CONFIG comma-delimited string with six
  individual environment variables: NETWORK_RADIO_PROFILE, _FREQUENCY,
  _BANDWIDTH, _SPREADING_FACTOR, _CODING_RATE, _TX_POWER
- Radio config fields now use raw numeric types (float/int) with units
  applied dynamically via RadioConfig.format_for_display()
- Add FEATURE_RADIO_CONFIG feature flag to control radio config panel
  visibility on the home page (default: enabled)
- Remove from_config_string class method (no backwards compatibility)
- Update Click CLI options, create_app() signature, and _build_config_json()
- Update docker-compose.yml, .env.example, README.md, AGENTS.md
- Add upgrading.md v0.12.0 section with migration instructions
- Add test coverage for schema, config, and feature flag
2026-06-07 14:35:40 +01:00
Louis King f8c2a7bb40 Rename channel visibility 'public' to 'community'
- Rename ChannelVisibility.PUBLIC to ChannelVisibility.COMMUNITY
- Update stored value from 'public' to 'community' across model, schema, API, CLI, and frontend
- Add Alembic migration to update existing database rows
- Consolidate upgrade docs: merge v0.11.0, v0.12.0, v0.13.0 into single v0.11.0 section
- Add i18n visibility level translation keys (en, nl)
- Update section headings on channels page to use t() for i18n
- Keep visibility badges lowercase per UI design
2026-06-04 14:07:12 +01:00
Louis King 5f6d44c7b8 Add database-backed channels with role-based visibility and web dashboard
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
2026-05-20 00:37:05 +01:00
Louis King dd36a240ba feat: add network announcement flash banner with Markdown support
Add NETWORK_ANNOUNCEMENT env var that displays a dismissible flash banner
on every page when set. Announcement text supports Markdown (bold, italic,
links, inline code) rendered to HTML server-side at startup.
2026-05-09 12:27:20 +01:00
Louis King cee487ef42 feat: hide users with test OIDC role from public views
Add OIDC_ROLE_TEST config var (default: 'test') to exclude test users
from dashboard stats, member counts, and the Members page. Uses
server-side filtering with exclude_test query param (default: true) and
client-side defense-in-depth filter in members.js.

- Add oidc_role_test to WebSettings in config.py
- Exclude test users from operator/member count queries in dashboard.py
- Add exclude_test param to GET /api/v1/user/profiles in user_profiles.py
- Filter test users client-side in members.js via role_names.test config
- Wire oidc_role_test into app.state and frontend config in web/app.py
- Document OIDC_ROLE_TEST in AGENTS.md and .env.example
2026-05-09 00:31:03 +01:00
Louis King e034dfe81d docs: sync documentation with source code before release
Remove stale members.yaml references (post-members-refactor), add
missing env vars to AGENTS.md (API_HOST, WEB_HOST, CORS_ORIGINS,
NETWORK_*), fix project structure tree, add backward-compat note for
MQTT_TOPIC_PREFIX, and update nl.json/i18n.md members section.
2026-05-06 19:23:35 +01:00
Louis King 28255261fb fix: correct NODE_CLEANUP_DAYS default from 7 to 30 and add missing env vars to docs
NODE_CLEANUP_DAYS source of truth in Pydantic Settings is 30, not 7. Fixed in README, .env.example, docker-compose.yml, and docker-source-guide.md. Also added missing OIDC_POST_LOGOUT_REDIRECT_URI, WEB_AUTO_REFRESH_SECONDS, NETWORK_DOMAIN to docker-compose.yml web service. Added WEB_LOCALE and WEB_DATETIME_LOCALE to AGENTS.md env vars list.
2026-05-05 14:27:52 +01:00
Louis King 31418e6847 Add user profiles with node adoption via /v1/adoptions endpoint
Move adopt/release from profile routes to dedicated /v1/adoptions endpoint.
Node API now returns adopted_by field. Profile page shows read-only adopted
nodes. Node detail page has adopt/release buttons (operator adopts, admin
can release any). Admin release bypasses ownership check.
2026-04-30 00:07:49 +01:00
Louis King a37971b05c Add docs/auth.md and update documentation cross-references
- Create docs/auth.md covering OIDC architecture, login flow, configuration,
  local development setup, and LogTo provider guide
- Remove direnv-specific quoting note from .env.example
- Clarify OIDC_DISCOVERY_URL auto-appends .well-known/openid-configuration
- Fix admin route references from /a/ to /admin/
- Add auth.md links to README.md and AGENTS.md doc lists
- Add docs/auth.md to AGENTS.md documentation sync rule
2026-04-29 12:50:50 +01:00
Louis King d1b6f0d0a7 Fix OIDC logout redirect and username display for LogTo
- Pass client_id in logout redirect so LogTo can validate post_logout_redirect_uri
- Add OIDC_POST_LOGOUT_REDIRECT_URI config option with fallback derivation
- Move session.clear() after logout_redirect() to allow state data save
- Add 'username' to strip_userinfo() name fallback chain (LogTo uses this)
- Strip quotes from OIDC_SCOPES and pass as list to Authlib (fixes direnv
  quoting issue where literal quotes were sent in the authorization URL)
- Add OIDC_POST_LOGOUT_REDIRECT_URI to config, app state, and docs
- Add INFO-level logging to callback and logout handlers for diagnostics
- Update .env.example, README.md, AGENTS.md, docs/upgrading.md
2026-04-28 22:44:01 +01:00
Louis King 02c0a8f1b7 Add OIDC/OAuth2 authentication via Authlib
Replace WEB_ADMIN_ENABLED with full OIDC support using Authlib.
Admin access now requires authenticated sessions with IdP-assigned
roles instead of an open toggle.

- Add authlib and itsdangerous dependencies
- Add OIDC settings to WebSettings (13 env vars)
- Create web/oidc.py module (OAuth registry, session helpers)
- Add /auth/login, /auth/callback, /auth/logout, /auth/user routes
- Gate API proxy writes to admin sessions when OIDC enabled
- Protect /a/ routes with session check (redirect to login)
- Add SessionMiddleware for signed session cookies
- Add renderAuthSection navbar component (login/avatar dropdown)
- Add 401/403 interceptor in api.js for auto-redirect
- Exclude /auth/ from SPA client-side router interception
- Render auth section after translations load (fixes raw key display)
- Add custom error pages for 500s (standalone HTML, no JS deps)
- Update docker-compose.yml to pass OIDC_* env vars to web container
- Update .env.example, README, AGENTS.md, upgrading.md, i18n.md
- Add auth.* and errors.* i18n keys
- Add 200 tests (OIDC, admin, error pages)
2026-04-28 17:36:44 +01:00
Louis King 9873aa202b Remove header-based auth (ProxyHeadersMiddleware, is_authenticated config, OAuth2 SPA flows)
Remove the reverse-proxy header authentication pattern (X-Forwarded-User,
X-Auth-Request-User, Basic auth forwarding) from the web dashboard. Admin
access is now controlled solely by the WEB_ADMIN_ENABLED flag.

- Remove web_trusted_proxy_hosts config field and ProxyHeadersMiddleware
- Remove _is_authenticated_proxy_request() and api_proxy() 401 guard
- Remove is_authenticated from SPA config JSON
- Remove OAuth2 login/sign-out UI from admin pages and router
- Remove auth_required i18n keys (en, nl)
- Remove auth-related tests and fixtures
- Delete docs/hosting/nginx-proxy-manager.md
- Update README, AGENTS.md, .env.example, docs/i18n.md, agents docs-sync refs

572 tests pass, pre-commit clean.
2026-04-28 13:33:52 +01:00
Louis King 0938ba4fd0 Add TRAEFIK_PRIORITY support for multi-instance deployments 2026-04-21 19:58:42 +01:00
Louis King 7be5b0585c Sync documentation: add missing env vars, fix project structure trees 2026-04-17 23:04:35 +01:00
Louis King 5bec26e5ff Align MQTT_TRANSPORT and MQTT_WS_PATH defaults with MeshCore broker requirements
The MeshCore MQTT broker requires WebSocket transport, but Python defaults
were still tcp and /mqtt from the Mosquitto era. Align all defaults to
websockets and / to match Docker Compose and documented behavior.

Also fixes docs-sync audit findings:
- Remove stale PLAN.md/TASKS.md references from AGENTS.md
- Add missing NETWORK_DOMAIN, NETWORK_NAME vars to AGENTS.md env list
- Add missing WEBHOOK_CHANNEL_MESSAGE_SECRET and WEBHOOK_DIRECT_MESSAGE_SECRET
  to AGENTS.md webhook table
- Add native install note for MQTT_HOST in .env.example
- Update UPGRADING.md note to reflect aligned defaults
2026-04-17 18:16:25 +01:00
Louis King f0b44d28ab Rename Docker volumes: hub_data→data, mqtt_broker_data→mqtt_data; rename db-migrate service to migrate 2026-04-16 23:47:34 +01:00
Louis King b5fc4c06a0 Rename receiver profile to observer, packet-capture service to observer, hub-dev fallback to hub
- Rename compose profile 'receiver' -> 'observer' across all docs
- Rename docker-compose service 'packet-capture' -> 'observer' and volume
  'packetcapture_data' -> 'observer_data'
- Change COMPOSE_PROJECT_NAME fallback from 'hub-dev' to 'hub' across all
  compose files, Makefile, .env, docs
- Remove legacy interface-mock service from test compose file
2026-04-16 23:12:22 +01:00
Louis King 1e4a75f074 Rename COLLECTOR_LETSMESH_DECODER_KEYS to COLLECTOR_CHANNEL_KEYS
Simplify the variable name to remove the legacy LetsMesh decoder prefix.
Also fix unparenthesized except tuples in web/app.py and promote the
parenthesized-exception rule to a prominent position in AGENTS.md.
2026-04-14 22:41:27 +01:00
Louis King f4648d7fe7 Split Docker Compose into base/dev/prod/traefik overrides with multi-instance support
- Split docker-compose.yml into base config + environment overrides
  - docker-compose.dev.yml: port mappings for local development
  - docker-compose.prod.yml: external proxy-net network, no exposed ports
  - docker-compose.traefik.yml: optional Traefik auto-discovery labels
- Parameterize container and volume names with COMPOSE_PROJECT_NAME
  - Default: hub-dev (containers: hub-dev-api, volumes: hub-dev_hub_data)
  - Override per instance for multi-instance deployments (hub-prod, hub-beta)
- Add Makefile with build/up/down/logs/backup/restore targets
- Add TRAEFIK_DOMAIN env var for Traefik routing configuration
- Update UPGRADING.md with volume migration instructions (rename + copy methods)
- Update README.md with multi-instance deployment and backup/restore sections
2026-04-14 20:41:26 +01:00
Louis King dbd68e9a9f Replace obsolete MQTT broker image with custom build, add connection retry and upgrade docs
- Add custom Dockerfile for meshcore-mqtt-broker (Node 22 Alpine, built from michaelhart/meshcore-mqtt-broker source)
- Add GitHub Actions workflow for weekly multi-arch MQTT broker image builds
- Add local build script (etc/docker/meshcore-mqtt-broker/build.sh)
- Update docker-compose.yml and test compose to use new ghcr.io image
- Add MQTT connection retry logic with exponential backoff to collector subscriber
- Create UPGRADING.md with migration guide for breaking changes
- Update README.md and AGENTS.md for accuracy (Python 3.14, removed commands, fixed MQTT defaults)
- Remove obsolete files (etc/mosquitto.conf, .agentmap.yaml, CLAUDE.md)
2026-04-13 22:27:39 +01:00
Louis King bf6b86696c Migrate from Node.js meshcore-decoder CLI to native Python meshcoredecoder library
Replace subprocess-based packet decoding with native Python meshcoredecoder>=0.3.2.
The decoder is now always enabled (no toggle), removing ENABLED/COMMAND/TIMEOUT
config vars. Adds _enrich_payload_decoded() to compensate for payload classes that
lack to_dict() overrides in the library. Removes Node.js/npm from Dockerfile and
deletes the patches/ directory. Adds parenthesized exception rule to AGENTS.md.
2026-04-12 18:03:20 +01:00
Louis King 58499c420b Replace native interface with external packet capture and rename receiver to observer
Remove the meshcore_interface component in favor of external
meshcore-packet-capture for data ingestion. Rename receiver_node_id
to observer_node_id across all models, schemas, handlers, and API
routes. Add Alembic migration for the column/table renames. Fix
frontend JS property name mismatch that prevented the Receiver column
from displaying observer data.
2026-04-12 14:07:14 +01:00
Louis King c7655b5242 Add external packet capture receiver to Docker Compose
Integrate the meshcore-packet-capture image as an alternative to the
native interface-receiver, reorganize compose profiles so the receiver
profile uses packet capture while native-receiver covers the built-in
and mock receivers, and switch the default collector ingest mode to
letsmesh_upload.
2026-04-11 18:37:32 +01:00
yellowcooln 2f40b4a730 Add LetsMesh compatibility ingest, decoder integration, and admin auth updates 2026-03-03 16:18:54 -05:00
Louis King 5a20da3afa Add Prometheus metrics endpoint, Alertmanager, and 1h stats window
Add /metrics endpoint with Prometheus gauges for nodes, messages,
advertisements, telemetry, trace paths, events, and members. Include
per-node last_seen timestamps for alerting. Add Alertmanager service
to Docker Compose metrics profile with default blackhole receiver.
Add NodeNotSeen alert rule (48h threshold). Add 1h time window to
all windowed metrics alongside existing 24h/7d/30d windows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 23:06:07 +00:00
Louis King a8cb20fea5 Add configurable auto-refresh for list pages
Nodes, advertisements, and messages pages now auto-refresh on a
configurable interval (WEB_AUTO_REFRESH_SECONDS, default 30s). A
pause/play toggle in the page header lets users control it. Setting
the interval to 0 disables auto-refresh entirely.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 14:37:33 +00:00
Louis King 5272a72647 Refactor i18n, add translation guide, and audit documentation
## i18n Refactoring

- Refactor admin translations to use common composable patterns
- Add common patterns: delete_entity_confirm, entity_added_success, move_entity_to_another_node, etc.
- Remove 18 duplicate keys from admin_members and admin_node_tags sections
- Update all admin JavaScript files to use new common patterns with dynamic entity composition
- Fix label consistency: rename first_seen to first_seen_label to match naming convention

## Translation Documentation

- Create comprehensive translation reference guide (languages.md) with 200+ documented keys
- Add translation architecture documentation to AGENTS.md with examples and best practices
- Add "Help Translate" call-to-action section in README with link to translation guide
- Add i18n feature to README features list

## Documentation Audit

- Add undocumented config options: API_KEY, WEB_LOCALE, WEB_DOMAIN to README and .env.example
- Fix outdated CLI syntax: interface --mode receiver → interface receiver
- Update database migration commands to use CLI wrapper (meshcore-hub db) instead of direct alembic
- Add static/locales/ directory to project structure section
- Add i18n configuration (WEB_LOCALE, WEB_THEME) to docker-compose.yml

## Testing

- All 438 tests passing
- All pre-commit checks passing (black, flake8, mypy)
- Added tests for new common translation patterns

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-13 22:19:37 +00:00
Louis King fb6cc6f5a9 Update docs to reflect recent features and config options
- Add contact cleanup, admin UI, content home, and webhook secret
  settings to .env.example and README
- Update AGENTS.md project structure with pages.py, example content
  dirs, and corrected receiver init steps
- Document new API endpoints (prefix lookup, members, dashboard
  activity, send-advertisement) in README
- Fix Docker Compose core profile to include db-migrate service
2026-02-10 23:49:31 +00:00
Louis King 70ecb5e4da Add light mode theme with dark/light toggle
- Add sun/moon toggle in navbar (top-right) using DaisyUI swap component
- Store user theme preference in localStorage, default to server config
- Add WEB_THEME env var to configure default theme (dark/light)
- Add light mode color palette with adjusted section colors for contrast
- Use CSS filter to invert white SVG logos in light mode
- Add section-colored hover/active backgrounds for navbar items
- Style hero buttons with thicker outlines and white text on hover
- Soften hero heading color in light mode
- Change member callsign badges from green to neutral
- Update AGENTS.md, .env.example with WEB_THEME documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 18:11:11 +00:00
Louis King 706c32ae01 Add feature flags to control web dashboard page visibility
Operators can now disable specific pages (Dashboard, Nodes, Advertisements,
Messages, Map, Members, Pages) via FEATURE_* environment variables. Disabled
features are fully hidden: removed from navigation, return 404 on routes,
and excluded from sitemap/robots.txt. Dashboard auto-disables when all of
Nodes/Advertisements/Messages are off. Map auto-disables when Nodes is off.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 15:43:23 +00:00
Louis King 9eb1acfc02 Add TZ variable to .env.example
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-09 00:47:58 +00:00
Louis King 8fbac2cbd6 Add NETWORK_CONTACT_YOUTUBE config for footer link
Add YouTube channel URL configuration option alongside existing
GitHub/Discord/Email contact links. Also crop logo SVG to content
bounds and pass YouTube env var through docker-compose.
2026-02-08 23:36:40 +00:00
Louis King db86b3198e Some minor UI improvements, updated env.example, and docs 2025-12-08 23:06:04 +00:00
JingleManSweep f0cee14bd8 Merge pull request #48 from ipnet-mesh/feature/mqtt-tls
Added support for MQTT TLS
2025-12-07 21:16:13 +00:00
Louis King 5ff8d16bcb Added support for MQTT TLS 2025-12-07 21:15:05 +00:00
Louis King 3bc47a33bc Added data retention and node cleanup 2025-12-06 21:27:19 +00:00
claude[bot] dc3c771c76 docs: Document MESHCORE_DEVICE_NAME configuration option
Add documentation for the new MESHCORE_DEVICE_NAME environment variable
that was introduced in this PR. Updates include:

- Added to .env.example with description
- Added to Interface Settings table in README.md
- Added to CLI Reference examples in README.md
- Added to Interface configuration table in PLAN.md

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

Co-authored-by: JingleManSweep <jinglemansweep@users.noreply.github.com>
Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-06 19:07:57 +00:00
Louis King 714c3cbbd2 Set sensible Docker tag label 2025-12-06 15:32:15 +00:00
Louis King f0531c9e40 Updated env example 2025-12-06 15:16:26 +00:00
Louis King cf2c3350cc Updates 2025-12-04 18:10:29 +00:00
Louis King fc0dc1a448 Updates 2025-12-04 16:12:51 +00:00
Claude 862820bbd3 Add DATA_HOME configuration for centralized data directory management
- Add DATA_HOME setting to CommonSettings (default: ./data)
- Update CollectorSettings with:
  - effective_database_url property (default: sqlite:///{DATA_HOME}/collector/meshcore.db)
  - effective_tags_file property (default: {DATA_HOME}/collector/tags.json)
  - collector_data_dir property
- Update APISettings with effective_database_url property
- Update WebSettings with:
  - effective_members_file property (default: {DATA_HOME}/web/members.json)
  - web_data_dir property
- Update CLI commands (collector, api, web) to:
  - Accept --data-home option
  - Use effective_* properties for defaults
  - Auto-create data directories on startup
- Update docker-compose.yml.example to use DATA_HOME volume mounts
- Update .env.example with DATA_HOME documentation
- Update PLAN.md and AGENTS.md with data directory structure docs
- Add comprehensive tests for new configuration properties
2025-12-03 19:14:13 +00:00
Louis King df6f6b779c Update directory structure: move example data to example/data
- Update docker-compose.yml.example to use example/data paths
- Update .env.example with example/data paths and comments
- Update README.md and AGENTS.md project structure documentation
- Add data/ to .gitignore (runtime data directory)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-03 17:58:42 +00:00
Louis King 594ac14509 Add webhook configuration support for collector
Add environment variable configuration for forwarding events to external
HTTP endpoints. Supports separate webhooks for advertisements, channel
messages, and direct messages with configurable timeouts, retries, and
authentication via X-Webhook-Secret header.

- Add webhook fields to CollectorSettings (config.py)
- Add create_webhooks_from_settings() to build webhooks from env vars
- Integrate WebhookDispatcher into Subscriber with background processing
- Update collector CLI to load and display webhook configuration
- Document webhook settings in README, AGENTS.md, .env.example
- Update docker-compose.yml.example with webhook env vars

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-03 17:47:39 +00:00
Louis King 0cbd01fe05 Updates 2025-12-03 17:13:18 +00:00
Louis King e6b3ceb639 Updates 2025-12-03 17:02:57 +00:00
Claude 3c1625d4c9 Implement Phase 1: Foundation for MeshCore Hub
This commit establishes the complete foundation for the MeshCore Hub project:

- Project setup with pyproject.toml (Python 3.11+, all dependencies)
- Development tools: black, flake8, mypy, pytest configuration
- Pre-commit hooks for code quality
- Package structure with all components (interface, collector, api, web)

Common package includes:
- Pydantic settings for all component configurations
- SQLAlchemy models for nodes, messages, advertisements, traces, telemetry
- Pydantic schemas for events, API requests/responses, commands
- MQTT client utilities with topic builder
- Logging configuration

Database infrastructure:
- Alembic setup with initial migration for all tables
- Database manager with session handling

CLI entry point:
- Click-based CLI with subcommands for all components
- Database migration commands (upgrade, downgrade, revision)

Tests:
- Basic test suite for config and models
- pytest fixtures for in-memory database testing
2025-12-02 23:10:53 +00:00