Commit Graph

105 Commits

Author SHA1 Message Date
Louis King 973bf23fe8 docs: centralise env vars and split deployment/observer/maintenance docs
Move scattered configuration tables and operational sections out of the
README into dedicated reference documents:

- docs/configuration.md: single source of truth for all environment
  variables, grouped into 12 sections (Common, Database, Caching,
  Collector, Webhooks, Auth, Data Retention, API, Web Dashboard,
  Feature Flags, Traefik, Prometheus & Alertmanager)
- docs/deployment.md: production setup, reverse proxy, multi-instance,
  API scaling, Redis caching
- docs/observer.md: remote observers plus PACKETCAPTURE_* and
  SERIAL_PORT reference
- docs/maintenance.md: backup and restore

README is reduced from 712 to 385 lines; the ARM32/Raspberry Pi note
is dropped. database.md, auth.md, webhooks.md, and content.md have
their env-var tables removed and link back to configuration.md. Stale
cross-references in database.md, upgrading.md, and .env.example are
updated to point at the new locations.
2026-06-18 12:20:49 +01:00
Louis King 30e8e88ee9 docs: add SQLite deprecation notice and consolidate DB docs
Replace the lingering v0.9 'Breaking Changes' alert with a concise v0.14
'DEPRECATION NOTICE' for SQLite (dual compatibility for ~3 months, then
PostgreSQL-only).

Move all database-specific instructions (SQLite + PostgreSQL) out of the
README into a new canonical docs/database.md covering:
- SQLite zero-config default (DATA_HOME / meshcore.db, WAL/single-host)
- PostgreSQL: DATABASE_* env vars, bundled Docker profile, production
  role/database provisioning (mirrors the ipnet-mesh/infrastructure init
  script), managed/external Postgres and DATABASE_URL
- Schema-per-instance (search_path) isolation for multiple instances on a
  shared cluster
- Pointer to the SQLite->PostgreSQL migration runbook in upgrading.md

Update the README Multi-Instance Deployments and Scaling the API sections
to link to docs/database.md, and add the new doc to the docs list and
project tree. Add a pointer in .env.example and a Postgres note in
AGENTS.md.

Consolidate docs/upgrading.md v0.14: move the env-var/schema/provisioning
reference to docs/database.md (single source of truth) and keep only the
upgrade-time migration runbook and dashboard chart fix.
2026-06-17 15:14:53 +01:00
Louis King 93d894888c Merge branch 'main' of github.com:ipnet-mesh/meshcore-hub into feat/postgres-support 2026-06-14 20:38:42 +01:00
Louis King 17e6b65f8c feat(web): add system announcement banner and maintenance mode
Add two operator-controlled, startup-time settings:

- SYSTEM_ANNOUNCEMENT: non-dismissable Markdown banner rendered above the
  network announcement on every page (navbar -> system -> network).
- SYSTEM_MAINTENANCE: when enabled, forces all feature flags off so the nav
  collapses to Home, hides the profile menu, and the SPA renders a maintenance
  page for every route. The maintenance page makes no API calls, so the API
  and database can be offline while the web component keeps running.

CLI exposes --system-announcement and tri-state --system-maintenance; the bool
falls back to pydantic settings to parse SYSTEM_MAINTENANCE reliably from env.
Adds i18n strings (en/nl), tests, and docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:42:42 +01:00
Louis King afda05403f Phase 5: document optional Postgres backend + migration runbook
Add a Database Backend section to the README (config vars, compose profile,
schema-per-instance) and an "Optional PostgreSQL Backend" section to
docs/upgrading.md covering enablement, search_path isolation, role/db
provisioning, and the SQLite -> Postgres data-migration runbook.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 08:37:26 +01:00
Louis King 87e7d7676b Link rows to packet-detail page with path-hash node lookup
Adverts/Messages rows now link directly to the deduplicated packet-detail
page (/packets/hash/:hash). Each path hop renders as a clickable badge
opening a popover that looks up nodes by public-key prefix via the new
pubkey_prefix query param on GET /api/v1/nodes (case-insensitive
startswith). Adds a derived path_hash_bytes field on GroupedPacketRead.

Defaults changed: FEATURE_PACKETS now defaults to true and
RAW_PACKET_RETENTION_DAYS to 7 (independent of DATA_RETENTION_DAYS).

Fixes a mypy arg-type error by explicitly annotating the packet_hash
list as list[str].
2026-06-13 16:47:22 +01:00
Louis King 03603a83e2 feat(api): configurable worker processes via API_WORKERS
Add a --workers / API_WORKERS option so the API can run multiple worker
processes in a single container for multi-core concurrency, without
needing to scale containers (which would conflict with the api service's
fixed container_name and complicate per-stack ops/monitoring).

The existing create_app() carries hardcoded defaults (sqlite:///./
meshcore.db, Redis off), so forked workers cannot use it — they would
open the wrong database and run without caching. Add an env-driven
factory, create_app_from_env(), that rebuilds the app from APISettings
plus the CLI-only env vars (CORS_ORIGINS, METRICS_ENABLED,
METRICS_CACHE_TTL), mirroring the single-process resolution. workers > 1
runs uvicorn against this factory via an import string; workers == 1
keeps the single-process object path so local CLI flags still apply.

Wire API_WORKERS into the api compose service (default 1, unchanged
behaviour) and document it in the README (new "Scaling the API"
section + env table) and the v0.12 upgrading notes, including the
SQLite single-host caveat and the env-vs-CLI-flag note for workers.

Tests: create_app_from_env reads DB/Redis/MQTT/metrics from env,
honours explicit overrides, and derives the collector DB path from
DATA_HOME rather than the bare create_app default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 12:09:05 +01:00
Louis King 385d1ab141 feat: add optional Redis caching layer for API endpoints
Add Redis-backed response caching for read-heavy API endpoints (nodes,
advertisements, messages, channels, dashboard, profiles) with configurable
TTL, key prefix isolation, and graceful fallback when Redis is unavailable.

New files:
- common/redis.py: CacheBackend, NullCache, RedisCacheBackend
- api/cache.py: @cached decorator, sorted_query_string helper
- tests/test_api/test_cache.py: 23 unit tests

Changes:
- pyproject.toml: add redis[hiredis] dependency
- common/config.py: 8 Redis settings on APISettings
- api/cli.py: Redis Click options + startup banner
- api/app.py: Redis lifespan init/cleanup, X-Cache middleware, health check
- 6 route files: apply @cached decorator to list endpoints
- docker-compose.yml: Redis service (cache profile), env vars
- docker-compose.dev.yml: Redis port exposure
- .env.example, README.md, AGENTS.md, docs/upgrading.md: documentation

Redis is disabled by default (REDIS_ENABLED=false). Enable with
--profile cache and REDIS_ENABLED=true.
2026-06-09 23:08:49 +01:00
Louis King f7d9901c9b Split NETWORK_RADIO_CONFIG into individual env vars and add FEATURE_RADIO_CONFIG flag
- Replace single NETWORK_RADIO_CONFIG comma-delimited string with six
  individual environment variables: NETWORK_RADIO_PROFILE, _FREQUENCY,
  _BANDWIDTH, _SPREADING_FACTOR, _CODING_RATE, _TX_POWER
- Radio config fields now use raw numeric types (float/int) with units
  applied dynamically via RadioConfig.format_for_display()
- Add FEATURE_RADIO_CONFIG feature flag to control radio config panel
  visibility on the home page (default: enabled)
- Remove from_config_string class method (no backwards compatibility)
- Update Click CLI options, create_app() signature, and _build_config_json()
- Update docker-compose.yml, .env.example, README.md, AGENTS.md
- Add upgrading.md v0.12.0 section with migration instructions
- Add test coverage for schema, config, and feature flag
2026-06-07 14:35:40 +01:00
Louis King 5f6d44c7b8 Add database-backed channels with role-based visibility and web dashboard
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
2026-05-20 00:37:05 +01:00
Louis King 1d4d046b41 Update README observer section to match current contrib/packetcapture config 2026-05-19 20:21:40 +01:00
Louis King 8fa56337bf docs: use bullet list for networks and move section after Overview 2026-05-11 22:10:57 +01:00
Louis King 13e915feef docs: separate UK and non-UK networks onto own lines 2026-05-11 22:07:52 +01:00
Louis King 0168d98990 docs: add MeshCore Hub Networks section and move translation alert to Contributing 2026-05-11 22:05:28 +01:00
Louis King e034dfe81d docs: sync documentation with source code before release
Remove stale members.yaml references (post-members-refactor), add
missing env vars to AGENTS.md (API_HOST, WEB_HOST, CORS_ORIGINS,
NETWORK_*), fix project structure tree, add backward-compat note for
MQTT_TOPIC_PREFIX, and update nl.json/i18n.md members section.
2026-05-06 19:23:35 +01:00
Louis King 28255261fb fix: correct NODE_CLEANUP_DAYS default from 7 to 30 and add missing env vars to docs
NODE_CLEANUP_DAYS source of truth in Pydantic Settings is 30, not 7. Fixed in README, .env.example, docker-compose.yml, and docker-source-guide.md. Also added missing OIDC_POST_LOGOUT_REDIRECT_URI, WEB_AUTO_REFRESH_SECONDS, NETWORK_DOMAIN to docker-compose.yml web service. Added WEB_LOCALE and WEB_DATETIME_LOCALE to AGENTS.md env vars list.
2026-05-05 14:27:52 +01:00
Louis King 486178a471 Add OIDC-gated member filter to Nodes/Advertisements/Map pages, fix profile page issues
- Add member filter dropdown to Nodes, Advertisements, and Map pages
  (visible only when OIDC is enabled), showing profiles as
  "Name (Callsign)" format
- Add adopted_by query param to /map/data endpoint for server-side
  member filtering on the map
- Fix members feature flag: auto-disables when OIDC is disabled
- Fix profile page: remove duplicate adopted nodes section,
  extract renderMemberSince(), align form labels with fixed-width
  label column
- Add i18n keys: common.all_members, common.filter_member_label
- Add map endpoint adopted_by tests, update features tests
2026-05-02 14:43:55 +01:00
Louis King d37b30a05b Replace Member model with UserProfile-backed data
Remove the static Member model/table, CRUD API, YAML seed files, and
admin UI. Replace with UserProfile-driven members page that reads roles
from OIDC identity provider. Key changes:

- Drop members table, add roles column to user_profiles (Alembic migration)
- Add GET /api/v1/user/profiles (paginated, no user_id exposed)
- Add GET /api/v1/user/profile/me (auto-creates profile for current user)
- Replace member_id node tag filter with adopted_by (profile UUID)
- Members page now shows profiles grouped by operator/member roles
- Profile page supports public view (/profile/:id) and owner edit (/profile)
- Node detail page shows adoption card side-by-side with public key card
- Auto-create user profile during OIDC login callback
- Hide Adopted Nodes section for non-operator/admin users
- Add member since date to profile cards
- Add role badges and adopted node badges to member tiles
- Add antenna/users icons to Members page group headers
2026-04-30 20:57:26 +01:00
Louis King 31418e6847 Add user profiles with node adoption via /v1/adoptions endpoint
Move adopt/release from profile routes to dedicated /v1/adoptions endpoint.
Node API now returns adopted_by field. Profile page shows read-only adopted
nodes. Node detail page has adopt/release buttons (operator adopts, admin
can release any). Admin release bypasses ownership check.
2026-04-30 00:07:49 +01:00
Louis King a14dbaec45 Add OIDC authentication section to README before Webhooks 2026-04-29 12:55:40 +01:00
Louis King a37971b05c Add docs/auth.md and update documentation cross-references
- Create docs/auth.md covering OIDC architecture, login flow, configuration,
  local development setup, and LogTo provider guide
- Remove direnv-specific quoting note from .env.example
- Clarify OIDC_DISCOVERY_URL auto-appends .well-known/openid-configuration
- Fix admin route references from /a/ to /admin/
- Add auth.md links to README.md and AGENTS.md doc lists
- Add docs/auth.md to AGENTS.md documentation sync rule
2026-04-29 12:50:50 +01:00
Louis King d1b6f0d0a7 Fix OIDC logout redirect and username display for LogTo
- Pass client_id in logout redirect so LogTo can validate post_logout_redirect_uri
- Add OIDC_POST_LOGOUT_REDIRECT_URI config option with fallback derivation
- Move session.clear() after logout_redirect() to allow state data save
- Add 'username' to strip_userinfo() name fallback chain (LogTo uses this)
- Strip quotes from OIDC_SCOPES and pass as list to Authlib (fixes direnv
  quoting issue where literal quotes were sent in the authorization URL)
- Add OIDC_POST_LOGOUT_REDIRECT_URI to config, app state, and docs
- Add INFO-level logging to callback and logout handlers for diagnostics
- Update .env.example, README.md, AGENTS.md, docs/upgrading.md
2026-04-28 22:44:01 +01:00
Louis King 02c0a8f1b7 Add OIDC/OAuth2 authentication via Authlib
Replace WEB_ADMIN_ENABLED with full OIDC support using Authlib.
Admin access now requires authenticated sessions with IdP-assigned
roles instead of an open toggle.

- Add authlib and itsdangerous dependencies
- Add OIDC settings to WebSettings (13 env vars)
- Create web/oidc.py module (OAuth registry, session helpers)
- Add /auth/login, /auth/callback, /auth/logout, /auth/user routes
- Gate API proxy writes to admin sessions when OIDC enabled
- Protect /a/ routes with session check (redirect to login)
- Add SessionMiddleware for signed session cookies
- Add renderAuthSection navbar component (login/avatar dropdown)
- Add 401/403 interceptor in api.js for auto-redirect
- Exclude /auth/ from SPA client-side router interception
- Render auth section after translations load (fixes raw key display)
- Add custom error pages for 500s (standalone HTML, no JS deps)
- Update docker-compose.yml to pass OIDC_* env vars to web container
- Update .env.example, README, AGENTS.md, upgrading.md, i18n.md
- Add auth.* and errors.* i18n keys
- Add 200 tests (OIDC, admin, error pages)
2026-04-28 17:36:44 +01:00
Louis King 9873aa202b Remove header-based auth (ProxyHeadersMiddleware, is_authenticated config, OAuth2 SPA flows)
Remove the reverse-proxy header authentication pattern (X-Forwarded-User,
X-Auth-Request-User, Basic auth forwarding) from the web dashboard. Admin
access is now controlled solely by the WEB_ADMIN_ENABLED flag.

- Remove web_trusted_proxy_hosts config field and ProxyHeadersMiddleware
- Remove _is_authenticated_proxy_request() and api_proxy() 401 guard
- Remove is_authenticated from SPA config JSON
- Remove OAuth2 login/sign-out UI from admin pages and router
- Remove auth_required i18n keys (en, nl)
- Remove auth-related tests and fixtures
- Delete docs/hosting/nginx-proxy-manager.md
- Update README, AGENTS.md, .env.example, docs/i18n.md, agents docs-sync refs

572 tests pass, pre-commit clean.
2026-04-28 13:33:52 +01:00
Louis King 51d7de9d88 Update observer table: SNR/Path headers and value suffix cleanup
- Rename SNR header to 'SNR (dB)', Path header to 'Hops'
- Remove ' dB' and ' hop/s' suffixes from column values
- Add i18n keys for snr_db and hops to en.json and docs/i18n.md
- Fix README ARM note: only 32-bit ARM unsupported, RPi 3/4/5 work
2026-04-26 14:50:28 +01:00
Louis King 74901441f5 Update MeshCore URLs to meshcore.io and tagline
- Replace meshcore.dev and meshcore.co.uk with meshcore.io across all files
- Replace flasher.meshcore.co.uk with flasher.meshcore.io
- Update MeshCore tagline to 'Off-Grid, Open-Source Encrypted Messaging'
2026-04-24 11:15:23 +01:00
Louis King 0938ba4fd0 Add TRAEFIK_PRIORITY support for multi-instance deployments 2026-04-21 19:58:42 +01:00
Louis King 4f202c068b Replace all CDN dependencies with locally built vendor assets
- Add Node.js 22 LTS frontend build stage (package.json, build.js)
- Build Tailwind CSS v4 + DaisyUI v5 via CLI instead of runtime CDN
- Vendor lit-html, Leaflet, Chart.js, QRCode.js locally
- Update Dockerfile with multi-stage Node.js frontend build
- Migrate all DaisyUI v4 CSS variables to v5 syntax in app.css
- Fix dashboard grid layout (static class names for Tailwind JIT)
- Fix map popup transparent background (v5 variable rename)
- Fix leaflet.js.map 404 (include source maps in vendor copy)
- Fix custom page button contrast (remove btn-neutral)
- Replace table-compact with table-sm (DaisyUI v5 rename)
2026-04-20 13:16:13 +01:00
Louis King 710263b248 Add observer Docker Compose setup and update README instructions
Add contrib/packetcapture/ with ready-made docker-compose.yml and .env.example
for remote observers. Parameterize MQTT port and TLS for local network use.
Rewrite README observer section with wget-based setup workflow.
2026-04-19 21:47:57 +01:00
Louis King e513a627bb Add content and webhooks docs links to README 2026-04-18 10:04:10 +01:00
Louis King 7be5b0585c Sync documentation: add missing env vars, fix project structure trees 2026-04-17 23:04:35 +01:00
Louis King f1dc155a0f Move Webhooks and Custom Content sections to dedicated docs 2026-04-17 22:38:15 +01:00
Louis King ccadfa73fd Remove redundant docker-compose commands, move ARM note to Simple Self-Hosted Setup 2026-04-17 22:28:19 +01:00
Louis King d565e2d7a2 Add ARM/Raspberry Pi note for observer service with native install guidance 2026-04-17 22:25:18 +01:00
Louis King bba1dce1ad Fix acknowledgments: add meshcore-mqtt-broker, remove outdated packet-capture image note 2026-04-17 22:16:04 +01:00
Louis King 2cea27a85f Move translation guide to docs/i18n.md, update all references 2026-04-17 21:58:36 +01:00
Louis King dfda1a3ec1 Combine Setup and Manual Installation into single Development section 2026-04-17 21:47:47 +01:00
Louis King 34f6b69803 Move Seed Data to docs/seeding.md, move Manual Installation to Development, note packet-capture Docker image 2026-04-17 21:01:37 +01:00
Louis King 0b0a755400 Promote Feature Flags heading to same level as other config sections 2026-04-17 20:54:24 +01:00
Louis King cdfe18ff20 Move Reverse Proxy section to Production Setup, restore Feature Flags heading 2026-04-17 20:52:30 +01:00
Louis King 0e87f906b4 Move Nginx Proxy Manager setup to docs/hosting/, add Reverse Proxy section with links 2026-04-17 20:48:48 +01:00
Louis King 03afe57a16 Move UPGRADING.md to docs/ with versioned headings, extract LetsMesh decoding to docs/letsmesh.md 2026-04-17 20:41:37 +01:00
Louis King 8d8d52afea Fix 'all' profile services list in README to include mqtt and observer 2026-04-16 23:51:44 +01:00
Louis King f0b44d28ab Rename Docker volumes: hub_data→data, mqtt_broker_data→mqtt_data; rename db-migrate service to migrate 2026-04-16 23:47:34 +01:00
Louis King b5fc4c06a0 Rename receiver profile to observer, packet-capture service to observer, hub-dev fallback to hub
- Rename compose profile 'receiver' -> 'observer' across all docs
- Rename docker-compose service 'packet-capture' -> 'observer' and volume
  'packetcapture_data' -> 'observer_data'
- Change COMPOSE_PROJECT_NAME fallback from 'hub-dev' to 'hub' across all
  compose files, Makefile, .env, docs
- Remove legacy interface-mock service from test compose file
2026-04-16 23:12:22 +01:00
Louis King 3614eaf24b Restructure README: move profiles to Getting Started, add Production Setup, remove Multi-Instance
- Move Docker Compose Profiles section under Getting Started before Simple Self-Hosted Setup
- Add Production Setup section with reverse proxy config and Traefik instructions
- Remove Multi-Instance Deployment section
2026-04-16 22:54:03 +01:00
Louis King 0a37010db6 Remove bundled Prometheus/Alertmanager, fix Getting Started and docker compose examples
- Remove monitoring services from all docker compose files — monitoring is now
  user-managed infrastructure (users point their own Prometheus at /metrics)
- Remove metrics profile, prometheus/alertmanager volumes from Makefile
- Update README Getting Started: packet capture is included via --profile receiver,
  not a separate prerequisite; add remote observers guide
- Add --profile all to all docker compose command examples in README and UPGRADING.md
- Simplify UPGRADING.md backup/migration to only meshcore_hub_data
2026-04-16 22:47:07 +01:00
Louis King 1e4a75f074 Rename COLLECTOR_LETSMESH_DECODER_KEYS to COLLECTOR_CHANNEL_KEYS
Simplify the variable name to remove the legacy LetsMesh decoder prefix.
Also fix unparenthesized except tuples in web/app.py and promote the
parenthesized-exception rule to a prominent position in AGENTS.md.
2026-04-14 22:41:27 +01:00
Louis King f4648d7fe7 Split Docker Compose into base/dev/prod/traefik overrides with multi-instance support
- Split docker-compose.yml into base config + environment overrides
  - docker-compose.dev.yml: port mappings for local development
  - docker-compose.prod.yml: external proxy-net network, no exposed ports
  - docker-compose.traefik.yml: optional Traefik auto-discovery labels
- Parameterize container and volume names with COMPOSE_PROJECT_NAME
  - Default: hub-dev (containers: hub-dev-api, volumes: hub-dev_hub_data)
  - Override per instance for multi-instance deployments (hub-prod, hub-beta)
- Add Makefile with build/up/down/logs/backup/restore targets
- Add TRAEFIK_DOMAIN env var for Traefik routing configuration
- Update UPGRADING.md with volume migration instructions (rename + copy methods)
- Update README.md with multi-instance deployment and backup/restore sections
2026-04-14 20:41:26 +01:00
Louis King 123dc180f0 Move warning block above screenshot in README 2026-04-13 22:52:08 +01:00