Commit Graph

74 Commits

Author SHA1 Message Date
Louis King 179e3bd39b Add tests to improve patch coverage for PR #148
Covers duplicate event dedup/observer paths, IntegrityError handling,
sender name resolution, observer lists in API responses, subscriber
dispatch lifecycle, MQTT retry, and CLI group/run/seed commands.
2026-04-17 17:27:06 +01:00
Louis King b5fc4c06a0 Rename receiver profile to observer, packet-capture service to observer, hub-dev fallback to hub
- Rename compose profile 'receiver' -> 'observer' across all docs
- Rename docker-compose service 'packet-capture' -> 'observer' and volume
  'packetcapture_data' -> 'observer_data'
- Change COMPOSE_PROJECT_NAME fallback from 'hub-dev' to 'hub' across all
  compose files, Makefile, .env, docs
- Remove legacy interface-mock service from test compose file
2026-04-16 23:12:22 +01:00
Louis King 1e4a75f074 Rename COLLECTOR_LETSMESH_DECODER_KEYS to COLLECTOR_CHANNEL_KEYS
Simplify the variable name to remove the legacy LetsMesh decoder prefix.
Also fix unparenthesized except tuples in web/app.py and promote the
parenthesized-exception rule to a prominent position in AGENTS.md.
2026-04-14 22:41:27 +01:00
Louis King 0302c0c661 Add channel filter to Messages page, fix channel label display and decoder key parsing
- Add Channel dropdown filter to Messages page (uses existing API channel_idx param)
- Add i18n keys: entities.channel, common.all_channels
- Stop auto-prefixing # on non-hashtag channel labels (Ipswich, IPNet, etc.)
- Strip quotes from decoder key entries to fix .env parsing in Docker Compose
- Add debug logging for decoder initialization and failed decryption
- Fix except syntax (AttributeError, TypeError -> parenthesized tuple)
2026-04-14 22:05:00 +01:00
Louis King dbd68e9a9f Replace obsolete MQTT broker image with custom build, add connection retry and upgrade docs
- Add custom Dockerfile for meshcore-mqtt-broker (Node 22 Alpine, built from michaelhart/meshcore-mqtt-broker source)
- Add GitHub Actions workflow for weekly multi-arch MQTT broker image builds
- Add local build script (etc/docker/meshcore-mqtt-broker/build.sh)
- Update docker-compose.yml and test compose to use new ghcr.io image
- Add MQTT connection retry logic with exponential backoff to collector subscriber
- Create UPGRADING.md with migration guide for breaking changes
- Update README.md and AGENTS.md for accuracy (Python 3.14, removed commands, fixed MQTT defaults)
- Remove obsolete files (etc/mosquitto.conf, .agentmap.yaml, CLAUDE.md)
2026-04-13 22:27:39 +01:00
Louis King bf6b86696c Migrate from Node.js meshcore-decoder CLI to native Python meshcoredecoder library
Replace subprocess-based packet decoding with native Python meshcoredecoder>=0.3.2.
The decoder is now always enabled (no toggle), removing ENABLED/COMMAND/TIMEOUT
config vars. Adds _enrich_payload_decoded() to compensate for payload classes that
lack to_dict() overrides in the library. Removes Node.js/npm from Dockerfile and
deletes the patches/ directory. Adds parenthesized exception rule to AGENTS.md.
2026-04-12 18:03:20 +01:00
Louis King 58499c420b Replace native interface with external packet capture and rename receiver to observer
Remove the meshcore_interface component in favor of external
meshcore-packet-capture for data ingestion. Rename receiver_node_id
to observer_node_id across all models, schemas, handlers, and API
routes. Add Alembic migration for the column/table renames. Fix
frontend JS property name mismatch that prevented the Receiver column
from displaying observer data.
2026-04-12 14:07:14 +01:00
Louis King 9664d4ee76 Fix Starlette 1.0 incompatibility and bump to Python 3.14
Pin starlette<1.0.0 to avoid TemplateResponse breaking change, update
the TemplateResponse call to new-style request-first signature, and
bump Python version to 3.14 across Dockerfile, pyproject.toml, and
pre-commit hooks.
2026-04-11 18:33:26 +01:00
Louis King 29b5820ed1 feat: support multibyte path hashes for MeshCore firmware v1.14+
Update path hash handling to accept variable-length hex-encoded hashes
(e.g. "4a" for single-byte, "b3fa" for multibyte) instead of requiring
exactly 2-character hashes. Bump meshcore dependency to >=2.3.0.

- Update normalizer to accept even-length hex strings >= 2 chars
- Update schemas and model docstrings for variable-length hashes
- Add tests for multibyte and mixed-length path hash round-trips
- Fix web test flakiness from local .env datetime locale leaking
2026-03-17 22:59:29 +00:00
Louis King 4b58160f31 fix: harden security across auth, XSS, and proxy trust
- Use hmac.compare_digest for constant-time API key comparison in auth
  and metrics endpoints to prevent timing attacks
- Escape user-controlled data in admin JS templates (members, node-tags)
  to prevent XSS via innerHTML
- Escape </script> sequences in embedded JSON config to prevent XSS
  breakout from <script> blocks
- Add configurable WEB_TRUSTED_PROXY_HOSTS setting instead of trusting
  all proxy headers unconditionally
- Warn on startup when admin is enabled with default trust-all proxy
- Remove legacy HTML dashboard endpoint (unused, superseded by SPA)
- Add comprehensive auth and dashboard test coverage
2026-03-09 22:53:53 +00:00
yellowcooln 2a380f88b4 Fix review items 001/003/005 for decoder, channel labels, and node filters 2026-03-04 20:07:37 -05:00
yellowcooln c22274c4e5 Add LetsMesh structured event parity mappings 2026-03-03 16:18:54 -05:00
yellowcooln 6a66eab663 Refine LetsMesh status ingest and custom logo behavior 2026-03-03 16:18:54 -05:00
yellowcooln 2f40b4a730 Add LetsMesh compatibility ingest, decoder integration, and admin auth updates 2026-03-03 16:18:54 -05:00
Louis King 2a153a5239 Add role label to node last seen metric and filter alerts by role
Joins NodeTag (key='role') to the node last seen Prometheus metric so
alert rules can target infrastructure nodes only (role="infra").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 00:01:20 +00:00
Louis King 5a20da3afa Add Prometheus metrics endpoint, Alertmanager, and 1h stats window
Add /metrics endpoint with Prometheus gauges for nodes, messages,
advertisements, telemetry, trace paths, events, and members. Include
per-node last_seen timestamps for alerting. Add Alertmanager service
to Docker Compose metrics profile with default blackhole receiver.
Add NodeNotSeen alert rule (48h threshold). Add 1h time window to
all windowed metrics alongside existing 24h/7d/30d windows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 23:06:07 +00:00
Louis King 189eb3a139 Add HTTP caching for web dashboard resources
Implement cache-control middleware to optimize browser caching and reduce
bandwidth usage. Static files are cached for 1 year when accessed with
version parameters, while dynamic content is never cached.

Changes:
- Add CacheControlMiddleware with path-based caching logic
- Register middleware in web app after ProxyHeadersMiddleware
- Add version query parameters to CSS, JS, and app.js references
- Create comprehensive test suite (20 tests) for all cache behaviors

Cache strategy:
- Static files with ?v=X.Y.Z: 1 year (immutable)
- Static files without version: 1 hour (fallback)
- SPA shell HTML: no-cache (dynamic config)
- Health endpoints: no-cache, no-store (always fresh)
- Map data: 5 minutes (location updates)
- Custom pages: 1 hour (stable markdown)
- API proxy: pass-through (backend controls)

All 458 tests passing, 95% middleware coverage.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-14 00:01:08 +00:00
Louis King 5272a72647 Refactor i18n, add translation guide, and audit documentation
## i18n Refactoring

- Refactor admin translations to use common composable patterns
- Add common patterns: delete_entity_confirm, entity_added_success, move_entity_to_another_node, etc.
- Remove 18 duplicate keys from admin_members and admin_node_tags sections
- Update all admin JavaScript files to use new common patterns with dynamic entity composition
- Fix label consistency: rename first_seen to first_seen_label to match naming convention

## Translation Documentation

- Create comprehensive translation reference guide (languages.md) with 200+ documented keys
- Add translation architecture documentation to AGENTS.md with examples and best practices
- Add "Help Translate" call-to-action section in README with link to translation guide
- Add i18n feature to README features list

## Documentation Audit

- Add undocumented config options: API_KEY, WEB_LOCALE, WEB_DOMAIN to README and .env.example
- Fix outdated CLI syntax: interface --mode receiver → interface receiver
- Update database migration commands to use CLI wrapper (meshcore-hub db) instead of direct alembic
- Add static/locales/ directory to project structure section
- Add i18n configuration (WEB_LOCALE, WEB_THEME) to docker-compose.yml

## Testing

- All 438 tests passing
- All pre-commit checks passing (black, flake8, mypy)
- Added tests for new common translation patterns

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-13 22:19:37 +00:00
Louis King a15e91c754 Further refine i18n structure
- Remove "nav" section, use "entities" references instead
- Remove composite strings like "Total Nodes", "Recent Advertisements"
  - Use composed patterns: t('common.total_entity', { entity: t('entities.nodes') })
  - Use common.recent_entity, common.edit_entity, common.add_entity patterns
- Hardcode MeshCore tagline (official trademark, not configurable)
- Update all page components and templates to use entity-based translations
- Update tests to reflect new structure
- Remove redundant page-specific composite keys

This maximizes reusability and reduces duplication across translations.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-13 21:32:36 +00:00
Claude 127cd7adf6 Add i18n support for web dashboard
Implement lightweight i18n infrastructure with shared JSON translation
files used by both server-side Jinja2 templates and client-side SPA.

- Add custom i18n module (Python + JS, ~80 lines total, zero deps)
- Create en.json with ~200 translation keys covering all web strings
- Add WEB_LOCALE config setting (default: 'en', with localStorage override)
- Translate all navigation labels, page titles, and footer in spa.html
- Translate all 13 SPA page modules (home, dashboard, nodes, etc.)
- Translate shared components (pagination, relative time, charts)
- Translate all 3 admin pages (index, members, node-tags)
- Fix Adverts/Advertisements inconsistency (standardize to Advertisements)
- Add i18n unit tests with 100% coverage

https://claude.ai/code/session_01FbnUnwYAwPrsQmAh5EuSkF
2026-02-13 18:49:06 +00:00
Louis King 706c32ae01 Add feature flags to control web dashboard page visibility
Operators can now disable specific pages (Dashboard, Nodes, Advertisements,
Messages, Map, Members, Pages) via FEATURE_* environment variables. Disabled
features are fully hidden: removed from navigation, return 404 on routes,
and excluded from sitemap/robots.txt. Dashboard auto-disables when all of
Nodes/Advertisements/Messages are off. Map auto-disables when Nodes is off.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 15:43:23 +00:00
Claude 9b09e32d41 Fix admin authentication bypass in web dashboard
The admin pages only checked config.admin_enabled but not
config.is_authenticated, allowing unauthenticated users to access
admin functionality when WEB_ADMIN_ENABLED=true. Additionally, the
API proxy forwarded the service-level Bearer token on all requests
regardless of user authentication, granting full admin API access
to unauthenticated browsers.

Server-side: block POST/PUT/DELETE/PATCH through the API proxy when
admin is enabled and no X-Forwarded-User header is present.

Client-side: add is_authenticated check to all three admin pages,
showing a sign-in prompt instead of admin content.

https://claude.ai/code/session_01HYuz5XLjYZ6JaowWqz643A
2026-02-10 01:20:04 +00:00
Louis King f1bceb5780 Rewrite web dashboard as Single Page Application
Replace server-side rendered Jinja2 page routes with a client-side SPA
using ES modules, lit-html templating, and a custom History API router.
All page rendering now happens in the browser with efficient DOM diffing.

Key changes:
- Add SPA router, API client, shared components, and 14 page modules
- Serve single spa.html shell template with catch-all route
- Remove server-side page routes (web/routes/) and legacy JS files
- Add centralized OKLCH color palette in CSS custom properties
- Add colored nav icons, navbar spacing, and loading spinner
- Add canonical URL and SEO path exclusions to SPA router
- Update charts.js to read from shared color palette
- Update tests for SPA architecture (template-agnostic assertions)
- Update AGENTS.md and README.md with SPA documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 00:23:45 +00:00
Louis King 8f660d6b94 Enhance map page with GPS fallback, infrastructure filter, and UI improvements
- Add GPS coordinate fallback: use tag coords, fall back to model coords
- Filter out nodes at (0, 0) coordinates (likely unset defaults)
- Add "Show" filter to toggle between All Nodes and Infrastructure Only
- Add "Show Labels" checkbox (labels hidden by default, appear on hover)
- Infrastructure nodes display network logo instead of emoji
- Add radius-based bounds filtering (20km) to prevent outlier zoom issues
- Position labels underneath pins, centered with transparent background
- Calculate and return infra_center for infrastructure node focus
- Initial map view focuses on infrastructure nodes when available
- Update popup button to outline style
- Add comprehensive tests for new functionality

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-07 20:05:56 +00:00
Louis King b18b3c9aa4 Refactor PAGES_HOME to CONTENT_HOME and add custom logo support
- Replace PAGES_HOME with CONTENT_HOME configuration (default: ./content)
- Content directory now contains pages/ and media/ subdirectories
- Add support for custom logo at $CONTENT_HOME/media/images/logo.svg
- Custom logo replaces favicon and navbar/home logos when present
- Mount media directory as /media for serving custom assets
- Simplify default logo to generic WiFi-style radiating arcs
- Update documentation and example directory structure
- Update tests for new CONTENT_HOME structure

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-07 13:45:42 +00:00
Louis King deaab9b9de Rename /network to /dashboard and add reusable icon macros
- Renamed network route, template, and tests to dashboard
- Added logo.svg for favicon and navbar branding
- Created reusable Jinja2 icon macros for navigation and UI elements
- Updated home page hero layout with centered content and larger logo
- Added Map button alongside Dashboard button in hero section
- Navigation menu items now display icons before labels

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-06 22:53:36 +00:00
Louis King 0f50bf4a41 Add custom markdown pages feature to web dashboard
Allows adding static content pages (About, FAQ, etc.) as markdown files
with YAML frontmatter. Pages are stored in PAGES_HOME directory (default:
./pages), automatically appear in navigation menu, and are included in
the sitemap.

- Add PageLoader class to parse markdown with frontmatter
- Add /pages/{slug} route for rendering custom pages
- Add PAGES_HOME config setting to WebSettings
- Add prose CSS styles for markdown content
- Add pages to navigation and sitemap
- Update docker-compose.yml with pages volume mount
- Add comprehensive tests for PageLoader and routes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-06 18:36:23 +00:00
Louis King 345ffd219b Separate API prefix search from exact match endpoint
- Add /api/v1/nodes/prefix/{prefix} for prefix-based node lookup
- Change /api/v1/nodes/{public_key} to exact match only
- /n/{prefix} now simply redirects to /nodes/{prefix}
- /nodes/{key} resolves prefixes via API and redirects to full key
2026-01-26 22:27:15 +00:00
Louis King 9661b22390 Fix node detail 404 to use custom error page 2026-01-26 22:11:48 +00:00
Louis King 0b3ac64845 Add prefix matching support to node API endpoint
Allow users to navigate to a node using any prefix of its public key
instead of requiring the full 64-character key. If multiple nodes match
the prefix, the first one alphabetically is returned.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 21:27:36 +00:00
Louis King f7b4df13a7 Added more test coverage 2026-01-12 21:00:02 +00:00
Louis King 13bae5c8d7 Added more test coverage 2026-01-12 20:34:53 +00:00
Louis King 307f3935e0 Add access denied page for unauthenticated admin access
When users try to access /a/ without valid OAuth2Proxy headers (e.g.,
GitHub account not in org), they now see a friendly 403 page instead
of a 500 error. Added authentication checks to all admin routes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 13:34:03 +00:00
Louis King ed2cf09ff3 Improve admin UI and remove unused coordinate tag type
- Replace node type badge with icon in admin tag editor
- Add Edit/Add Tags button on node detail page (when admin enabled and authenticated)
- Remove automatic seed container startup to prevent overwriting user changes
- Remove unused 'coordinate' value type from node tags (only string, number, boolean remain)
2026-01-11 12:49:34 +00:00
Claude d8a0f2abb8 Fix security vulnerabilities and add validation
- Fix XSS vulnerability by using data attributes instead of inline
  onclick handlers in node_tags.html template
- Fix URL injection by using urlencode for all redirect URL parameters
- Add validation to reject moves where source and destination nodes
  are the same (returns 400 Bad Request)
- Add error handling for response.json() calls that may fail
- Add missing test coverage for update endpoint error scenarios
2026-01-11 11:51:57 +00:00
Claude 367f838371 Add admin interface for managing node tags
Implement CRUD operations for NodeTags in the admin interface:

- Add NodeTagMove schema for moving tags between nodes
- Add PUT /nodes/{public_key}/tags/{key}/move API endpoint
- Add web routes at /a/node-tags for tag management
- Create admin templates with node selector and tag management UI
- Support editing, adding, moving, and deleting tags via API calls
- Add comprehensive tests for new functionality

The interface allows selecting a node from a dropdown, viewing its
tags, and performing all CRUD operations including moving a tag
to a different node without having to delete and recreate it.
2026-01-11 01:34:07 +00:00
Louis King 61d6b6287e Add contact cleanup to interface RECEIVER mode
- Add CONTACT_CLEANUP_ENABLED and CONTACT_CLEANUP_DAYS settings
- Implement remove_contact and schedule_remove_contact on device classes
- During contact sync, remove stale contacts from companion node
- Stale contacts (not advertised for > N days) not published to MQTT
- Update Python version to 3.13 across project config
- Remove brittle config tests that assumed default env values
2026-01-08 10:22:27 +00:00
Louis King a290db0491 Updated chart stats 2025-12-08 19:37:45 +00:00
Louis King 9e621c0029 Fixed test 2025-12-08 16:42:13 +00:00
Louis King 57f51c741c Fixed Member model 2025-12-08 15:13:24 +00:00
Louis King 65b8418af4 Fixed last seen issue 2025-12-08 00:15:25 +00:00
Louis King 64ec1a7135 Receiver nodes now sync contacts to MQTT on every advert received 2025-12-07 23:34:33 +00:00
Louis King fbd29ff78e Removed friendly name support and tidied tags 2025-12-07 23:02:19 +00:00
Louis King 84b8614e29 Updates 2025-12-06 21:42:33 +00:00
Louis King 3bc47a33bc Added data retention and node cleanup 2025-12-06 21:27:19 +00:00
Louis King d715e4e4f0 Updates 2025-12-06 13:33:02 +00:00
Claude 6e3b86a1ad Add collector-level event deduplication using content hashes
Replace presentation-layer deduplication with collector-level approach:
- Add event_hash column to messages, advertisements, trace_paths, telemetry tables
- Handlers compute content hashes and skip duplicate events at insertion time
- Use 5-minute time buckets for advertisements and telemetry
- Include Alembic migration for schema changes
2025-12-06 12:23:14 +00:00
Claude c80986fe67 Add event deduplication at presentation layer
When multiple receiver nodes are running, the same mesh events (messages,
advertisements) are reported multiple times. This causes duplicate entries
in the Web UI.

Changes:
- Add hash_utils.py with deterministic hash functions for each event type
- Add `dedupe` parameter to messages and advertisements API endpoints (default: True)
- Update dashboard stats to use distinct counts for messages/advertisements
- Deduplicate recent advertisements and channel messages in dashboard
- Add comprehensive tests for hash utilities

Hash strategy:
- Messages: hash of text + pubkey_prefix + channel_idx + sender_timestamp + txt_type
- Advertisements: hash of public_key + name + adv_type + flags + 5-minute time bucket
2025-12-06 12:08:07 +00:00
Louis King 23f6c290c9 Updates 2025-12-05 21:17:34 +00:00
Claude a4b13d3456 Add member-node association support
Members can now have multiple associated nodes, each with a public_key
and node_role (e.g., 'chat', 'repeater'). This replaces the single
public_key field on members with a one-to-many relationship.

Changes:
- Add MemberNode model for member-node associations
- Update Member model to remove public_key, add nodes relationship
- Update Pydantic schemas with MemberNodeCreate/MemberNodeRead
- Update member_import.py to handle nodes list in seed files
- Update API routes to handle nodes in create/update/read operations
- Add Alembic migration to create member_nodes table and migrate data
- Update example seed file with new format
2025-12-05 20:34:09 +00:00