Replace presentation-layer deduplication with collector-level approach:
- Add event_hash column to messages, advertisements, trace_paths, telemetry tables
- Handlers compute content hashes and skip duplicate events at insertion time
- Use 5-minute time buckets for advertisements and telemetry
- Include Alembic migration for schema changes
When multiple receiver nodes are running, the same mesh events (messages,
advertisements) are reported multiple times. This causes duplicate entries
in the Web UI.
Changes:
- Add hash_utils.py with deterministic hash functions for each event type
- Add `dedupe` parameter to messages and advertisements API endpoints (default: True)
- Update dashboard stats to use distinct counts for messages/advertisements
- Deduplicate recent advertisements and channel messages in dashboard
- Add comprehensive tests for hash utilities
Hash strategy:
- Messages: hash of text + pubkey_prefix + channel_idx + sender_timestamp + txt_type
- Advertisements: hash of public_key + name + adv_type + flags + 5-minute time bucket
Members can now have multiple associated nodes, each with a public_key
and node_role (e.g., 'chat', 'repeater'). This replaces the single
public_key field on members with a one-to-many relationship.
Changes:
- Add MemberNode model for member-node associations
- Update Member model to remove public_key, add nodes relationship
- Update Pydantic schemas with MemberNodeCreate/MemberNodeRead
- Update member_import.py to handle nodes list in seed files
- Update API routes to handle nodes in create/update/read operations
- Add Alembic migration to create member_nodes table and migrate data
- Update example seed file with new format
Replace the channel counts table with actual recent messages per channel:
- Added ChannelMessage schema for channel message summaries
- Dashboard API now fetches latest 5 messages for each channel with sender name lookups
- Network page displays messages grouped by channel with sender names and timestamps
- Only shows channels that have messages
The adv_type from the Advertisement record is often null, but the linked
Node has the correct adv_type. Now falls back to source_node.adv_type
when adv.adv_type is null.
- Swap Time and Type columns (Type now first)
- Add receiver_name and receiver_friendly_name to MessageRead schema
- Update messages API to fetch receiver node names and tags
- Make Receiver column a link showing name with public key prefix
- Rename Name column to Node
- Remove separate Public Key column
- Show name with public key prefix below (like Advertisements list)
- Add whitespace-nowrap to Last Seen column
- Join with Node table to get node names and tags for both source
and receiver nodes
- Display friendly_name (from tags), node_name, or advertised name
with priority in that order
- Show name with public key preview for both Node and Received By columns
- Remove Name and Type columns (usually null)
- Reorder columns: Node first, then Received By, then Time
- Link both Node and Received By to their node detail pages
- Show node name with public key preview when available
- Reduce hero section size and add stats bar with node/message counts
- Add new Advertisements page with public key filtering
- Update hero navigation buttons: Dashboard, Nodes, Advertisements, Messages
- Add Advertisements to main navigation menu
- Remove Hops column from messages list (always empty)
- Display full message text with proper multi-line wrapping
Internal database UUIDs (id, node_id, receiver_node_id) were being
exposed in API responses. These are implementation details that should
not be visible to API consumers. The canonical identifier for nodes
is the 64-char hex public_key.
Changes:
- Remove id, node_id from NodeTagRead, NodeRead schemas
- Remove id from MemberRead schema
- Remove id, receiver_node_id, node_id from MessageRead, AdvertisementRead,
TracePathRead, TelemetryRead schemas
- Update web map component to use public_key instead of member.id
for owner filtering
- Update tests to not assert on removed fields
- Update license from MIT to GPL-3.0-or-later in pyproject.toml
- Update project URLs from meshcore-dev to ipnet-mesh organization
- Add explicit GPL-3.0 license statement to README
- Fix AGENTS.md venv directory reference (.venv vs venv)
- Remove undocumented NETWORK_LOCATION from README
- Fix stats endpoint path in README (/api/v1/dashboard/stats)
- Clarify seed and data directory descriptions in project structure
- Return total_nodes, nodes_with_coords, and error in response
- Display meaningful messages when no nodes or no coordinates found
- Log API errors and node counts for debugging
- Use inline styles for marker colors instead of CSS classes for reliable rendering
- Center map on node locations when data is first loaded
- Refactor filter logic to separate recentering behavior
- Update legend to use inline styles
- Enhanced /map/data endpoint to include node role tag and member ownership
- Added client-side filtering for node type (chat, repeater, room)
- Added toggle to filter for infrastructure nodes only (role: infra)
- Added dropdown filter for member owner (nodes linked via public_key)
- Color-coded markers by node type with gold border for infrastructure
- Added legend showing marker types
- Dynamic count display showing total vs filtered nodes
- Update Docker Compose section: core services run by default (mqtt,
collector, api, web), optional profiles for interfaces and utilities
- Document automatic seeding on collector startup
- Add SEED_HOME environment variable documentation
- Document new Members model and YAML seed file format
- Update node_tags format to YAML with public_key-keyed structure
- Update project structure to reflect seed/ and data/ directories
- Add CLI reference for collector seed commands
Change debug logging to info level so contact processing is visible
in default log output. This helps verify that contact events are
being received and processed correctly.
Instead of sending all contacts in one MQTT message, the interface
now splits the device's contacts response into individual 'contact'
events. This is more consistent with other event patterns and makes
the collector simpler.
Interface changes:
- Add _publish_contacts() to split contacts dict into individual events
- Publish each contact as 'contact' event (not 'contacts')
Collector changes:
- Rename handle_contacts to handle_contact for single contact
- Simplify handler to process one contact per message
- Register handler for 'contact' events
The device sends contact entries with different field names than
originally expected:
- adv_name (not name) for the advertised node name
- type (numeric: 0=none, 1=chat, 2=repeater, 3=room) instead of node_type
Changes:
- Update handle_contacts to extract adv_name and convert numeric type
- Add NODE_TYPE_MAP for type conversion
- Always update node name if different (not just if empty)
- Add debug logging for node updates
- Update ContactInfo schema with actual device fields
Move _setup_event_subscriptions() from run() to connect() so that
event subscriptions are active before get_contacts() is called during
device initialization. Previously, CONTACTS events were lost because
subscriptions weren't set up until run() was called.
Trigger get_contacts() during receiver initialization to fetch the
device's contact database and broadcast CONTACTS events over MQTT.
This enables the collector to associate broadcast node names with
node records in the database.
Changes:
- Add get_contacts() abstract method to BaseMeshCoreDevice
- Implement get_contacts() in MeshCoreDevice using meshcore library
- Implement get_contacts() in MockMeshCoreDevice for testing
- Call get_contacts() in Receiver._initialize_device() after startup
- Replace JSON seed files with YAML format for better readability
- Auto-detect YAML primitive types (number, boolean, string) from values
- Add automatic seed import on collector startup
- Split lat/lon into separate tags instead of combined coordinate string
- Add PyYAML dependency and types-PyYAML for type checking
- Update example/seed and contrib/seed/ipnet with clean YAML format
- Update tests to verify YAML primitive type detection
- Add Member database model with name, callsign, role, description, contact, and public_key fields
- Add Member Pydantic schemas (MemberCreate, MemberUpdate, MemberRead, MemberList)
- Add members table to initial migration
- Add members API endpoints (GET/POST/PUT/DELETE /api/v1/members)
- Add member_import.py for importing from JSON files
- Update web layer to fetch members from API instead of file
- Add SEED_HOME setting (defaults to ./seed) for seed data files
- Add 'collector seed' command to import node_tags.json and members.json
- Rename tags.json to node_tags.json for consistency
- Move example seed data from example/data/* to example/seed/
- Update tests and configuration
- Add test_get_node_with_tags to verify GET /nodes/{pk} includes tags
- Add test_list_nodes_includes_tags to verify GET /nodes includes tags
- Update existing tests to assert tags field is present
The Node API was already correctly returning tags via the
lazy="selectin" relationship loading strategy. These tests
document and verify that behavior.
- Add tag_import.py module with JSON file parsing and database upsert
- Convert collector CLI to group with subcommands for extensibility
- Add 'import-tags' command to import tags from JSON file
- Update docker-compose.yml.example with separated data directories:
- data/collector for tags.json
- data/web for members.json
- Add import-tags Docker service for easy containerized imports
- Add example data files in example/data/collector and example/data/web
- Add comprehensive test coverage (20 tests) for tag import
- Add --e2e flag to pytest to run e2e tests
- E2E tests skip by default with clear message
- Fix type annotations in webhook.py for mypy compliance
- Add proper type hints for comparison operations
- Add WebhookDispatcher for sending events to external services
- Webhook configuration loading from dict config
- JSONPath-like filter expression support for event filtering
- Async HTTP POST sending with httpx
- Retry logic with exponential backoff
- Comprehensive test suite
- Add health check infrastructure for Interface and Collector
- HealthReporter class for periodic status file updates
- CLI commands: meshcore-hub health interface/collector
- Updated Docker Compose to use CLI health checks
- File-based health status for non-HTTP components
- Update TASKS.md progress to 99% (218/221 tasks)
- Remaining 3 tasks are optional (docs/ directory)
Health Checks (6.3):
- Add is_healthy property and get_health_status() to Receiver/Sender
- Add is_healthy property and get_health_status() to Collector Subscriber
- Track device, MQTT, and database connection status
Documentation (6.5):
- Update README with Docker Compose profiles documentation
- Add serial device access instructions
- Update API documentation URLs and add health check info
CI/CD (6.6):
- Add .github/workflows/ci.yml for linting, testing, and building
- Add .github/workflows/docker.yml for Docker image builds
- Support Python 3.11 and 3.12 in CI matrix
- Configure Codecov for coverage reporting
End-to-End Testing (6.7):
- Add tests/e2e/ directory with Docker Compose test configuration
- Add e2e test fixtures with service health waiting
- Add comprehensive e2e tests for API, Web, and auth flows
- Create conftest.py with MockHttpClient for testing web routes
- Add test_home.py with 9 tests for home page
- Add test_members.py with 11 tests for members page and load_members function
- Add test_network.py with 7 tests for network overview page
- Add test_nodes.py with 15 tests for nodes list and detail pages
- Add test_map.py with 12 tests for map page and data endpoint
- Add test_messages.py with 13 tests for messages page with filtering
- All 67 web tests pass, 184 total tests pass
- Update TASKS.md to mark Phase 5 as 100% complete (186/221 total)
The dashboard router was mounted with prefix /dashboard and the HTML
route was also /dashboard, making the full path /api/v1/dashboard/dashboard.
Changed the route to / so it's accessible at /api/v1/dashboard.
Pass _env_file=None to settings classes to prevent pydantic-settings
from loading values from .env files, which would override the default
values the tests are meant to verify.
- Update .flake8 and pre-commit config to properly use flake8 config
- Add B008 to ignored errors (FastAPI Depends pattern)
- Add E402 to ignored errors (intentional module-level imports)
- Remove unused imports from test files and source files
- Fix f-strings without placeholders
- Add type annotations to inner async functions
- Fix SQLAlchemy execute() to use text() wrapper
- Add type: ignore comments for alembic.command imports
- Exclude alembic/ directory from mypy in pre-commit
- Update mypy overrides for test files to not require type annotations
- Fix type annotations for params dicts in web routes
- Fix generator return type in test fixtures
- Return 401 instead of 403 for invalid API keys in require_read
- Add /dashboard prefix to dashboard router so routes are at /api/v1/dashboard/*
- Rename message filter param from 'type' to 'message_type' for clarity
- Add GET /nodes/{public_key}/tags/{key} endpoint for single tag retrieval
- Use meshcore library's self_info property directly after connection
instead of waiting for SELF_INFO event (which was already processed)
- Add --node-address CLI option to override device public key/address
- Support NODE_ADDRESS environment variable
- Add node_address parameter to DeviceConfig and all related functions
CLI fixes:
- Add --mqtt-username and --mqtt-password options to receiver/sender
shortcut commands so they work with authenticated MQTT brokers
- These options read from MQTT_USERNAME/MQTT_PASSWORD env vars
Device integration:
- Integrate with meshcore>=2.2.0 library for actual serial device support
- Implement async-to-sync bridge for meshcore's async API
- Add proper event subscription mapping between meshcore and hub events
- Add meshcore>=2.2.0 to dependencies in pyproject.toml
- Add load_dotenv() call at module import to load .env before Click
processes envvar parameters
- Add python-dotenv>=1.0.0 as explicit dependency in pyproject.toml
Click's envvar only reads from OS environment, not .env files.
By loading dotenv early, all CLI commands now respect .env configuration.
Add web dashboard with FastAPI and Jinja2 templates for visualizing
network status, nodes, messages, and members with an interactive map.
Features:
- FastAPI app with Jinja2 templating and httpx client for API
- Responsive UI using Tailwind CSS with DaisyUI components
- Interactive map with Leaflet.js for node visualization
- Pages: home, network stats, nodes list/detail, messages, map, members
- CLI with extensive configuration (network info, API, members file)
- Development mode with uvicorn auto-reload support
- Add FastAPI application with lifespan management
- Implement bearer token authentication (read/admin levels)
- Create comprehensive REST API routes:
- Nodes: list, get by public key
- Node tags: CRUD operations
- Messages: list with filters, get by ID
- Advertisements: list with filters, get by ID
- Telemetry: list with filters, get by ID
- Trace paths: list with filters, get by ID
- Commands: send message, channel message, advertisement
- Dashboard: stats API and HTML dashboard
- Add API CLI command for running the server
- Create API test suite with 44 passing tests
Routes use proper RESTful status codes (201 Created, 204 No Content).
Authentication is optional - when keys not configured, endpoints are open.
This commit establishes the complete foundation for the MeshCore Hub project:
- Project setup with pyproject.toml (Python 3.11+, all dependencies)
- Development tools: black, flake8, mypy, pytest configuration
- Pre-commit hooks for code quality
- Package structure with all components (interface, collector, api, web)
Common package includes:
- Pydantic settings for all component configurations
- SQLAlchemy models for nodes, messages, advertisements, traces, telemetry
- Pydantic schemas for events, API requests/responses, commands
- MQTT client utilities with topic builder
- Logging configuration
Database infrastructure:
- Alembic setup with initial migration for all tables
- Database manager with session handling
CLI entry point:
- Click-based CLI with subcommands for all components
- Database migration commands (upgrade, downgrade, revision)
Tests:
- Basic test suite for config and models
- pytest fixtures for in-memory database testing
Claude