Adverts/Messages rows now link directly to the deduplicated packet-detail
page (/packets/hash/:hash). Each path hop renders as a clickable badge
opening a popover that looks up nodes by public-key prefix via the new
pubkey_prefix query param on GET /api/v1/nodes (case-insensitive
startswith). Adds a derived path_hash_bytes field on GroupedPacketRead.
Defaults changed: FEATURE_PACKETS now defaults to true and
RAW_PACKET_RETENTION_DAYS to 7 (independent of DATA_RETENTION_DAYS).
Fixes a mypy arg-type error by explicitly annotating the packet_hash
list as list[str].
Add tests covering the previously-uncovered new lines flagged by Codecov:
- /packets route: search, packet_type, channel_idx, route_type, observed_by,
decryptable (both), max_snr, path-len ranges, since/until, observer-tag
hydration, detail 404, ascending sort, and the role-aware cache key builder.
- store_raw_packet: existing-observer update branch, senderPublicKey source
fallback, and the no-source-prefix case.
- normalizer: advertisement payload carries the wire packet_hash.
- CollectorSettings: raw-packet retention default/override and capture default.
Also stub the test decoder via setattr so mypy's method-assign analysis is
deterministic across incremental-cache states.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three related performance fixes for slow (>500ms) API responses.
1. Stop blocking the event loop. Route handlers were declared `async def`
but ran synchronous SQLAlchemy queries (and synchronous Redis calls via
the cache decorator) directly on the event loop, serializing requests.
Convert all handlers to sync `def` so FastAPI runs them in its
threadpool, and make the `@cached` decorator dual-mode (sync wrapper for
sync handlers, async wrapper preserved for a future async/Postgres path).
2. Tune SQLite for concurrency. Enable WAL, busy_timeout and
synchronous=NORMAL on every connection, and size the pool above the
threadpool so handlers don't wait on connections. In-memory SQLite is
guarded (no overflow-pool kwargs).
3. Precompute an indexed `nodes.is_observer` flag. The `observer=true`
filter scanned ~68k+ event rows to find a handful of observers (the
advertisements page calls it with limit=500). Replace the 5-way OR of
subqueries with `WHERE nodes.is_observer = ?`. The collector sets the
flag on first observation (in add_event_observer); the cleanup job
clears it once a node's events are all pruned; a migration adds the
column/index and backfills from the existing union.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace single NETWORK_RADIO_CONFIG comma-delimited string with six
individual environment variables: NETWORK_RADIO_PROFILE, _FREQUENCY,
_BANDWIDTH, _SPREADING_FACTOR, _CODING_RATE, _TX_POWER
- Radio config fields now use raw numeric types (float/int) with units
applied dynamically via RadioConfig.format_for_display()
- Add FEATURE_RADIO_CONFIG feature flag to control radio config panel
visibility on the home page (default: enabled)
- Remove from_config_string class method (no backwards compatibility)
- Update Click CLI options, create_app() signature, and _build_config_json()
- Update docker-compose.yml, .env.example, README.md, AGENTS.md
- Add upgrading.md v0.12.0 section with migration instructions
- Add test coverage for schema, config, and feature flag
- Rename ChannelVisibility.PUBLIC to ChannelVisibility.COMMUNITY
- Update stored value from 'public' to 'community' across model, schema, API, CLI, and frontend
- Add Alembic migration to update existing database rows
- Consolidate upgrade docs: merge v0.11.0, v0.12.0, v0.13.0 into single v0.11.0 section
- Add i18n visibility level translation keys (en, nl)
- Update section headings on channels page to use t() for i18n
- Keep visibility badges lowercase per UI design
- Group channel cards by visibility with section headings
- Move channels before messages in all nav menus for logical grouping
- Add optgroup labels (Standard/Custom) to message channel filter
- Capitalize built-in "Test" channel name for consistency
- Shorten "Advertisements" to "Adverts" in UI labels
- Lay out channel cards with side-by-side QR codes
- Shrink homepage nav cards for better fit
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
Track advertisement route type (flood/transport_flood/direct/transport_direct)
and node advert timestamp to distinguish zero-hop from flood adverts, improve
deduplication with 300s buckets, and default all dashboard/ad-API queries to
flood-only (including NULL for historical records).
- Fix Members page badge showing higher count than displayed profiles
by counting operators + members instead of all profiles
- Filter dropdowns on Nodes, Advertisements, and Map pages to show
only operators (since only operators can adopt nodes)
- Add roles field to /map/data profiles for client-side filtering
- Add all_operators and filter_operator_label i18n keys (en, nl)
- Fix flash banner test isolation from .env NETWORK_ANNOUNCEMENT
Add NETWORK_ANNOUNCEMENT env var that displays a dismissible flash banner
on every page when set. Announcement text supports Markdown (bold, italic,
links, inline code) rendered to HTML server-side at startup.
Replace the dedicated admin tag management page with inline tag editing
on the node detail page. Operators can now edit tags directly on nodes
they've adopted; admins retain unrestricted access.
Key changes:
- Remove admin SPA page (admin/index.js, admin/node-tags.js)
- Add inline tag editor to node-detail.js with add/edit/delete modals
- Replace RequireAdmin with RequireOperatorOrAdmin for tag API routes
- Add ownership check: operators restricted to adopted nodes only
- Add validate_and_coerce_tag_value for number/boolean coercion
- Remove unused bulk endpoints (copy, move, replace all)
- Use AbortController for event listeners to prevent accumulation
on lit-html DOM reuse across re-renders
- Track Leaflet map instance at module scope for defensive cleanup
- Fix checkAuthResponse to only redirect on 401 (not 403)
- Update tests for new OIDC-based auth model
- Update en.json locale, i18n.md, upgrading.md, AGENTS.md
Remove the static Member model/table, CRUD API, YAML seed files, and
admin UI. Replace with UserProfile-driven members page that reads roles
from OIDC identity provider. Key changes:
- Drop members table, add roles column to user_profiles (Alembic migration)
- Add GET /api/v1/user/profiles (paginated, no user_id exposed)
- Add GET /api/v1/user/profile/me (auto-creates profile for current user)
- Replace member_id node tag filter with adopted_by (profile UUID)
- Members page now shows profiles grouped by operator/member roles
- Profile page supports public view (/profile/:id) and owner edit (/profile)
- Node detail page shows adoption card side-by-side with public key card
- Auto-create user profile during OIDC login callback
- Hide Adopted Nodes section for non-operator/admin users
- Add member since date to profile cards
- Add role badges and adopted node badges to member tiles
- Add antenna/users icons to Members page group headers
- Add path_len column to event_observers table with Alembic migration
- Create shared observer_utils.py for building observer responses
- Extract SNR and path_len from LetsMesh uploads via normalizer
- Pass snr/path_len from all 4 handlers to add_event_observer()
- Add expandable observer detail sub-tables to messages and ads pages
- Rename Receivers column to Observers with updated i18n key
- Add satellite dish icon to observer badges and detail rows
- Add mobile card observer detail toggle
- Show full datetime tooltip on hover for relative times
- Hide Path column from advertisement observer tables (not populated)
The LetsMesh normalizer stored public keys as UPPERCASE while the tag
importer stored them as lowercase, creating duplicate nodes for the same
device. Normalize all public keys to lowercase throughout:
- MQTT topic parsing (event, command, LetsMesh upload)
- LetsMesh normalizer output
- Node model __init__ enforcement
- Alembic migration to merge duplicates and normalize existing data
Simplify the variable name to remove the legacy LetsMesh decoder prefix.
Also fix unparenthesized except tuples in web/app.py and promote the
parenthesized-exception rule to a prominent position in AGENTS.md.
Remove the meshcore_interface component in favor of external
meshcore-packet-capture for data ingestion. Rename receiver_node_id
to observer_node_id across all models, schemas, handlers, and API
routes. Add Alembic migration for the column/table renames. Fix
frontend JS property name mismatch that prevented the Receiver column
from displaying observer data.
Update path hash handling to accept variable-length hex-encoded hashes
(e.g. "4a" for single-byte, "b3fa" for multibyte) instead of requiring
exactly 2-character hashes. Bump meshcore dependency to >=2.3.0.
- Update normalizer to accept even-length hex strings >= 2 chars
- Update schemas and model docstrings for variable-length hashes
- Add tests for multibyte and mixed-length path hash round-trips
- Fix web test flakiness from local .env datetime locale leaking
## i18n Refactoring
- Refactor admin translations to use common composable patterns
- Add common patterns: delete_entity_confirm, entity_added_success, move_entity_to_another_node, etc.
- Remove 18 duplicate keys from admin_members and admin_node_tags sections
- Update all admin JavaScript files to use new common patterns with dynamic entity composition
- Fix label consistency: rename first_seen to first_seen_label to match naming convention
## Translation Documentation
- Create comprehensive translation reference guide (languages.md) with 200+ documented keys
- Add translation architecture documentation to AGENTS.md with examples and best practices
- Add "Help Translate" call-to-action section in README with link to translation guide
- Add i18n feature to README features list
## Documentation Audit
- Add undocumented config options: API_KEY, WEB_LOCALE, WEB_DOMAIN to README and .env.example
- Fix outdated CLI syntax: interface --mode receiver → interface receiver
- Update database migration commands to use CLI wrapper (meshcore-hub db) instead of direct alembic
- Add static/locales/ directory to project structure section
- Add i18n configuration (WEB_LOCALE, WEB_THEME) to docker-compose.yml
## Testing
- All 438 tests passing
- All pre-commit checks passing (black, flake8, mypy)
- Added tests for new common translation patterns
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Replace node type badge with icon in admin tag editor
- Add Edit/Add Tags button on node detail page (when admin enabled and authenticated)
- Remove automatic seed container startup to prevent overwriting user changes
- Remove unused 'coordinate' value type from node tags (only string, number, boolean remain)
- Add CONTACT_CLEANUP_ENABLED and CONTACT_CLEANUP_DAYS settings
- Implement remove_contact and schedule_remove_contact on device classes
- During contact sync, remove stale contacts from companion node
- Stale contacts (not advertised for > N days) not published to MQTT
- Update Python version to 3.13 across project config
- Remove brittle config tests that assumed default env values
Replace presentation-layer deduplication with collector-level approach:
- Add event_hash column to messages, advertisements, trace_paths, telemetry tables
- Handlers compute content hashes and skip duplicate events at insertion time
- Use 5-minute time buckets for advertisements and telemetry
- Include Alembic migration for schema changes
When multiple receiver nodes are running, the same mesh events (messages,
advertisements) are reported multiple times. This causes duplicate entries
in the Web UI.
Changes:
- Add hash_utils.py with deterministic hash functions for each event type
- Add `dedupe` parameter to messages and advertisements API endpoints (default: True)
- Update dashboard stats to use distinct counts for messages/advertisements
- Deduplicate recent advertisements and channel messages in dashboard
- Add comprehensive tests for hash utilities
Hash strategy:
- Messages: hash of text + pubkey_prefix + channel_idx + sender_timestamp + txt_type
- Advertisements: hash of public_key + name + adv_type + flags + 5-minute time bucket
- Replace JSON seed files with YAML format for better readability
- Auto-detect YAML primitive types (number, boolean, string) from values
- Add automatic seed import on collector startup
- Split lat/lon into separate tags instead of combined coordinate string
- Add PyYAML dependency and types-PyYAML for type checking
- Update example/seed and contrib/seed/ipnet with clean YAML format
- Update tests to verify YAML primitive type detection
- Add Member database model with name, callsign, role, description, contact, and public_key fields
- Add Member Pydantic schemas (MemberCreate, MemberUpdate, MemberRead, MemberList)
- Add members table to initial migration
- Add members API endpoints (GET/POST/PUT/DELETE /api/v1/members)
- Add member_import.py for importing from JSON files
- Update web layer to fetch members from API instead of file
- Add SEED_HOME setting (defaults to ./seed) for seed data files
- Add 'collector seed' command to import node_tags.json and members.json
- Rename tags.json to node_tags.json for consistency
- Move example seed data from example/data/* to example/seed/
- Update tests and configuration
Pass _env_file=None to settings classes to prevent pydantic-settings
from loading values from .env files, which would override the default
values the tests are meant to verify.
- Update .flake8 and pre-commit config to properly use flake8 config
- Add B008 to ignored errors (FastAPI Depends pattern)
- Add E402 to ignored errors (intentional module-level imports)
- Remove unused imports from test files and source files
- Fix f-strings without placeholders
- Add type annotations to inner async functions
- Fix SQLAlchemy execute() to use text() wrapper
- Add type: ignore comments for alembic.command imports
- Exclude alembic/ directory from mypy in pre-commit
- Update mypy overrides for test files to not require type annotations
- Fix type annotations for params dicts in web routes
- Fix generator return type in test fixtures
This commit establishes the complete foundation for the MeshCore Hub project:
- Project setup with pyproject.toml (Python 3.11+, all dependencies)
- Development tools: black, flake8, mypy, pytest configuration
- Pre-commit hooks for code quality
- Package structure with all components (interface, collector, api, web)
Common package includes:
- Pydantic settings for all component configurations
- SQLAlchemy models for nodes, messages, advertisements, traces, telemetry
- Pydantic schemas for events, API requests/responses, commands
- MQTT client utilities with topic builder
- Logging configuration
Database infrastructure:
- Alembic setup with initial migration for all tables
- Database manager with session handling
CLI entry point:
- Click-based CLI with subcommands for all components
- Database migration commands (upgrade, downgrade, revision)
Tests:
- Basic test suite for config and models
- pytest fixtures for in-memory database testing
Louis King