Planning doc for adding Postgres support: code compatibility fixes,
a postgres container, component-based connection config, and a
SQLAlchemy ORM-based migration command for existing SQLite databases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The observed_by filter on messages, advertisements, telemetry, and
trace_paths matched only the first observer (stored in observer_node_id),
silently excluding events whose secondary observers appear only in the
event_observers junction table. This caused filtered lists to appear
'several hours behind' when a dominant observer consistently won the
first-insert race for recent events.
Replace the ObserverNode.public_key predicate with an IN subquery against
the event_observers junction table (the canonical multi-observer source
already used for display). Add a shared observed_by_filter_clause() helper
in observer_utils.py to avoid duplication across all four routes.
Add regression tests proving a secondary observer (present only in
event_observers) sees events via the filter. Update existing fixtures and
inline test data to seed event_hash and EventObserver rows.
Fixes#239
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace single NETWORK_RADIO_CONFIG comma-delimited string with six
individual environment variables: NETWORK_RADIO_PROFILE, _FREQUENCY,
_BANDWIDTH, _SPREADING_FACTOR, _CODING_RATE, _TX_POWER
- Radio config fields now use raw numeric types (float/int) with units
applied dynamically via RadioConfig.format_for_display()
- Add FEATURE_RADIO_CONFIG feature flag to control radio config panel
visibility on the home page (default: enabled)
- Remove from_config_string class method (no backwards compatibility)
- Update Click CLI options, create_app() signature, and _build_config_json()
- Update docker-compose.yml, .env.example, README.md, AGENTS.md
- Add upgrading.md v0.12.0 section with migration instructions
- Add test coverage for schema, config, and feature flag
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
Track advertisement route type (flood/transport_flood/direct/transport_direct)
and node advert timestamp to distinguish zero-hop from flood adverts, improve
deduplication with 300s buckets, and default all dashboard/ad-API queries to
flood-only (including NULL for historical records).
The node detail page sends a public_key query param when fetching
recent advertisements, but the API silently ignored it (FastAPI
ignores unknown query params). This returned the 10 most recent ads
network-wide instead of for the specific node, making all dates
appear as today.
The async SQLAlchemy engine was missing PRAGMA foreign_keys=ON, causing
ondelete="CASCADE" constraints to be silently ignored when the collector
deleted inactive nodes. This left orphaned rows in user_profile_nodes,
event_observers, and node_tags, which crashed the API with AttributeError
when accessing assoc.node.public_key on null relationships.
- Add FK PRAGMA listener to async engine (database.py)
- Add null-guard in _build_adopted_nodes() and refactor list_profiles()
- Add cleanup_orphaned_node_relations() covering all 3 dependent tables
- Integrate orphan cleanup into scheduled retention cycle (subscriber.py)
- Add --node-cleanup/--node-cleanup-days flags to CLI cleanup command
- Fix truncate cascade warning to include user_profile_nodes/event_observers
- Add FK PRAGMA to test fixtures for cascade verification
- Add upgrade note to docs/upgrading.md
- Fix Members page badge showing higher count than displayed profiles
by counting operators + members instead of all profiles
- Filter dropdowns on Nodes, Advertisements, and Map pages to show
only operators (since only operators can adopt nodes)
- Add roles field to /map/data profiles for client-side filtering
- Add all_operators and filter_operator_label i18n keys (en, nl)
- Fix flash banner test isolation from .env NETWORK_ANNOUNCEMENT
Add NETWORK_ANNOUNCEMENT env var that displays a dismissible flash banner
on every page when set. Announcement text supports Markdown (bold, italic,
links, inline code) rendered to HTML server-side at startup.
Add OIDC_ROLE_TEST config var (default: 'test') to exclude test users
from dashboard stats, member counts, and the Members page. Uses
server-side filtering with exclude_test query param (default: true) and
client-side defense-in-depth filter in members.js.
- Add oidc_role_test to WebSettings in config.py
- Exclude test users from operator/member count queries in dashboard.py
- Add exclude_test param to GET /api/v1/user/profiles in user_profiles.py
- Filter test users client-side in members.js via role_names.test config
- Wire oidc_role_test into app.state and frontend config in web/app.py
- Document OIDC_ROLE_TEST in AGENTS.md and .env.example
The web proxy's endpoint access mapping previously required one of
(admin/operator/member) roles for PUT /api/v1/user/profile. This
blocked OIDC users with no assigned roles from saving their own profile.
Add an _AUTHENTICATED sentinel access level that grants access to any
logged-in user regardless of roles, and apply it to the profile PUT
endpoint. The API layer already enforces owner-only checks via
RequireUserOwner, so the proxy role gate was redundant.
- Add .prose > :first-child margin-top:0 to eliminate double-spacing before headings
- Add nested list CSS (circle/square bullets, lower-alpha/roman numbering)
- Replace map popup <p>/space-y-1 layout with CSS grid for aligned label-value pairs
- Convert map filter card to collapsible <details> with state persistence
- Add nested list HTML output tests and markdown features docs
Replace the label:value list in the Network Info panel with a 3×2 grid of
compact tiles, each showing a themed cyan icon, label, and value. Adds 5 new
SVG icon functions and a --color-radio CSS custom property with dark/light variants.
Move mobile hamburger menu from navbar-start to navbar-end so it stays
right-aligned regardless of logo width. Mobile nav items are now rendered
via JS (renderMobileNav) using icon functions instead of duplicated inline
SVGs. Increase dropdown widths (w-52→w-56), icon sizes (h-4→h-5), and add
larger mobile tap targets via CSS media query.
- Change API and frontend default sort from name/asc to last_seen/desc
- Add mobileSortSelect() shared component for native select dropdown
- Add mobile sort select to nodes, advertisements, and messages pages
- Add i18n sort labels for all three list pages
- Update sort tests for new default with staggered timestamps
Bundle SPA JavaScript with esbuild for production builds, generating
content-hashed filenames for immutable caching. Vendor assets (Leaflet,
Chart.js, QRCode.js) get SHA256-based query params. Locale JSON files
get a combined hash version. Falls back to unbuiltsources when dist/
is absent.
Add sort/order query parameters to Nodes, Advertisements, and Messages
API endpoints. Nodes default to alpha-by-name (via COALESCE of name tag,
node name, public key). Ads and Messages default to newest-first.
Frontend adds sortableTableHeader() component with asc/desc toggle
indicators. Sort state is preserved in URL params, surviving
auto-refresh and pagination.
- Replace removed form-control/label-text/label-text-alt with DaisyUI v5
fieldset/fieldset-label equivalents in node-detail.js and map.js
- Switch JS error display from innerHTML+label-text-alt to
textContent+classList hidden toggling
- Redesign home hero navigation buttons as square cards with section
colors, border hover animation, and centered icons
- Move custom page links to a separate row with compact btn-outline style
- Remove deprecated btn-outline CSS overrides from app.css
- Add observer multi-select (<select multiple size=2>) to Advertisements and
Messages filter bars, populated from /api/v1/nodes?observer=true
- Make all filter sections collapsible via <details> on Nodes, Advertisements,
and Messages pages; collapsed by default, auto-expands when active filters
exist, preserves open state across auto-refresh ticks
- Add backend observer=true|false query param to GET /api/v1/nodes for
observer-only or non-observer-only node filtering via subquery
- Change observed_by in Advertisements/Messages API from single public_key
to list[str] with .in_() for multi-select support
- Fix router.js and api.js to handle array query params (duplicate keys
promoted to arrays, .append() per element)
- Fix createFilterHandler to use FormData.getAll() for multi-value support
- Replace DaisyUI form-control/label/label-text classes with Tailwind-native
equivalents (flex flex-col gap-1, flex items-center py-1, opacity-80 text-sm)
since DaisyUI CSS is tree-shaken from the build output
- Thicker collapsible border (border-2 border-base-content/25) visible in
both light and dark themes
- Bottom-align Filter/Clear buttons via two-row form layout
- Move Observer filter to last position on Advertisements page
- Add filter_observer_label i18n key
- Add tests for observer=true node filtering and multi observer params
Replace the dedicated admin tag management page with inline tag editing
on the node detail page. Operators can now edit tags directly on nodes
they've adopted; admins retain unrestricted access.
Key changes:
- Remove admin SPA page (admin/index.js, admin/node-tags.js)
- Add inline tag editor to node-detail.js with add/edit/delete modals
- Replace RequireAdmin with RequireOperatorOrAdmin for tag API routes
- Add ownership check: operators restricted to adopted nodes only
- Add validate_and_coerce_tag_value for number/boolean coercion
- Remove unused bulk endpoints (copy, move, replace all)
- Use AbortController for event listeners to prevent accumulation
on lit-html DOM reuse across re-renders
- Track Leaflet map instance at module scope for defensive cleanup
- Fix checkAuthResponse to only redirect on 401 (not 403)
- Update tests for new OIDC-based auth model
- Update en.json locale, i18n.md, upgrading.md, AGENTS.md
Replace the role=infra NodeTag convention with UserProfileNode adoption
as the canonical infrastructure indicator across map, Prometheus metrics,
and alerting. Renames is_infra to is_adopted, infra_center to
adopted_center. Map icons change to blue (adopted) / green (normal),
with all adoption UI gated on OIDC_ENABLED. Adds meshcore_nodes_adopted
gauge and Alembic migration to clean up obsolete tags.
Remove the static Member model/table, CRUD API, YAML seed files, and
admin UI. Replace with UserProfile-driven members page that reads roles
from OIDC identity provider. Key changes:
- Drop members table, add roles column to user_profiles (Alembic migration)
- Add GET /api/v1/user/profiles (paginated, no user_id exposed)
- Add GET /api/v1/user/profile/me (auto-creates profile for current user)
- Replace member_id node tag filter with adopted_by (profile UUID)
- Members page now shows profiles grouped by operator/member roles
- Profile page supports public view (/profile/:id) and owner edit (/profile)
- Node detail page shows adoption card side-by-side with public key card
- Auto-create user profile during OIDC login callback
- Hide Adopted Nodes section for non-operator/admin users
- Add member since date to profile cards
- Add role badges and adopted node badges to member tiles
- Add antenna/users icons to Members page group headers
- Add OPTIONS to the web API proxy route methods for CORS preflight support
- Fix event listener accumulation in admin/node-tags.js and admin/members.js
using AbortController with cleanup functions returned to the SPA router.
lit-html reuses DOM elements across re-renders, causing addEventListener
calls to accumulate and fire multiple times per form submission.
- Rename admin routes from /a/ prefix to /admin/ for clarity
- Add debug logging for admin route access and OIDC role checks
- Move auth section in navbar after theme toggle
- Update tests and AGENTS.md accordingly
- Add path_len column to event_observers table with Alembic migration
- Create shared observer_utils.py for building observer responses
- Extract SNR and path_len from LetsMesh uploads via normalizer
- Pass snr/path_len from all 4 handlers to add_event_observer()
- Add expandable observer detail sub-tables to messages and ads pages
- Rename Receivers column to Observers with updated i18n key
- Add satellite dish icon to observer badges and detail rows
- Add mobile card observer detail toggle
- Show full datetime tooltip on hover for relative times
- Hide Path column from advertisement observer tables (not populated)
Louis King