Add an optional, off-by-default spam-detection feature that scores each
message's spam likelihood at ingest, stores the score on the row, and lets
the display layer hide likely-spam by default behind a "show potential spam"
toggle. Nothing is ever dropped at ingest, so the threshold can be retuned
without reprocessing.
Scoring (collector/spam.py): windowed COUNT(*) over new
(path_prefix, received_at) and (sender_normalized, received_at) indexes —
joint path+sender signal plus a sender-name signal (trailing-digit suffix
stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or
absent, the name signal stands alone at full weight so local spam is still
flaggable. A background sweep re-scores recent rows with hindsight to catch
the leading edge of bursts. The collector logs each score (WARNING at/above
the threshold).
Display: the messages API gains include_spam and a master-switch-aware
hide-filter; the SPA shows the toggle + a badge only when the feature is on.
Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in
Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors
the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off.
Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration
for the three new columns + two indexes, and backend-aware collector test
fixtures (lifted db_backend/db_url into the shared conftest).
Also: move the meshcore-hub image pull_policy out of the base compose file.
It lived in docker-compose.yml as pull_policy: daily and made `make up` pull
the published image over a freshly built local one. Base is now policy-neutral
(default missing); dev sets pull_policy: build on the hub services so it only
ever uses local builds. Prod refreshes images via a manual `docker compose
... pull`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dashboard charts (activity, message-activity, node-count) rendered as
flat zeros on Postgres because func.date() returns a str on SQLite but
a datetime.date on Postgres — the dict lookup by string key always
missed. Fixed with a dialect-neutral _date_bucket_key() helper and
pinned the Postgres session timezone to UTC at the engine level.
Also adds dual-backend test infrastructure (TEST_DATABASE_BACKEND env
var), per-worker Postgres databases for pytest-xdist isolation, and
strengthened regression tests asserting non-zero date buckets.
After the Postgres migration, nodes with no last_seen timestamp floated
to the top of the default list because Postgres sorts NULLs first under
ORDER BY ... DESC, whereas SQLite (the previous backend) sorts them last.
Wrap the last_seen ORDER BY with SQLAlchemy nullslast() so NULL last_seen
nodes always sink to the end regardless of database backend or sort
direction. Adds three regression tests covering DESC, ASC, and all-NULL
cases.
The SPA reaches the backend via the web api_proxy, which forwarded query
params using dict(request.query_params). dict() on Starlette's QueryParams
multidict keeps only the last value of a repeated key, so a multi-valued
filter like ?observed_by=A&observed_by=B was forwarded to the backend as
observed_by=B only. The backend then filtered to B's events, making a
message observed only by A disappear as soon as B was also selected — the
reported "filters act like AND" symptom. This happens independently of the
Redis response cache.
Forward request.query_params.multi_items() (a list of (key, value) tuples)
so all repeated values reach the backend. Add web proxy regression tests
asserting both observed_by values are forwarded, and capture forwarded
params in the MockHttpClient.
This is the primary fix; the earlier cache-key change (multi_items in
sorted_query_string) addressed the same collapse pattern at the cache layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The API response cache key was built from request.query_params.items(),
which in Starlette keeps only the last value of a repeated key. Repeated
params like observed_by (?observed_by=A&observed_by=B) therefore collapsed
to observed_by=B in the key, colliding with any other filter set sharing the
same last value. The first response cached under that key was then served
for the whole TTL, making observer-filtered Messages/Adverts return stale,
wrong results (e.g. an A-only message vanishing when B was also enabled).
Use multi_items() so all repeated values are included, sorted by the full
(key, value) tuple so order-independent filter sets map to one key. Add
regression tests covering preservation, order-independence, and the
{A,B} vs {B} collision case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Messages containing runs of newlines were preserved by the pre-wrap
styling on the messages view, stretching table rows and cards. Collapse
any run of newlines (and surrounding whitespace) into a single space at
display time, leaving stored text untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add two operator-controlled, startup-time settings:
- SYSTEM_ANNOUNCEMENT: non-dismissable Markdown banner rendered above the
network announcement on every page (navbar -> system -> network).
- SYSTEM_MAINTENANCE: when enabled, forces all feature flags off so the nav
collapses to Home, hides the profile menu, and the SPA renders a maintenance
page for every route. The maintenance page makes no API calls, so the API
and database can be offline while the web component keeps running.
CLI exposes --system-announcement and tri-state --system-maintenance; the bool
falls back to pydantic settings to parse SYSTEM_MAINTENANCE reliably from env.
Adds i18n strings (en/nl), tests, and docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the multi-select Observer dropdown buried in the filter panel with
a row of clickable observer badges rendered between the filter panel and the
data list (and below the Sorting dropdown on mobile).
- Selection is stored in localStorage (shared across Adverts and Messages)
as the disabled set, so new observers default to enabled.
- Badge style reflects enabled (filled) vs disabled (muted) state; the last
enabled observer cannot be toggled off.
- Observer filter is sourced from localStorage instead of the URL query;
a two-phase fetch resolves the enabled include-list (the API filters by
inclusion only) before fetching data, with no flash of unfiltered results.
- Toggling re-scopes data and resets to page 1.
- Add enable/disable tooltip strings (en/nl) and the previously-missing
Dutch observer label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the DATABASE_BACKEND explicit switch, schema-per-instance isolation
via search_path for shared-cluster prod/stg, the no-admin-credentials
provisioning decision, the migrate-to-postgres command internals, the
operator runbook, and the phased implementation order with SQLite gates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Planning doc for adding Postgres support: code compatibility fixes,
a postgres container, component-based connection config, and a
SQLAlchemy ORM-based migration command for existing SQLite databases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The observed_by filter on messages, advertisements, telemetry, and
trace_paths matched only the first observer (stored in observer_node_id),
silently excluding events whose secondary observers appear only in the
event_observers junction table. This caused filtered lists to appear
'several hours behind' when a dominant observer consistently won the
first-insert race for recent events.
Replace the ObserverNode.public_key predicate with an IN subquery against
the event_observers junction table (the canonical multi-observer source
already used for display). Add a shared observed_by_filter_clause() helper
in observer_utils.py to avoid duplication across all four routes.
Add regression tests proving a secondary observer (present only in
event_observers) sees events via the filter. Update existing fixtures and
inline test data to seed event_hash and EventObserver rows.
Fixes#239
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace single NETWORK_RADIO_CONFIG comma-delimited string with six
individual environment variables: NETWORK_RADIO_PROFILE, _FREQUENCY,
_BANDWIDTH, _SPREADING_FACTOR, _CODING_RATE, _TX_POWER
- Radio config fields now use raw numeric types (float/int) with units
applied dynamically via RadioConfig.format_for_display()
- Add FEATURE_RADIO_CONFIG feature flag to control radio config panel
visibility on the home page (default: enabled)
- Remove from_config_string class method (no backwards compatibility)
- Update Click CLI options, create_app() signature, and _build_config_json()
- Update docker-compose.yml, .env.example, README.md, AGENTS.md
- Add upgrading.md v0.12.0 section with migration instructions
- Add test coverage for schema, config, and feature flag
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
Track advertisement route type (flood/transport_flood/direct/transport_direct)
and node advert timestamp to distinguish zero-hop from flood adverts, improve
deduplication with 300s buckets, and default all dashboard/ad-API queries to
flood-only (including NULL for historical records).
The node detail page sends a public_key query param when fetching
recent advertisements, but the API silently ignored it (FastAPI
ignores unknown query params). This returned the 10 most recent ads
network-wide instead of for the specific node, making all dates
appear as today.
The async SQLAlchemy engine was missing PRAGMA foreign_keys=ON, causing
ondelete="CASCADE" constraints to be silently ignored when the collector
deleted inactive nodes. This left orphaned rows in user_profile_nodes,
event_observers, and node_tags, which crashed the API with AttributeError
when accessing assoc.node.public_key on null relationships.
- Add FK PRAGMA listener to async engine (database.py)
- Add null-guard in _build_adopted_nodes() and refactor list_profiles()
- Add cleanup_orphaned_node_relations() covering all 3 dependent tables
- Integrate orphan cleanup into scheduled retention cycle (subscriber.py)
- Add --node-cleanup/--node-cleanup-days flags to CLI cleanup command
- Fix truncate cascade warning to include user_profile_nodes/event_observers
- Add FK PRAGMA to test fixtures for cascade verification
- Add upgrade note to docs/upgrading.md
- Fix Members page badge showing higher count than displayed profiles
by counting operators + members instead of all profiles
- Filter dropdowns on Nodes, Advertisements, and Map pages to show
only operators (since only operators can adopt nodes)
- Add roles field to /map/data profiles for client-side filtering
- Add all_operators and filter_operator_label i18n keys (en, nl)
- Fix flash banner test isolation from .env NETWORK_ANNOUNCEMENT
Add NETWORK_ANNOUNCEMENT env var that displays a dismissible flash banner
on every page when set. Announcement text supports Markdown (bold, italic,
links, inline code) rendered to HTML server-side at startup.
Add OIDC_ROLE_TEST config var (default: 'test') to exclude test users
from dashboard stats, member counts, and the Members page. Uses
server-side filtering with exclude_test query param (default: true) and
client-side defense-in-depth filter in members.js.
- Add oidc_role_test to WebSettings in config.py
- Exclude test users from operator/member count queries in dashboard.py
- Add exclude_test param to GET /api/v1/user/profiles in user_profiles.py
- Filter test users client-side in members.js via role_names.test config
- Wire oidc_role_test into app.state and frontend config in web/app.py
- Document OIDC_ROLE_TEST in AGENTS.md and .env.example
The web proxy's endpoint access mapping previously required one of
(admin/operator/member) roles for PUT /api/v1/user/profile. This
blocked OIDC users with no assigned roles from saving their own profile.
Add an _AUTHENTICATED sentinel access level that grants access to any
logged-in user regardless of roles, and apply it to the profile PUT
endpoint. The API layer already enforces owner-only checks via
RequireUserOwner, so the proxy role gate was redundant.
- Add .prose > :first-child margin-top:0 to eliminate double-spacing before headings
- Add nested list CSS (circle/square bullets, lower-alpha/roman numbering)
- Replace map popup <p>/space-y-1 layout with CSS grid for aligned label-value pairs
- Convert map filter card to collapsible <details> with state persistence
- Add nested list HTML output tests and markdown features docs
Replace the label:value list in the Network Info panel with a 3×2 grid of
compact tiles, each showing a themed cyan icon, label, and value. Adds 5 new
SVG icon functions and a --color-radio CSS custom property with dark/light variants.
Move mobile hamburger menu from navbar-start to navbar-end so it stays
right-aligned regardless of logo width. Mobile nav items are now rendered
via JS (renderMobileNav) using icon functions instead of duplicated inline
SVGs. Increase dropdown widths (w-52→w-56), icon sizes (h-4→h-5), and add
larger mobile tap targets via CSS media query.
- Change API and frontend default sort from name/asc to last_seen/desc
- Add mobileSortSelect() shared component for native select dropdown
- Add mobile sort select to nodes, advertisements, and messages pages
- Add i18n sort labels for all three list pages
- Update sort tests for new default with staggered timestamps
Bundle SPA JavaScript with esbuild for production builds, generating
content-hashed filenames for immutable caching. Vendor assets (Leaflet,
Chart.js, QRCode.js) get SHA256-based query params. Locale JSON files
get a combined hash version. Falls back to unbuiltsources when dist/
is absent.
Add sort/order query parameters to Nodes, Advertisements, and Messages
API endpoints. Nodes default to alpha-by-name (via COALESCE of name tag,
node name, public key). Ads and Messages default to newest-first.
Frontend adds sortableTableHeader() component with asc/desc toggle
indicators. Sort state is preserved in URL params, surviving
auto-refresh and pagination.
- Replace removed form-control/label-text/label-text-alt with DaisyUI v5
fieldset/fieldset-label equivalents in node-detail.js and map.js
- Switch JS error display from innerHTML+label-text-alt to
textContent+classList hidden toggling
- Redesign home hero navigation buttons as square cards with section
colors, border hover animation, and centered icons
- Move custom page links to a separate row with compact btn-outline style
- Remove deprecated btn-outline CSS overrides from app.css
- Add observer multi-select (<select multiple size=2>) to Advertisements and
Messages filter bars, populated from /api/v1/nodes?observer=true
- Make all filter sections collapsible via <details> on Nodes, Advertisements,
and Messages pages; collapsed by default, auto-expands when active filters
exist, preserves open state across auto-refresh ticks
- Add backend observer=true|false query param to GET /api/v1/nodes for
observer-only or non-observer-only node filtering via subquery
- Change observed_by in Advertisements/Messages API from single public_key
to list[str] with .in_() for multi-select support
- Fix router.js and api.js to handle array query params (duplicate keys
promoted to arrays, .append() per element)
- Fix createFilterHandler to use FormData.getAll() for multi-value support
- Replace DaisyUI form-control/label/label-text classes with Tailwind-native
equivalents (flex flex-col gap-1, flex items-center py-1, opacity-80 text-sm)
since DaisyUI CSS is tree-shaken from the build output
- Thicker collapsible border (border-2 border-base-content/25) visible in
both light and dark themes
- Bottom-align Filter/Clear buttons via two-row form layout
- Move Observer filter to last position on Advertisements page
- Add filter_observer_label i18n key
- Add tests for observer=true node filtering and multi observer params
Replace the dedicated admin tag management page with inline tag editing
on the node detail page. Operators can now edit tags directly on nodes
they've adopted; admins retain unrestricted access.
Key changes:
- Remove admin SPA page (admin/index.js, admin/node-tags.js)
- Add inline tag editor to node-detail.js with add/edit/delete modals
- Replace RequireAdmin with RequireOperatorOrAdmin for tag API routes
- Add ownership check: operators restricted to adopted nodes only
- Add validate_and_coerce_tag_value for number/boolean coercion
- Remove unused bulk endpoints (copy, move, replace all)
- Use AbortController for event listeners to prevent accumulation
on lit-html DOM reuse across re-renders
- Track Leaflet map instance at module scope for defensive cleanup
- Fix checkAuthResponse to only redirect on 401 (not 403)
- Update tests for new OIDC-based auth model
- Update en.json locale, i18n.md, upgrading.md, AGENTS.md
Replace the role=infra NodeTag convention with UserProfileNode adoption
as the canonical infrastructure indicator across map, Prometheus metrics,
and alerting. Renames is_infra to is_adopted, infra_center to
adopted_center. Map icons change to blue (adopted) / green (normal),
with all adoption UI gated on OIDC_ENABLED. Adds meshcore_nodes_adopted
gauge and Alembic migration to clean up obsolete tags.
Remove the static Member model/table, CRUD API, YAML seed files, and
admin UI. Replace with UserProfile-driven members page that reads roles
from OIDC identity provider. Key changes:
- Drop members table, add roles column to user_profiles (Alembic migration)
- Add GET /api/v1/user/profiles (paginated, no user_id exposed)
- Add GET /api/v1/user/profile/me (auto-creates profile for current user)
- Replace member_id node tag filter with adopted_by (profile UUID)
- Members page now shows profiles grouped by operator/member roles
- Profile page supports public view (/profile/:id) and owner edit (/profile)
- Node detail page shows adoption card side-by-side with public key card
- Auto-create user profile during OIDC login callback
- Hide Adopted Nodes section for non-operator/admin users
- Add member since date to profile cards
- Add role badges and adopted node badges to member tiles
- Add antenna/users icons to Members page group headers
- Add OPTIONS to the web API proxy route methods for CORS preflight support
- Fix event listener accumulation in admin/node-tags.js and admin/members.js
using AbortController with cleanup functions returned to the SPA router.
lit-html reuses DOM elements across re-renders, causing addEventListener
calls to accumulate and fire multiple times per form submission.
- Rename admin routes from /a/ prefix to /admin/ for clarity
- Add debug logging for admin route access and OIDC role checks
- Move auth section in navbar after theme toggle
- Update tests and AGENTS.md accordingly
- Add path_len column to event_observers table with Alembic migration
- Create shared observer_utils.py for building observer responses
- Extract SNR and path_len from LetsMesh uploads via normalizer
- Pass snr/path_len from all 4 handlers to add_event_observer()
- Add expandable observer detail sub-tables to messages and ads pages
- Rename Receivers column to Observers with updated i18n key
- Add satellite dish icon to observer badges and detail rows
- Add mobile card observer detail toggle
- Show full datetime tooltip on hover for relative times
- Hide Path column from advertisement observer tables (not populated)
Louis King