Commit Graph

881 Commits

Author SHA1 Message Date
Louis King a465a28d9c Merge branch 'main' of github.com:ipnet-mesh/meshcore-hub into feat/react-frontend-phase1 2026-07-22 23:03:45 +01:00
JingleManSweep 2929e15753 Merge pull request #322 from ipnet-mesh/renovate/anomalyco-opencode-1.x
chore(deps): update anomalyco/opencode action to v1.18.4
2026-07-22 23:03:14 +01:00
renovate[bot] 5cd8af10b0 chore(deps): update anomalyco/opencode action to v1.18.4 2026-07-22 22:01:09 +00:00
Louis King 9c47a3d229 Merge branch 'main' of github.com:ipnet-mesh/meshcore-hub into feat/react-frontend-phase1 2026-07-22 23:00:59 +01:00
JingleManSweep b0b07f4695 Merge pull request #323 from ipnet-mesh/renovate/daisyui-5.x-lockfile
chore(deps): update dependency daisyui to v5.7.0
2026-07-22 23:00:48 +01:00
Louis King a3136e59e0 ci: add Node.js setup to Lint job for frontend-typecheck hook 2026-07-22 23:00:09 +01:00
Louis King c8cc049651 feat(web): React SPA shell, markdown migration, vitest coverage, v0.17 docs
- Migrate footer, error page, and announcements to React; slim Jinja2
  shell to SEO/head/config/fonts/theme-init only
- Replace server-side markdown rendering with react-markdown (removes
  Python `markdown` dependency); custom pages + announcements ship
  raw markdown to the client
- Add heading-anchor deep-links with CSS override for inherited colors
- Extract pure helpers from pages (messageHelpers, mapMath, routesHelpers,
  profileHelpers, packetHelpers, packetGroupHelpers) for unit testability
- Add comprehensive vitest suite: 59 test files, 315 tests covering all
  components, pages, and extracted helpers; global i18next mock + matchMedia
  stub + AbortController-aware apiMock in test infrastructure
- Fix SortableTable test DOM nesting (<th> inside proper <table> context)
- Update docs for v0.17.0: upgrading.md release notes, README.md project
  structure + build description, i18n.md path fix
- Delete REACT_MIGRATION.md (migration complete, unreferenced)
- Add announcements dismiss-persistence + custom-page 404 E2E specs
2026-07-22 22:55:20 +01:00
Louis King 51d16c7766 fix(e2e): restore observer-node coordinates in seed
A flake8 B007 fix at commit time renamed the whole observer-specs tuple
to (_area, _lat, _lon), but the Node() call below still referenced lat/lon.
Those names then resolved to the leaked values from the preceding
content_specs loop (last iteration = Delta's None/None), so all observer
nodes were seeded without coordinates and dropped from the map (7 -> 3
markers). Underscore only `area` (genuinely unused in that loop); lat/lon
are used in Node(), so they stay as loop vars.
2026-07-22 14:37:51 +01:00
Louis King 94dd4c9b42 test(e2e): replace Python e2e suite with Playwright (headless Chromium)
Replaces the httpx-based smoke tests in tests/e2e/ with a browser E2E suite
under e2e/, running against a self-contained throwaway stack that never
touches the local development database.

Stack & data isolation (e2e/docker-compose.test.yml):
- Own ephemeral Postgres 17 (schema via the `migrate` service / Alembic),
  distinct project name + named volumes, no host DB port, no \${VAR}
  interpolation. `make e2e-down` destroys everything.
- OIDC enabled with a known session secret; WEB_AUTO_REFRESH_SECONDS=2 so
  polling is assertable; CONTENT_HOME mounts a test markdown page.
- Deterministic seeder (e2e/seed_data.py) run via global setup inside the
  collector container: nodes/observers with `area` tags, adverts, messages
  on the public (17) + a custom channel, raw packets + path hops keyed to
  node prefixes, a route + health/history, profiles + adoptions, and
  event_observers rows so the observer filter/badges resolve.

Auth & tests:
- Forged signed `meshcore-session` cookies (e2e/mint_session.py,
  itsdangerous) for admin/member identities -> storageState; no real IdP
  needed. 31 specs across 12 files cover global nav/theme, profile
  menu+edit, home hero, dashboard widgets, list filters/auto-refresh/row
  actions, observer toggles, the path-node overlay, map filters +
  show-labels, members, markdown pages, and routes add/edit/delete
  (persistence, validation, confirm dialog). workers:1 / fullyParallel:false
  against the single shared backend; targeted data-testids added to React
  components for stable selectors.

Fixes surfaced while running the suite on fresh Postgres:
- Migration 5e3b712ccf10 aborted on Postgres: its route-health backfill
  queries the live Route model (which now has max_path_length) before that
  column exists. The swallowed error left the transaction aborted, killing
  the subsequent alembic_version stamp. Wrapped the backfill in a SAVEPOINT
  so a failure rolls back cleanly without blocking the migration.
- Restored the full ABUSE_* env set required by the MQTT broker, and set
  the web service's API_KEY to the admin key (admin writes go through the
  proxy as a Bearer token).

31/31 passing; tsc (frontend+e2e), pytest (1460), and pre-commit green.
2026-07-22 10:41:53 +01:00
Louis King 789b756aa8 Merge branch 'main' of github.com:ipnet-mesh/meshcore-hub into feat/react-frontend-phase1 2026-07-22 08:19:32 +01:00
Louis King de9784211d feat(web): adopt TanStack Query for SPA data layer; consolidate shared UI
Migrate the React SPA off the bespoke useApiFetch hook and raw useEffect fetches to TanStack Query: useQuery/useQueries for reads, useMutation + invalidateQueries for writes; a central query-key factory and invalidation helpers mirroring the backend cache_invalidation prefixes (channels/routes/nodes/messages/profiles/dashboard/adoptions); polling via refetchInterval (useAutoRefresh now returns pause state only); RouteCard self-fetches its detail/history, dropping the hand-rolled caches. Adds QueryClientProvider in App, a renderWithProviders test helper, and deletes useApiFetch.

Also consolidates recurring UI into reusable components (Breadcrumbs, ListToolbar, Modal/ConfirmDialog, NotFoundState, TimeAgo, Definition, CopyableValue, MeshQrCode, Badges, SectionGroup, PageHeader) and fixes the FilterForm clear button, merged toggle wrappers, and role-aware channels/messages cache keys so client invalidation matches the server.

Verified: tsc --noEmit clean, vitest 113 passed, pytest test_web 251 + test_cache 119 passed, pre-commit --all-files green.
2026-07-21 23:24:58 +01:00
Louis King bf8e0620a5 fix(web): path-node popup scrolls with page; wire frontend tests/typecheck into make + pre-commit
- PacketGroupDetail: render the path-node popover via createPortal(document.body)
  with position:absolute + document coordinates (rect + scrollX/scrollY) instead of
  position:fixed with one-shot viewport coords, so it scrolls with the page rather
  than staying pinned to the viewport. Outside-click/Escape close unchanged (DOM-based).
- Makefile: 'make test' now runs pytest then the frontend vitest suite via a new
  'test-frontend' target (npm run test:frontend).
- pre-commit: add a local 'frontend-typecheck' hook (language: system) running
  'npm run typecheck' (tsc --noEmit) on TS/TSX + tsconfig/package(-lock).json changes.
- AGENTS.md: document the pre-commit TS gate and that 'make test' includes the frontend.

Verified: tsc clean, vitest 49 passed, pre-commit --all-files green, make test
(1459 backend passed + 49 frontend passed).
2026-07-21 20:25:50 +01:00
Louis King 527c860bf8 feat(web): frontend CI, vitest suite, navbar→React SPA shell — Phase 5
Frontend CI (closes the no-coverage gap for the TSX):
- ci.yml: new 'frontend' job — npm ci, tsc --noEmit, test:frontend, build.
- package.json: engines.node>=20, test:frontend/typecheck scripts.

vitest unit + component tests (jsdom, @testing-library/react):
- utils/charts.test.ts — tier math + every chart builder.
- utils/format.test.ts — parseAppDate, formatNumber, truncateKey, emoji helpers,
  formatRelativeTime.
- components/Navbar.test.tsx — feature-gated nav, custom pages, OIDC/maintenance
  auth gating.
- components/Announcements.test.tsx — banner rendering, ordering, dismiss +
  sessionStorage (covers behaviour moved out of the Python suite).

Navbar → React (full SPA shell):
- New Navbar/ThemeToggle/Announcements components + useNavItems hook (shared
  feature-gated nav for desktop + mobile); nav uses react-router NavLink
  (client-side nav + auto active) — drops the imperative data-nav-link bridge.
- main.tsx single root; App.tsx renders Navbar + Announcements above routed <main>.
- spa.html slimmed to SEO/config/footer shell (Jinja2 navbar/banners/theme script
  removed; #app is a plain div React fills).
- Backend: _build_config_json exposes system_announcement/network_announcement.

Tests (server-rendered nav/banner assertions → config):
- conftest.py: get_app_config() helper (robust __APP_CONFIG__ extraction).
- test_features/app/home/pages/dashboard rewritten to assert __APP_CONFIG__;
  dashboard client-rendered-stats tests replaced with a shell assertion.

Docs: REACT_MIGRATION.md Phase 5 (incl. deliberately-skipped react-query/
Storybook/Playwright), AGENTS.md Frontend section.

Verified: tsc clean, npm run build, vitest (49 passed), pytest tests/test_web
(251 passed), full suite (1459 passed), pre-commit (passed).
2026-07-21 20:06:52 +01:00
Louis King a5fabf7d46 chore(web): remove lit-html fallback & legacy code — Phase 4
The React frontend is complete (Phases 1-3); the lit-html fallback is dead
(its vendor globals were removed in Phase 3). Delete it and the scaffolding:

- Delete the entire src/meshcore_hub/web/static/js/spa/ lit-html tree,
  LitBridge.tsx, and legacy.d.ts.
- Remove the @legacy alias from vite.config.ts and tsconfig.json.
- Remove lit-html and qrcodejs from package.json (both unused now).
- Remove the lit-html fallback {% else %} branch from spa.html — the Vite
  build is now required to serve the UI (no fallback bundle).

Tests (fallback no longer exists):
- test_home/advertisements/nodes/messages.py: assert the React mount point
  (id="app") instead of a bundled-or-fallback script tag.
- test_caching.py: JS-cache tests are header-only (static JS is bundled into
  dist/, absent in test env; the middleware sets headers on 404 too); the
  dist-bundle HTML test drops its fallback branch.

Docs:
- AGENTS.md: new Frontend (React) section (host-run npm/vite/tsc toolchain,
  react-chartjs-2/react-leaflet/react-qr-code, CSS load order); clarified the
  compose-stack rule to exempt frontend tooling.
- REACT_MIGRATION.md: Phase 4 complete, final file structure, decisions.

Verified: tsc --noEmit clean, npm run build, full pytest (1463 passed,
22 skipped), pre-commit (passed).
2026-07-21 19:13:01 +01:00
Louis King 715659607a feat(web): React charts/maps/QR — Phase 3
Replace all window.Chart / window.L / window.QRCode globals and the
charts.js helper script with bundled React components:

- react-chartjs-2: typed config builders in utils/charts.ts
  (buildLineChart, buildActivityChart, buildStackedBar, buildRoutesTrend,
  buildRouteDetailStrip + ChartColors, averageRouteTier, routeQualityToTier)
  and wrappers in components/charts/Charts.tsx (ActivityChart,
  TrendLineChart, StackedBarChart, RoutesTrendChart, RouteDetailStrip).
  utils/charts.ts imports chart.js/auto. Wired into Home, Dashboard, Routes.
- react-leaflet: MapPage rewritten with MapContainer/TileLayer/Marker/Popup
  + a useMap MapController for fit-bounds and memoized markers; NodeDetail
  static hero map with divIcon marker + OffsetCenter. Both import
  leaflet/dist/leaflet.css.
- react-qr-code: replaces window.QRCode in Channels and NodeDetail.

Bundling & shell:
- Chart.js, Leaflet (+CSS), react-qr-code now bundled by Vite; removed the
  leaflet/chart.js/qrcodejs vendor <script>/<link> tags from spa.html and
  their copy steps from build.js (fonts stay vendored). Deleted charts.js.
- Moved the Vite CSS bundle (asset_app_css) into <head> before app.css so
  app.css dark-mode Leaflet overrides win over the bundled leaflet.css.
- Dropped the chart globals from the Window type declaration.

Tests/docs:
- test_caching.py: removed charts.js-specific tests; generic JS-cache tests
  now target spa/app.js.
- Updated charts.js cross-references in collector/routes.py + test_routes.py
  to point at spa-react/utils/charts.ts.

Verified: tsc --noEmit clean, npm run build (153 modules),
pytest tests/test_web (255 passed), pre-commit (passed).
2026-07-21 19:03:04 +01:00
Louis King 8322b5cf9f feat(web): convert all 15 SPA pages to native React — Phase 2
Convert every remaining lit-html page to React 19 + TypeScript and wire
them directly into the router, removing all LitBridge usage from App.tsx:

- Home, CustomPage, Profile, Members, Channels
- Advertisements, Messages, Routes, Nodes, NodeDetail
- Packets, PacketDetail, PacketGroupDetail, Dashboard, MapPage

Pages use the shared React infrastructure (apiGet<T>, useAutoRefresh,
usePageTitle, useFormatDateTime, Pagination, FilterForm, SortableTable,
NodeDisplay, ObserverBadges, RouteTypeBadge, JsonTree, StatCard, icons).

Charts/maps still call window.Chart / window.L / window.QRCode / charts.js
globals — these move to react-chartjs-2 / react-leaflet in Phase 3.

The old lit-html code in spa/ is intentionally kept as the spa.html
fallback (rendered only when the Vite bundle is absent) and is still
referenced by 5 web tests; it will be removed in Phase 4.

Added IconSatelliteDish, IconRuler, IconHopSpan, IconPathLength icons.

Verified: tsc --noEmit clean, npm run build (94 modules),
pytest tests/test_web/ (256 passed), pre-commit (passed).
2026-07-21 16:50:33 +01:00
Louis King 6cd14a58aa feat(web): React frontend scaffolding — Phase 1
- Add Vite 6 + TypeScript build replacing esbuild, React 19, React Router 7
- LitBridge wraps unconverted lit-html pages inside React app lifecycle
- Shared components: SortableTable, Pagination, FilterForm, StatCard, JsonTree,
  NodeDisplay, ObserverBadges, RouteTypeBadge, icons, ErrorBoundary, Alerts
- Hooks: useAutoRefresh, usePageTitle; utils: api, format, clipboard
- i18n via react-i18next mirroring existing translation keys
- Native React pages: NotFound, Maintenance; all other routes via LitBridge
- Jinja2 shell (spa.html) preserved for navbar, SEO, vendor globals, theme
- build.js generates legacy-compatible assets.json from Vite manifest
- REACT_MIGRATION.md documents full plan and conversion patterns
2026-07-21 10:21:39 +01:00
renovate[bot] 44eb79d011 chore(deps): update dependency daisyui to v5.7.0 2026-07-21 03:50:57 +00:00
JingleManSweep f232b90187 Merge pull request #320 from ipnet-mesh/renovate/actions-setup-python-7.x
chore(deps): update actions/setup-python action to v7
v0.16.2
2026-07-20 14:58:16 +01:00
renovate[bot] e71918ca13 chore(deps): update actions/setup-python action to v7 2026-07-20 11:52:15 +00:00
JingleManSweep 53c1b36262 Merge pull request #321 from ipnet-mesh/feat/routes-max-path-length
feat(routes): add per-route max_path_length cap
2026-07-20 12:51:15 +01:00
Louis King 1ba481483f feat(routes): add per-route max_path_length cap
Adds a new nullable per-route knob that caps the total number of hops
in a candidate packet's path. Packets whose path exceeds the cap are
dropped from matching consideration entirely (before the subsequence
matcher runs), so over-long paths never count toward
packet_count_threshold. Complements the existing max_hop_span, which
only constrains the gap between the first and last matched configured
node.

- Route model + migration (additive, nullable, default null = unlimited)
- Threaded through matcher chain (_subsequence_indices early-return)
- All 5 evaluate/preview/recent_matches call sites updated
- API serializer/create/update/preview passthrough
- CLI seed YAML import (update + create paths)
- Frontend: distinct icons for span (<-o->) vs path-length (|<->|),
  always-rendered badges with infinity fallback, hover tooltips on
  every stats row item, i18n keys (en + nl)
- Tests: matcher unit tests (within/exceeds cap), API round-trip,
  CLI seed import
2026-07-20 12:43:47 +01:00
JingleManSweep f2137d0429 Merge pull request #319 from ipnet-mesh/fix/route-clear-observers
fix(routes): clearing observers in the edit modal now persists
v0.16.1
2026-07-19 23:09:09 +01:00
Louis King d3f6d72775 fix(routes): clearing observers in the edit modal now persists
The route edit modal builds its PUT body with a ternary that collapses
an empty observer list to null:

    observer_public_keys: observerPublicKeys.length > 0 ? observerPublicKeys : null,

Pydantic parses null as None, and the PUT handler's guard

    if body.observer_public_keys is not None:
        _sync_observers(session, route, observer_nodes)

skips the sync entirely when the field is None. So removing all
observers in the modal sent null -> no DB change. Adding observers
worked because a non-empty array passed the guard and _sync_observers
deleted + recreated.

The None-means-skip semantic is correct for true partial updates, so
the fix is on the frontend: the edit modal is a full-form PUT, so it
must always send the array. When empty, it sends [], which Pydantic
parses as [] (not None), the guard passes, and _sync_observers deletes
every existing RouteObserver row.

Tests: add test_update_clear_observers_with_empty_list next to the
existing test_update_observers as a regression guard. It seeds one
observer via PUT, then PUTs observer_public_keys: [] and asserts both
the response and a fresh GET come back with an empty list.
2026-07-19 23:05:41 +01:00
JingleManSweep ddf747fe68 Merge pull request #318 from ipnet-mesh/chore/dashboard-routes-community-only
feat(dashboard): limit Route Health widget to community routes
v0.16.0
2026-07-19 21:59:57 +01:00
Louis King fdea046749 feat(dashboard): limit Route Health widget to community routes
The Route Health and Routes Trend dashboard widgets pull from
GET /api/v1/dashboard/routes-overview, which previously filtered
routes by the caller's role tier (anonymous saw community, admin saw
all four tiers). Operators and admins therefore saw member/operator/
admin-tier routes mixed into the dashboard, even though the dedicated
/routes page is the working surface for managing those private tiers.

This change makes the dashboard widget surface ONLY community-tier
routes, regardless of the caller's role. Operators and admins still
see all four tiers on the /routes management page (unchanged).

Implementation:
- get_routes_overview: drop resolve_user_role/get_max_visibility_level;
  push the filter into the SQL query
  (where Route.visibility == RouteVisibility.COMMUNITY.value) so
  higher-tier rows are no longer loaded just to be discarded.
- _dashboard_routes_overview_key_builder: drop the role dimension from
  the cache key. The response is now identical across roles, so the
  per-role cache slots were storing four identical copies. The prefix
  is preserved so invalidate_routes' pattern invalidation still hits.
- Drop unused VISIBILITY_LEVELS import; add RouteVisibility to the
  models import.

Tests:
- test_visibility_filter_hides_admin_routes renamed to
  test_dashboard_only_shows_community_routes_regardless_of_role;
  now seeds one route per visibility tier and asserts every role
  (anonymous/member/operator/admin) sees only ['Public'].
- test_cache_key_is_role_scoped renamed to
  test_cache_key_is_role_agnostic; asserts all four role-variants
  produce the SAME cache key (no 'role=' dimension).
2026-07-19 21:57:03 +01:00
JingleManSweep f3c8da56a4 Merge pull request #317 from ipnet-mesh/chore/routes-default-thresholds
chore(routes): default packet_count_threshold=5 and 3× clear multiplier
2026-07-19 21:44:59 +01:00
Louis King 8cf5dadc38 chore(routes): default packet_count_threshold=5 and clear multiplier=3x
Change the new-route defaults so the create modal, REST API, YAML
import, and preview helper pre-fill packet_count_threshold=5 instead
of 3, and bump the effective-clear auto-multiplier from 2x to 3x so
the default comfort bar tracks to 15 for a default threshold of 5.

Schema/model/cli/preview:
- schemas/routes.py: RouteCreate + RoutePreviewRequest
  packet_count_threshold default 3 -> 5 (clear_threshold stays None)
- models/route.py: Route INSERT default 3 -> 5 (clear_threshold stays
  nullable, no default)
- collector/cli.py: YAML import fallbacks 3 -> 5
- collector/routes.py: preview helper fallback 3 -> 5; module constant
  CLEAR_DEFAULT_MULTIPLIER 2 -> 3 (drives effective_clear_threshold)
- api/metrics.py: route_clear gauge help text 2x -> 3x

UI:
- web/spa/pages/routes.js: new-route modal pre-fills
  packet_count_threshold=5; placeholder shows 3x threshold; parseInt
  fallback 3 -> 5. clear_threshold field unchanged (clearing still
  sends null for the auto-tracking behaviour).

Docs:
- docs/routes.md defaults table: window_hours 24->48, threshold 3->5,
  clear (2x)->(3x), max_hop_span (unlimited)->8. Previous four
  entries now match the shipped defaults (some were stale from #316).
- docs/seeding.md YAML example: window_hours 24->48, threshold 3->5,
  commented clear_threshold example 10->15 with 3x note.

No migration: packet_count_threshold uses Python-side default= (no
server_default), so existing rows keep their stored values. Existing
routes with clear_threshold=NULL continue to track via the new 3x
multiplier at evaluation time.
2026-07-19 21:40:26 +01:00
JingleManSweep acd3a045ca Merge pull request #316 from ipnet-mesh/chore/routes-default-window-span
chore(routes): default window_hours=48 and max_hop_span=8
2026-07-19 21:12:42 +01:00
Louis King ae003ded58 chore(routes): default window_hours=48 and max_hop_span=8
Change the new-route defaults so the create modal and API/CLI paths
pre-fill a 48h evaluation window and a max-hop-span of 8 instead of
the previous 24h / unlimited (∞).

- schemas/routes.py: RouteCreate + RoutePreviewRequest defaults
- models/route.py: SQLAlchemy INSERT-time defaults
- collector/cli.py: YAML import fallbacks
- web/static/js/spa/pages/routes.js: new-route modal pre-fills 48/8

No migration: window_hours/max_hop_span use Python-side default=
(no server_default), so existing rows are untouched. Clearing the
max-hop-span field in the UI still sends null (no cap).
2026-07-19 21:09:19 +01:00
JingleManSweep c85798bb95 Merge pull request #315 from ipnet-mesh/ui/modal-save-spinner
UI polish: modal save spinners + route list sort
2026-07-19 21:00:57 +01:00
Louis King b86c514847 fix(routes): sort list by From then To
Routes list was sorted only by from_label; routes sharing the same
From appeared in backend-returned order, which looked random. Add
to_label as a secondary tiebreaker in the localeCompare comparator.
Case-sensitivity preserved (no behavior change for the primary key).
2026-07-19 20:56:56 +01:00
Louis King 05369377b0 feat(ui): show spinner on modal action buttons while in-flight
Adds a small DaisyUI loading-spinner to the primary action button and
disables both action buttons (Save/Delete + Cancel) while the request
is in flight. Prevents double-submit on slow operations like route
create/update, and prevents closing a modal mid-request which leaves
modalState in a confusing state.

Two patterns in the SPA, kept idiomatic to each:

- channels.js + routes.js (lit-html state-driven): thread a 'saving'
  boolean from modalState into the modal renderers, which add
  ?disabled + a leading <span class="loading loading-spinner
  loading-sm"></span> when set. Handlers set the flag and re-render
  before await; clear and re-render in the catch.

- node-detail.js (native <dialog> + imperative listeners): add an
  id to the Save button so the handler can toggle .disabled and swap
  .innerHTML on the button element directly, restored in a finally.

Covers all six modal action buttons: channel add/edit + delete,
route add/edit + delete, node-tag edit + delete.
2026-07-19 20:56:50 +01:00
JingleManSweep 138f2534d3 Merge pull request #314 from ipnet-mesh/feat/route-health-precompute
feat: precompute route health + consolidated migration + observer badge fix
2026-07-19 20:40:34 +01:00
Louis King fe284d51b5 docs: route health precompute var + upgrading.md corrections
- docker-compose.yml: passthrough ROUTE_HISTORY_BACKFILL_INTERVAL_SECONDS
  alongside the existing ROUTE_EVALUATOR_INTERVAL_SECONDS (operators
  setting the var in .env had no effect without the passthrough).
- configuration.md: document the new collector var in the Collector table.
- upgrading.md (Route Health Monitoring): correct the migration summary
  (seven tables, not five) and add a paragraph on the precompute
  behaviour; add the new env var row to the table.
- upgrading.md (Dashboard / Route cache consolidation): drop the stale
  'REDIS_CACHE_TTL_DASHBOARD raised to 3600' claim — the precompute
  revert lowered it back to 300 s. Renamed the section to
  'API endpoint reorganization' and retained the two ship-true changes
  (ROUTE_DETAIL TTL removed, /dashboard/recent-activity split).
2026-07-19 20:14:22 +01:00
Louis King bddec0c9c2 chore(migrations): consolidate route migrations into single step
Replaces ec40c67c8c83, 6b3430fd84f4 and cf8dd7eaba9b (never deployed
to production) with one migration that builds the final route-health
schema directly on top of the production head 57bb65130b97.

The consolidated migration creates the seven route tables, adds the
nullable event_hash column to raw_packets, and runs three idempotent
backfills: packet_path_hops from raw_packets.decoded, nodes dedup by
public_key, and route_result_history + quality_avg for enabled routes.

On a freshly-restored production snapshot the route-history backfill
is a no-op (no routes exist yet) but is retained for idempotency
against dev backups that do have routes.
2026-07-19 19:56:41 +01:00
Louis King 65ec669e48 fix: n is not defined in observer filter badges
Commit 37f8d7a re-introduced emoji extraction in observerFilterBadges
after d1a3605 had refactored the iteration variable from `n` (node
object) to `area` (plain string). Three references to n._displayName
and n.public_key were left behind, throwing ReferenceError the moment
any observer had an area tag.

Adverts and Messages pages surface this as a warning badge with the
error message; the early return hides the bug on deployments with no
area-tagged observers.

Replace n.X with area (the iteration variable). Emoji extraction and
label fallback now operate on the area string directly.
2026-07-19 19:56:34 +01:00
Louis King 938028a7a7 feat: precompute route health in background sweep
Persist route health derivations so the API layer no longer recomputes
them on every request. Two new tables (route_result_history,
route_recent_matches) plus a quality_avg column on route_results back
the dashboard strip and per-route history endpoints.

Collector:
- run_evaluation (60s) writes snapshot + quality_avg + recent_matches
- run_history_backfill (hourly) recomputes completed-day buckets
- subscriber wires a dedicated backfill scheduler thread

API:
- dashboard routes-overview bulk-loads via single history SELECT
- routes detail/detail history read from precomputed tables with
  live-compute fallback
- POST/PUT on routes upserts recent_matches and quality_avg inline
- dashboard cache TTL lowered from 1h to 5m (invalidation-aware)

Config: route_history_backfill_interval_seconds=3600,
        redis_cache_ttl_dashboard default 3600 -> 300
2026-07-19 19:56:24 +01:00
JingleManSweep 82a6d032d0 Merge pull request #313 from ipnet-mesh/feat/dashboard-routes-widgets
feat: dashboard route widgets + cache TTL consolidation
2026-07-19 17:43:47 +01:00
Louis King fc60cd201a feat: route badge reflects 7-day rolling average
The overall health badge on route cards now shows the rolling 7-day
average tier instead of the latest window-hours snapshot, so flapping
routes that are currently up still appear marginal/failing if the
week's mean warrants it. Same averaging drives the dashboard Route
Health widget's summary dot, the routes page summary strip counts,
and (already) the Route Trends chart line colors.

- compute_average_quality() in collector/routes.py (0/1/2 mean,
  thresholds 1.5/0.75, empty-history fallback) — kept in sync with
  the averageRouteTier JS helper in charts.js
- RouteRead / RouteDetail gain a 'quality_avg' field
- list/get/update handlers compute it per route; create skips (no
  meaningful history yet) and the frontend falls back to
  route_result.quality for brand-new routes
- diagnosis tooltip unchanged (still current-snapshot state text)
2026-07-19 17:23:54 +01:00
Louis King 37f8d7ae0b feat: dashboard route widgets + cache TTL consolidation
Adds Route Trends chart and Route Health strip grid to the dashboard,
backs them with a new GET /dashboard/routes-overview endpoint, and
consolidates the Redis cache layer (removes ROUTE_DETAIL TTL, raises
DASHBOARD default to 1h, splits recent-activity into its own short-TTL
endpoint so Recent panels stay fresh while aggregate counts cache 1h).
2026-07-19 16:49:19 +01:00
JingleManSweep f35be67da6 Merge pull request #312 from ipnet-mesh/renovate/fontsource-monorepo
chore(deps): update fontsource monorepo to v5.3.0
2026-07-19 14:31:57 +01:00
renovate[bot] ca2aac1c8f chore(deps): update fontsource monorepo to v5.3.0 2026-07-19 06:14:29 +00:00
JingleManSweep 76c6c00f44 Merge pull request #311 from ipnet-mesh/feat/cache-invalidation-on-writes
fix: route_result not refreshing after route edit (plus cache invalidation infra)
2026-07-18 14:05:40 +01:00
Louis King e93c7fd9d6 fix: re-evaluate route_result synchronously after route create/update
Reported symptom: user edits a Route (e.g. lowers packet_count_threshold
from 6 to 3), and the routes list card still shows the old threshold for
~30s. The cache stack was exonerated — x-cache: MISS on the stale GET,
response body had the new packet_count_threshold=3, but route_result.
threshold remained at 6.

Root cause was neither HTTP cache nor Redis cache. The list card's stats
row (renderStatsRow in routes.js:82) displays route_result.threshold /
effective_clear / matched_count, which are persisted by the background
route_evaluator on a 30-60s schedule — separate DB row from the route's
direct fields. The PUT handler updated the route row immediately but
never triggered a re-evaluation, so route_result carried the stale
snapshot from the prior evaluator cycle until the next sweep.

Fix: add _reevaluate_route(session, route) helper that runs evaluate_route
+ upsert_route_result synchronously after the route commit. Called from
create_route (initial evaluation) and update_route (refresh on every
config change). Disabled routes short-circuit (no point evaluating a
route that's turned off). One bounded DB scan per write — same cost as
a single-route evaluator tick.

After this fix, the PUT response itself carries a fresh route_result
reflecting the new packet_count_threshold / clear_threshold, and the
next GET /api/v1/routes returns it. The list card updates on the very
next render cycle.

Tests:
- test_update_threshold_immediately_reflects_in_route_result: seeds a
  stale RouteResult with the OLD threshold, sends a PUT, asserts the
  response's route_result.threshold/effective_clear now match the new
  route config.
- test_disabled_route_does_not_trigger_evaluation: monkeypatches
  evaluate_route to a spy, asserts it's never called for disabled routes.
2026-07-18 13:55:02 +01:00
Louis King b6740068d3 log: cache invalidation observability for production diagnosis
After PR #311's invalidation wiring and HTTP policy fix, a user reported
the routes list still showed stale data for ~30s after a PUT. The HAR
they sent actually showed the fix working (x-cache: MISS after PUT,
fresh data returned) but had been captured with DevTools' 'Disable
cache' enabled — so it didn't represent normal browsing. We had four
competing hypotheses and no way to pick between them.

Add structured INFO logging at the two points that matter:

  * meshcore_hub.api.cache_invalidation._drop now emits
    'Cache invalidate start: prefix=... backend=...' and
    'Cache invalidate ok: prefix=...'. The backend= field
    distinguishes RedisCacheBackend from NullCache in one glance,
    catching 'REDIS_ENABLED is actually false' cases.

  * meshcore_hub.common.redis.RedisCacheBackend.delete now emits
    'Redis cache delete: prefix=... full_prefix=... keys_deleted=N
    scan_iterations=N'. keys_deleted=0 after a mutation that should
    have invalidated entries is the smoking gun for a cache-key
    mismatch between the store path (key_builder) and the delete
    path (prefix glob).

  * NullCache.delete emits a DEBUG line for the same reason.

The error paths are also enriched with full_prefix for greppability.
No behavioral changes — invalidation still swallows errors so cache
outages never break a write.

After deploy, a single route-edit repro will produce log output whose
shape (start/ok/warning/missing, keys_deleted count) unambiguously
identifies which of the four hypotheses is correct, so we can write
a targeted fix instead of guessing.
2026-07-18 13:41:52 +01:00
Louis King 71f65714ee fix: force browser revalidation so cache invalidation reaches the UI
After PR #311 wired server-side invalidation, a user reported that editing
a Route still showed old values on the routes list for ~30s. Root cause:
the api_cache_middleware emitted 'Cache-Control: private, max-age=30' on
@cached GETs, which let the browser serve its local HTTP cache copy
without revalidating. The Redis invalidation fired correctly but never
mattered — the browser never asked the server.

Switch all /api/* GET responses to 'private, no-cache' (synonymous with
'max-age=0, must-revalidate'). The browser now sends If-None-Match on
every navigation; the server answers 304 when Redis is warm and unchanged
(cheap — no body) or 200 after an invalidation.

The per-endpoint Redis TTL (30s default, 300s dashboard/route-detail)
still bounds cache.set lifetime; only the HTTP-layer max-age disappears.
Most navigations are still 304s, so the cost is one tiny round-trip per
page load while guaranteeing freshness after any mutation.

Adds an end-to-end regression test (test_routes_list_refresh_after_mutation)
modelling the exact reported scenario: cache-fill, conditional 304,
PUT mutation, conditional 200 with fresh body.
2026-07-18 13:17:04 +01:00
Louis King fc0520461a feat: invalidate read caches on user/admin mutations
Mutation handlers (POST/PUT/DELETE) on channels, routes, user profiles,
node tags, and adoptions now drop the corresponding Redis cache entries
after commit so the UI reflects changes on the next page load instead of
waiting for the 30s/300s TTL.

Adds meshcore_hub.api.cache_invalidation with seven helpers that
encapsulate the two cache-key formats (endpoint-name keys like 'nodes:'
vs URL-path keys like '/api/v1/channels:') and swallow backend errors
so a cache outage never breaks a successful write. The helper is a
no-op when Redis is disabled.

Cross-entity embeddings are covered: node-tag writes invalidate nodes,
messages, advertisements, and dashboard; adoption writes invalidate
nodes, profiles, advertisements, and dashboard.

CLI/collector mutations remain TTL-bound (infrequent, operator-driven).
2026-07-18 13:04:06 +01:00
JingleManSweep 26982c4d68 Merge pull request #310 from ipnet-mesh/feat/api-caching-and-event-dedup
feat: HTTP Cache-Control, dashboard TTL bump, and route event-hash dedup
2026-07-18 12:45:32 +01:00
Louis King 3810e029ce feat: HTTP Cache-Control, dashboard TTL bump, and route event-hash dedup
Bundles four independent improvements to API caching and route evaluator
accuracy that all touch the same files in the cache/routes stack.

## HTTP Cache-Control headers (api_cache_control_enabled)

The API now emits HTTP Cache-Control on every /api/v1/* response and
handles ETag / If-None-Match -> 304 Not Modified on @cached endpoints.

- @cached GETs: private, max-age=<redis_ttl> + strong SHA-256 ETag.
  Matching If-None-Match returns 304 with empty body.
- Uncached GETs under /api/v1/*: private, max-age=0, must-revalidate.
- POST/PUT/DELETE/PATCH and /health*: no-store.
- All private (several @cached endpoints are role-aware).
- @cached decorator stores new envelope {body, etag} in Redis; legacy
  bare-JSON entries still read (hashed on the fly) and auto-migrate on
  the next miss.
- X-Cache: HIT|MISS observability header is always emitted.
- Kill switch: API_CACHE_CONTROL_ENABLED (default true).

## Dashboard cache TTL bump (30s -> 300s)

REDIS_CACHE_TTL_DASHBOARD default raised from 30s to 300s. Covers all
/dashboard/* endpoints and /api/v1/routes/{id}/history. These return
trend/aggregation data where minute-level staleness is invisible but
every MISS costs seconds of SQL/aggregation. Also resolves reported
short-TTL behaviour on the per-route healthchart (which shares this
setting). Operators with explicit env var: unchanged. Old Redis entries
expire naturally within <=30s; no flush needed.

## Route recent-packets hop truncation

In the routes page expanded card, path hops are now truncated to
2 + ellipsis + 2 when length > 5, matching the packet-detail page
styling. Reuses the existing packets.hops_hidden i18n key as the
tooltip on the ellipsis badge.

## Route event-hash deduplication

A single advert/message/telemetry/trace retransmitted or flooded
through the mesh produces one RawPacket per on-air copy, each with a
fresh wire packet_hash. Counting those as distinct matches let a single
underlying event satisfy packet_count_threshold within seconds and
biased route health.

- Add nullable event_hash column to raw_packets and packet_path_hops
  (alembic 6b3430fd84f4). No backfill; legacy rows keep NULL and the
  evaluator falls back to wire packet_hash until they age out of
  window_hours.
- Subscriber denormalizes the structured event's event_hash onto the
  captured raw packet after handler dispatch.
- Route evaluator prefers event_hash when set, collapsing all
  receptions of the same underlying event into one match.
2026-07-18 12:43:01 +01:00