diff --git a/AGENTS.md b/AGENTS.md index 07d1c11..5754ffe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -277,23 +277,23 @@ PYTHONPATH=. uv run pytest tests/ -v ``` Key test files: -- `tests/test_decoder.py` - Channel + direct message decryption, key exchange -- `tests/test_keystore.py` - Ephemeral key store -- `tests/test_event_handlers.py` - ACK tracking, repeat detection -- `tests/test_packet_pipeline.py` - End-to-end packet processing -- `tests/test_api.py` - API endpoints, read state tracking -- `tests/test_migrations.py` - Database migration system -- `tests/test_frontend_static.py` - Frontend static route registration (missing `dist`/`index.html` handling) -- `tests/test_messages_search.py` - Message search, around endpoint, forward pagination -- `tests/test_rx_log_data.py` - on_rx_log_data event handler integration -- `tests/test_ack_tracking_wiring.py` - DM ACK tracking extraction and wiring -- `tests/test_radio_lifecycle_service.py` - Radio reconnect/setup orchestration helpers -- `tests/test_radio_commands_service.py` - Radio config/private-key service workflows -- `tests/test_health_mqtt_status.py` - Health endpoint MQTT status field -- `tests/test_community_mqtt.py` - Community MQTT publisher (JWT, packet format, hash, broadcast) -- `tests/test_radio_sync.py` - Radio sync, periodic tasks, and contact offload back to the radio -- `tests/test_real_crypto.py` - Real cryptographic operations -- `tests/test_disable_bots.py` - MESHCORE_DISABLE_BOTS=true feature +- `tests/test_api.py` - Broad API integration coverage across routers and read-state flows +- `tests/test_packet_pipeline.py` - End-to-end packet processing, decrypt, dedup, and message creation +- `tests/test_event_handlers.py` - ACK tracking, fallback DM handling, and event subscription cleanup +- `tests/test_send_messages.py` - Outgoing DM/channel send workflows, retries, and bot-trigger wiring +- `tests/test_packets_router.py` - Historical decrypt, maintenance, and raw-packet detail endpoints +- `tests/test_repeater_routes.py` - Repeater command/telemetry/trace pane endpoints +- `tests/test_room_routes.py` - Room-server login/status/ACL/telemetry endpoints +- `tests/test_radio_router.py` - Radio config, advert, discovery, trace, and reconnect endpoints +- `tests/test_radio_sync.py` - Radio sync, periodic tasks, contact offload/reload, and pending-message flushes +- `tests/test_fanout.py` - Fanout config CRUD, scope matching, and manager dispatch +- `tests/test_fanout_integration.py` - Integration-module lifecycle and delivery behavior +- `tests/test_statistics.py` - Aggregated mesh/network statistics and noise-floor snapshots +- `tests/test_version_info.py` - Version/build metadata resolution +- `tests/test_websocket.py` - WS manager broadcast and cleanup behavior +- `tests/test_frontend_static.py` - Frontend static route registration and fallback behavior + +For the fuller backend inventory, see `app/AGENTS.md`. For frontend-specific suites, see `frontend/AGENTS.md`. ### Frontend (Vitest) @@ -319,6 +319,7 @@ All endpoints are prefixed with `/api` (e.g., `/api/health`). | PUT | `/api/radio/private-key` | Import private key to radio | | POST | `/api/radio/advertise` | Send advertisement (`mode`: `flood` or `zero_hop`, default `flood`) | | POST | `/api/radio/discover` | Run a short mesh discovery sweep for nearby repeaters/sensors | +| POST | `/api/radio/trace` | Send a multi-hop trace loop through known repeaters and back to the local radio | | POST | `/api/radio/reboot` | Reboot radio or reconnect if disconnected | | POST | `/api/radio/disconnect` | Disconnect from radio and pause automatic reconnect attempts | | POST | `/api/radio/reconnect` | Manual radio reconnection | @@ -341,6 +342,10 @@ All endpoints are prefixed with `/api` (e.g., `/api/health`). | POST | `/api/contacts/{public_key}/repeater/radio-settings` | Fetch repeater radio config via CLI | | POST | `/api/contacts/{public_key}/repeater/advert-intervals` | Fetch advert intervals | | POST | `/api/contacts/{public_key}/repeater/owner-info` | Fetch owner info | +| POST | `/api/contacts/{public_key}/room/login` | Log in to a room server | +| POST | `/api/contacts/{public_key}/room/status` | Fetch room-server status telemetry | +| POST | `/api/contacts/{public_key}/room/lpp-telemetry` | Fetch room-server CayenneLPP sensor data | +| POST | `/api/contacts/{public_key}/room/acl` | Fetch room-server ACL entries | | GET | `/api/channels` | List channels | | GET | `/api/channels/{key}/detail` | Comprehensive channel profile (message stats, top senders) | @@ -354,6 +359,7 @@ All endpoints are prefixed with `/api` (e.g., `/api/health`). | POST | `/api/messages/channel` | Send channel message | | POST | `/api/messages/channel/{message_id}/resend` | Resend channel message (default: byte-perfect within 30s; `?new_timestamp=true`: fresh timestamp, no time limit, creates new message row) | | GET | `/api/packets/undecrypted/count` | Count of undecrypted packets | +| GET | `/api/packets/{packet_id}` | Fetch one stored raw packet by row ID for on-demand inspection | | POST | `/api/packets/decrypt/historical` | Decrypt stored packets | | POST | `/api/packets/maintenance` | Delete old packets and vacuum | | GET | `/api/read-state/unreads` | Server-computed unread counts, mentions, last message times, and `last_read_ats` boundaries | @@ -368,6 +374,7 @@ All endpoints are prefixed with `/api` (e.g., `/api/health`). | POST | `/api/fanout` | Create new fanout config | | PATCH | `/api/fanout/{id}` | Update fanout config (triggers module reload) | | DELETE | `/api/fanout/{id}` | Delete fanout config (stops module) | +| POST | `/api/fanout/bots/disable-until-restart` | Stop bot fanout modules and keep bots disabled until the process restarts | | GET | `/api/statistics` | Aggregated mesh network statistics | | WS | `/api/ws` | Real-time updates | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e54d449..256d09a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -78,6 +78,7 @@ These tests are only guaranteed to run correctly in a narrow subset of environme ```bash cd tests/e2e +npm install npx playwright test # headless npx playwright test --headed # you can probably guess ``` diff --git a/app/AGENTS.md b/app/AGENTS.md index 105fe7f..3c44261 100644 --- a/app/AGENTS.md +++ b/app/AGENTS.md @@ -25,18 +25,22 @@ Keep it aligned with `app/` source files and router behavior. app/ ├── main.py # App startup/lifespan, router registration, static frontend mounting ├── config.py # Env-driven runtime settings +├── channel_constants.py # Public/default channel constants shared across sync/send logic ├── database.py # SQLite connection + base schema + migration runner ├── migrations.py # Schema migrations (SQLite user_version) ├── models.py # Pydantic request/response models and typed write contracts (for example ContactUpsert) +├── version_info.py # Unified version/build metadata resolution for debug + startup surfaces ├── repository/ # Data access layer (contacts, channels, messages, raw_packets, settings, fanout) ├── services/ # Shared orchestration/domain services │ ├── messages.py # Shared message creation, dedup, ACK application │ ├── message_send.py # Direct send, channel send, resend workflows │ ├── dm_ingest.py # Shared direct-message ingest / dedup seam for packet + fallback paths +│ ├── dm_ack_apply.py # Shared DM ACK application over pending/buffered ACK state │ ├── dm_ack_tracker.py # Pending DM ACK state │ ├── contact_reconciliation.py # Prefix-claim, sender-key backfill, name-history wiring │ ├── radio_lifecycle.py # Post-connect setup and reconnect/setup helpers │ ├── radio_commands.py # Radio config/private-key command workflows +│ ├── radio_noise_floor.py # In-memory local radio noise-floor sampling/history │ └── radio_runtime.py # Router/dependency seam over the global RadioManager ├── radio.py # RadioManager transport/session state + lock management ├── radio_sync.py # Polling, sync, periodic advertisement loop @@ -61,6 +65,8 @@ app/ ├── messages.py ├── packets.py ├── read_state.py + ├── rooms.py + ├── server_control.py ├── settings.py ├── fanout.py ├── repeaters.py @@ -174,6 +180,7 @@ app/ - `PUT /radio/private-key` - `POST /radio/advertise` — manual advert send; request body may set `mode` to `flood` or `zero_hop` (defaults to `flood`) - `POST /radio/discover` — short mesh discovery sweep for nearby repeaters/sensors +- `POST /radio/trace` — send a multi-hop trace loop through known repeaters and back to the local radio - `POST /radio/disconnect` - `POST /radio/reboot` - `POST /radio/reconnect` @@ -198,6 +205,10 @@ app/ - `POST /contacts/{public_key}/repeater/radio-settings` - `POST /contacts/{public_key}/repeater/advert-intervals` - `POST /contacts/{public_key}/repeater/owner-info` +- `POST /contacts/{public_key}/room/login` +- `POST /contacts/{public_key}/room/status` +- `POST /contacts/{public_key}/room/lpp-telemetry` +- `POST /contacts/{public_key}/room/acl` ### Channels - `GET /channels` @@ -216,6 +227,7 @@ app/ ### Packets - `GET /packets/undecrypted/count` +- `GET /packets/{packet_id}` — fetch one stored raw packet by row ID for on-demand inspection - `POST /packets/decrypt/historical` - `POST /packets/maintenance` @@ -236,6 +248,7 @@ app/ - `POST /fanout` — create new fanout config - `PATCH /fanout/{id}` — update fanout config (triggers module reload) - `DELETE /fanout/{id}` — delete fanout config (stops module) +- `POST /fanout/bots/disable-until-restart` — stop bot modules and keep bots disabled until restart ### Statistics - `GET /statistics` — aggregated mesh network stats (entity counts, message/packet splits, activity windows, busiest channels) @@ -322,9 +335,11 @@ tests/ ├── conftest.py # Shared fixtures ├── test_ack_tracking_wiring.py # DM ACK tracking extraction and wiring ├── test_api.py # REST endpoint integration tests +├── test_block_lists.py # Blocked keys/names filtering across list/search surfaces ├── test_bot.py # Bot execution and sandboxing -├── test_channels_router.py # Channels router endpoints ├── test_channel_sender_backfill.py # Sender-key backfill uniqueness rules for channel messages +├── test_channels_router.py # Channels router endpoints +├── test_community_mqtt.py # Community MQTT publisher (JWT, packet format, hash, broadcast) ├── test_config.py # Configuration validation ├── test_contact_reconciliation_service.py # Prefix/contact reconciliation service helpers ├── test_contacts_router.py # Contacts router endpoints @@ -332,40 +347,41 @@ tests/ ├── test_disable_bots.py # MESHCORE_DISABLE_BOTS=true feature ├── test_echo_dedup.py # Echo/repeat deduplication (incl. concurrent) ├── test_fanout.py # Fanout bus CRUD, scope matching, manager dispatch -├── test_fanout_integration.py # Fanout integration tests ├── test_fanout_hitlist.py # Fanout-related hitlist regression tests +├── test_fanout_integration.py # Fanout integration tests ├── test_event_handlers.py # ACK tracking, event registration, cleanup ├── test_frontend_static.py # Frontend static file serving ├── test_health_mqtt_status.py # Health endpoint MQTT status field ├── test_http_quality.py # Cache-control / gzip / basic-auth HTTP quality checks ├── test_key_normalization.py # Public key normalization ├── test_keystore.py # Ephemeral keystore +├── test_main_startup.py # App startup and lifespan +├── test_map_upload.py # Map upload fanout module ├── test_message_pagination.py # Cursor-based message pagination ├── test_message_prefix_claim.py # Message prefix claim logic -├── test_migrations.py # Schema migration system -├── test_community_mqtt.py # Community MQTT publisher (JWT, packet format, hash, broadcast) ├── test_mqtt.py # MQTT publisher topic routing and lifecycle +├── test_messages_search.py # Message search, around, forward pagination +├── test_migrations.py # Schema migration system ├── test_packet_pipeline.py # End-to-end packet processing ├── test_packets_router.py # Packets router endpoints (decrypt, maintenance) +├── test_path_utils.py # Path hex rendering helpers ├── test_radio.py # RadioManager, serial detection ├── test_radio_commands_service.py # Radio config/private-key service workflows ├── test_radio_lifecycle_service.py # Reconnect/setup orchestration helpers -├── test_radio_runtime_service.py # radio_runtime seam behavior and helpers -├── test_real_crypto.py # Real cryptographic operations ├── test_radio_operation.py # radio_operation() context manager ├── test_radio_router.py # Radio router endpoints +├── test_radio_runtime_service.py # radio_runtime seam behavior and helpers ├── test_radio_sync.py # Polling, sync, advertisement +├── test_real_crypto.py # Real cryptographic operations ├── test_repeater_routes.py # Repeater command/telemetry/trace + granular pane endpoints ├── test_repository.py # Data access layer +├── test_room_routes.py # Room-server login/status/telemetry/ACL endpoints ├── test_rx_log_data.py # on_rx_log_data event handler integration -├── test_messages_search.py # Message search, around, forward pagination -├── test_block_lists.py # Blocked keys/names filtering ├── test_security.py # Optional Basic Auth middleware / config behavior ├── test_send_messages.py # Outgoing messages, bot triggers, concurrent sends ├── test_settings_router.py # Settings endpoints, advert validation ├── test_statistics.py # Statistics aggregation -├── test_main_startup.py # App startup and lifespan -├── test_path_utils.py # Path hex rendering helpers +├── test_version_info.py # Version/build metadata resolution ├── test_websocket.py # WS manager broadcast/cleanup └── test_websocket_route.py # WS endpoint lifecycle ``` diff --git a/app/packet_processor.py b/app/packet_processor.py index 12ac466..32ddd9f 100644 --- a/app/packet_processor.py +++ b/app/packet_processor.py @@ -264,9 +264,10 @@ async def process_raw_packet( This is the main entry point for all incoming RF packets. Note: Packets are deduplicated by payload hash in the database. If we receive - a duplicate packet (same payload, different path), we still broadcast it to - the frontend (for the real-time packet feed) but skip decryption processing - since the original packet was already processed. + a duplicate payload (same payload, different path), we still broadcast it to + the frontend for realtime packet-feed fidelity. Some payload types are also + intentionally reprocessed on duplicate arrival so message-level dedup/path + merge logic and advert/path-history tracking still see each observation. """ ts = timestamp or int(time.time()) observation_id = next(_raw_observation_counter) diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index 2843206..5c44739 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -39,6 +39,8 @@ frontend/src/ ├── index.css # Global styles/utilities ├── styles.css # Additional global app styles ├── themes.css # Color theme definitions +├── contexts/ +│ └── DistanceUnitContext.tsx # Browser-local distance-unit context/provider ├── lib/ │ └── utils.ts # cn() — clsx + tailwind-merge helper ├── hooks/ @@ -53,10 +55,14 @@ frontend/src/ │ ├── useRadioControl.ts # Radio health/config state, reconnection, mesh discovery sweeps │ ├── useAppSettings.ts # Settings, favorites, preferences migration │ ├── useConversationRouter.ts # URL hash → active conversation routing -│ └── useContactsAndChannels.ts # Contact/channel loading, creation, deletion +│ ├── useContactsAndChannels.ts # Contact/channel loading, creation, deletion +│ ├── useBrowserNotifications.ts # Per-conversation browser notification preferences + dispatch +│ ├── useFaviconBadge.ts # Browser tab unread badge state +│ ├── useRawPacketStatsSession.ts # Session-scoped packet-feed stats history +│ └── useRememberedServerPassword.ts # Browser-local repeater/room password persistence ├── components/ -│ ├── AppShell.tsx # App-shell layout: status, sidebar, search/settings panes, cracker, modals -│ ├── ConversationPane.tsx # Active conversation surface selection (map/raw/repeater/chat/empty) +│ ├── AppShell.tsx # App-shell layout: status, sidebar, search/settings panes, cracker, modals, security warning +│ ├── ConversationPane.tsx # Active conversation surface selection (map/raw/trace/repeater/room/chat/empty) │ ├── visualizer/ │ │ ├── useVisualizerData3D.ts # Packet→graph data pipeline, repeat aggregation, simulation state │ │ ├── useVisualizer3DScene.ts # Three.js scene lifecycle, buffers, hover/pin interaction @@ -73,14 +79,17 @@ frontend/src/ │ ├── pubkey.ts # getContactDisplayName (12-char prefix fallback) │ ├── contactAvatar.ts # Avatar color derivation from public key │ ├── rawPacketIdentity.ts # observation_id vs id dedup helpers +│ ├── rawPacketStats.ts # Session packet stats windows, rankings, and coverage helpers │ ├── regionScope.ts # Regional flood-scope label/normalization helpers │ ├── visualizerUtils.ts # 3D visualizer node types, colors, particles │ ├── visualizerSettings.ts # LocalStorage persistence for visualizer options │ ├── a11y.ts # Keyboard accessibility helper +│ ├── distanceUnits.ts # Browser-local distance unit persistence/helpers │ ├── lastViewedConversation.ts # localStorage for last-viewed conversation │ ├── contactMerge.ts # Merge WS contact updates into list │ ├── localLabel.ts # Local label (text + color) in localStorage │ ├── radioPresets.ts # LoRa radio preset configurations +│ ├── publicChannel.ts # Public-channel resolution helpers for routing/hash defaults │ ├── fontScale.ts # Browser-local relative font scale persistence/application │ └── theme.ts # Theme switching helpers ├── components/ @@ -92,8 +101,12 @@ frontend/src/ │ ├── NewMessageModal.tsx │ ├── SearchView.tsx # Full-text message search pane │ ├── SettingsModal.tsx # Layout shell — delegates to settings/ sections +│ ├── SecurityWarningModal.tsx # Startup warning for trusted-network / bot execution posture │ ├── RawPacketList.tsx +│ ├── RawPacketFeedView.tsx # Live raw packet feed + session stats drawer +│ ├── RawPacketDetailModal.tsx # On-demand packet inspector dialog │ ├── MapView.tsx +│ ├── TracePane.tsx # Multi-hop route trace builder/results view │ ├── VisualizerView.tsx │ ├── PacketVisualizer3D.tsx │ ├── PathModal.tsx @@ -103,9 +116,14 @@ frontend/src/ │ ├── ContactAvatar.tsx │ ├── ContactInfoPane.tsx # Contact detail sheet (stats, name history, paths) │ ├── ContactStatusInfo.tsx # Contact status info component +│ ├── ContactPathDiscoveryModal.tsx # Forward/return path discovery dialog +│ ├── ContactRoutingOverrideModal.tsx # Manual direct-route override editor │ ├── RepeaterDashboard.tsx # Layout shell — delegates to repeater/ panes │ ├── RepeaterLogin.tsx # Repeater login form (password + guest) +│ ├── RoomServerPanel.tsx # Room-server auth gate + status banner ahead of room chat +│ ├── ServerLoginStatusBanner.tsx # Shared repeater/room login state banner │ ├── ChannelInfoPane.tsx # Channel detail sheet (stats, top senders) +│ ├── ChannelFloodScopeOverrideModal.tsx # Per-channel flood-scope override editor │ ├── DirectTraceIcon.tsx # Shared direct-trace glyph used in header/dashboard │ ├── NeighborsMiniMap.tsx # Leaflet mini-map for repeater neighbor locations │ ├── settings/ @@ -131,12 +149,13 @@ frontend/src/ │ └── ui/ # shadcn/ui primitives ├── types/ │ └── d3-force-3d.d.ts # Type declarations for d3-force-3d -└── test/ +└── test/ # Representative frontend test suites (not an exhaustive listing) ├── setup.ts ├── fixtures/websocket_events.json ├── api.test.ts ├── appFavorites.test.tsx ├── appStartupHash.test.tsx + ├── conversationPane.test.tsx ├── contactAvatar.test.ts ├── contactInfoPane.test.tsx ├── integration.test.ts @@ -147,18 +166,23 @@ frontend/src/ ├── rawPacketList.test.tsx ├── pathUtils.test.ts ├── prefetch.test.ts + ├── rawPacketDetailModal.test.tsx + ├── rawPacketFeedView.test.tsx ├── radioPresets.test.ts ├── rawPacketIdentity.test.ts ├── repeaterDashboard.test.tsx ├── repeaterFormatters.test.ts ├── repeaterLogin.test.tsx ├── repeaterMessageParsing.test.ts + ├── roomServerPanel.test.tsx + ├── securityWarningModal.test.tsx ├── localLabel.test.ts ├── messageInput.test.tsx ├── newMessageModal.test.tsx ├── settingsModal.test.tsx ├── sidebar.test.tsx ├── statusBar.test.tsx + ├── tracePane.test.tsx ├── unreadCounts.test.ts ├── urlHash.test.ts ├── appSearchJump.test.tsx @@ -170,12 +194,17 @@ frontend/src/ ├── useConversationMessages.race.test.ts ├── useConversationNavigation.test.ts ├── useAppShell.test.ts + ├── useBrowserNotifications.test.ts + ├── useFaviconBadge.test.ts ├── useRepeaterDashboard.test.ts + ├── useRememberedServerPassword.test.ts ├── useContactsAndChannels.test.ts ├── useRealtimeAppState.test.ts ├── useUnreadCounts.test.ts ├── useWebSocket.dispatch.test.ts ├── useWebSocket.lifecycle.test.ts + ├── rawPacketStats.test.ts + ├── fontScale.test.ts └── wsEvents.test.ts ``` @@ -191,6 +220,7 @@ frontend/src/ - search/settings surface switching - global cracker mount/focus behavior - new-message modal and info panes +- trusted-network `SecurityWarningModal` High-level state is delegated to hooks: - `useAppShell`: app-shell view state (settings section, sidebar, cracker, new-message modal) @@ -212,7 +242,9 @@ High-level state is delegated to hooks: - map view - visualizer - raw packet feed +- trace view - repeater dashboard +- room-server auth/status gate before room chat - normal chat chrome (`ChatHeader` + `MessageList` + `MessageInput`) ### Initial load + realtime @@ -273,12 +305,16 @@ Supported routes: - `#map/focus/{pubkey_or_prefix}` - `#visualizer` - `#search` +- `#trace` +- `#settings/{section}` - `#channel/{channelKey}` - `#channel/{channelKey}/{label}` - `#contact/{publicKey}` - `#contact/{publicKey}/{label}` -Legacy name-based hashes are still accepted for compatibility. +Where `{section}` is one of `radio`, `local`, `fanout`, `database`, `statistics`, or `about`. + +Legacy name-based channel/contact hashes are still accepted for compatibility. ## Conversation State Keys (`utils/conversationState.ts`) @@ -378,6 +414,12 @@ For repeater contacts (`type=2`), `ConversationPane.tsx` renders `RepeaterDashbo All state is managed by `useRepeaterDashboard` hook. State resets on conversation change. +## Room Server Panel + +For room contacts (`type=3`), `ConversationPane.tsx` keeps the normal chat surface but inserts `RoomServerPanel` above it. That panel handles room-server login/status messaging and gates room chat behind the room-authenticated state when required. + +`ServerLoginStatusBanner` is shared between repeater and room login surfaces for inline status/error display. + ## Message Search Pane The `SearchView` component (`components/SearchView.tsx`) provides full-text search across all DMs and channel messages. Key behaviors: