Add an optional, off-by-default spam-detection feature that scores each
message's spam likelihood at ingest, stores the score on the row, and lets
the display layer hide likely-spam by default behind a "show potential spam"
toggle. Nothing is ever dropped at ingest, so the threshold can be retuned
without reprocessing.
Scoring (collector/spam.py): windowed COUNT(*) over new
(path_prefix, received_at) and (sender_normalized, received_at) indexes —
joint path+sender signal plus a sender-name signal (trailing-digit suffix
stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or
absent, the name signal stands alone at full weight so local spam is still
flaggable. A background sweep re-scores recent rows with hindsight to catch
the leading edge of bursts. The collector logs each score (WARNING at/above
the threshold).
Display: the messages API gains include_spam and a master-switch-aware
hide-filter; the SPA shows the toggle + a badge only when the feature is on.
Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in
Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors
the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off.
Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration
for the three new columns + two indexes, and backend-aware collector test
fixtures (lifted db_backend/db_url into the shared conftest).
Also: move the meshcore-hub image pull_policy out of the base compose file.
It lived in docker-compose.yml as pull_policy: daily and made `make up` pull
the published image over a freshly built local one. Base is now policy-neutral
(default missing); dev sets pull_policy: build on the hub services so it only
ever uses local builds. Prod refreshes images via a manual `docker compose
... pull`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise patch coverage on the v0.14.0 Postgres backend:
- test_db_migrate: _is_superuser plus the full migrate_sqlite_to_postgres
flow (dry-run, copy, non-empty-target refusal, --truncate, missing schema),
routing create_database_engine to SQLite files so a postgresql:// target
satisfies the guard while the run executes SQLite -> SQLite in CI.
- test_main: the `db migrate-to-postgres` CLI command via CliRunner
(success, dry-run, row-count mismatch, ValueError -> ClickException).
- conftest: neutralise dotenv.load_dotenv before collection. Importing the
CLI entrypoint runs load_dotenv() at import time, which leaked a local .env
into os.environ and broke unrelated config/redis tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settings classes use env_file=".env" for deployments, which also caused tests
to pick up a developer's local .env (e.g. DATABASE_BACKEND=postgres without
DATABASE_HOST), failing unrelated API/CLI tests. Add an autouse fixture that
sets env_file=None on CommonSettings and every subclass, so tests depend only
on defaults and explicit monkeypatch.setenv overrides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit establishes the complete foundation for the MeshCore Hub project:
- Project setup with pyproject.toml (Python 3.11+, all dependencies)
- Development tools: black, flake8, mypy, pytest configuration
- Pre-commit hooks for code quality
- Package structure with all components (interface, collector, api, web)
Common package includes:
- Pydantic settings for all component configurations
- SQLAlchemy models for nodes, messages, advertisements, traces, telemetry
- Pydantic schemas for events, API requests/responses, commands
- MQTT client utilities with topic builder
- Logging configuration
Database infrastructure:
- Alembic setup with initial migration for all tables
- Database manager with session handling
CLI entry point:
- Click-based CLI with subcommands for all components
- Database migration commands (upgrade, downgrade, revision)
Tests:
- Basic test suite for config and models
- pytest fixtures for in-memory database testing
Louis King