feat(spam): score messages at ingest and hide likely spam

Add an optional, off-by-default spam-detection feature that scores each
message's spam likelihood at ingest, stores the score on the row, and lets
the display layer hide likely-spam by default behind a "show potential spam"
toggle. Nothing is ever dropped at ingest, so the threshold can be retuned
without reprocessing.

Scoring (collector/spam.py): windowed COUNT(*) over new
(path_prefix, received_at) and (sender_normalized, received_at) indexes —
joint path+sender signal plus a sender-name signal (trailing-digit suffix
stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or
absent, the name signal stands alone at full weight so local spam is still
flaggable. A background sweep re-scores recent rows with hindsight to catch
the leading edge of bursts. The collector logs each score (WARNING at/above
the threshold).

Display: the messages API gains include_spam and a master-switch-aware
hide-filter; the SPA shows the toggle + a badge only when the feature is on.

Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in
Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors
the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off.

Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration
for the three new columns + two indexes, and backend-aware collector test
fixtures (lifted db_backend/db_url into the shared conftest).

Also: move the meshcore-hub image pull_policy out of the base compose file.
It lived in docker-compose.yml as pull_policy: daily and made `make up` pull
the published image over a freshly built local one. Base is now policy-neutral
(default missing); dev sets pull_policy: build on the hub services so it only
ever uses local builds. Prod refreshes images via a manual `docker compose
... pull`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Louis KingandClaude Opus 4.8 committed 2026-06-23 00:11:39 +01:00
1 parent 6bf3c1c62b
commit c48db03afb
24 files changed
+1838 -125

No files matched your search

@@ -0,0 +1,52 @@
"""add spam scoring columns to messages
Revision ID: 38abdf4651fc
Revises: d4e5f6a7b8c9
Create Date: 2026-06-22 22:43:00.000000+00:00
Adds the three nullable spam-scoring columns (path_prefix, sender_normalized,
spam_score) and their two composite ``(*, received_at)`` indexes used by the
windowed COUNT(*) scorer. Uses batch mode so it applies on SQLite (which lacks
full ALTER TABLE) as well as Postgres.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "38abdf4651fc"
down_revision: Union[str, None] = "d4e5f6a7b8c9"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("messages", schema=None) as batch_op:
batch_op.add_column(
sa.Column("path_prefix", sa.String(length=48), nullable=True)
)
batch_op.add_column(
sa.Column("sender_normalized", sa.String(length=255), nullable=True)
)
batch_op.add_column(sa.Column("spam_score", sa.Float(), nullable=True))
batch_op.create_index(
"ix_messages_path_prefix_received_at",
["path_prefix", "received_at"],
unique=False,
)
batch_op.create_index(
"ix_messages_sender_normalized_received_at",
["sender_normalized", "received_at"],
unique=False,
)
def downgrade() -> None:
with op.batch_alter_table("messages", schema=None) as batch_op:
batch_op.drop_index("ix_messages_sender_normalized_received_at")
batch_op.drop_index("ix_messages_path_prefix_received_at")
batch_op.drop_column("spam_score")
batch_op.drop_column("sender_normalized")
batch_op.drop_column("path_prefix")