mirror of
https://github.com/ipnet-mesh/meshcore-hub.git
synced 2026-10-30 19:52:03 +01:00
feat(spam): score messages at ingest and hide likely spam
Add an optional, off-by-default spam-detection feature that scores each message's spam likelihood at ingest, stores the score on the row, and lets the display layer hide likely-spam by default behind a "show potential spam" toggle. Nothing is ever dropped at ingest, so the threshold can be retuned without reprocessing. Scoring (collector/spam.py): windowed COUNT(*) over new (path_prefix, received_at) and (sender_normalized, received_at) indexes — joint path+sender signal plus a sender-name signal (trailing-digit suffix stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or absent, the name signal stands alone at full weight so local spam is still flaggable. A background sweep re-scores recent rows with hindsight to catch the leading edge of bursts. The collector logs each score (WARNING at/above the threshold). Display: the messages API gains include_spam and a master-switch-aware hide-filter; the SPA shows the toggle + a badge only when the feature is on. Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off. Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration for the three new columns + two indexes, and backend-aware collector test fixtures (lifted db_backend/db_url into the shared conftest). Also: move the meshcore-hub image pull_policy out of the base compose file. It lived in docker-compose.yml as pull_policy: daily and made `make up` pull the published image over a freshly built local one. Base is now policy-neutral (default missing); dev sets pull_policy: build on the hub services so it only ever uses local builds. Prod refreshes images via a manual `docker compose ... pull`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
1 parent
6bf3c1c62b
commit
c48db03afb
24 files changed
+1838
-125
No files matched your search
@@ -0,0 +1,52 @@
|
||||
"""add spam scoring columns to messages
|
||||
|
||||
Revision ID: 38abdf4651fc
|
||||
Revises: d4e5f6a7b8c9
|
||||
Create Date: 2026-06-22 22:43:00.000000+00:00
|
||||
|
||||
Adds the three nullable spam-scoring columns (path_prefix, sender_normalized,
|
||||
spam_score) and their two composite ``(*, received_at)`` indexes used by the
|
||||
windowed COUNT(*) scorer. Uses batch mode so it applies on SQLite (which lacks
|
||||
full ALTER TABLE) as well as Postgres.
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "38abdf4651fc"
|
||||
down_revision: Union[str, None] = "d4e5f6a7b8c9"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("messages", schema=None) as batch_op:
|
||||
batch_op.add_column(
|
||||
sa.Column("path_prefix", sa.String(length=48), nullable=True)
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("sender_normalized", sa.String(length=255), nullable=True)
|
||||
)
|
||||
batch_op.add_column(sa.Column("spam_score", sa.Float(), nullable=True))
|
||||
batch_op.create_index(
|
||||
"ix_messages_path_prefix_received_at",
|
||||
["path_prefix", "received_at"],
|
||||
unique=False,
|
||||
)
|
||||
batch_op.create_index(
|
||||
"ix_messages_sender_normalized_received_at",
|
||||
["sender_normalized", "received_at"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("messages", schema=None) as batch_op:
|
||||
batch_op.drop_index("ix_messages_sender_normalized_received_at")
|
||||
batch_op.drop_index("ix_messages_path_prefix_received_at")
|
||||
batch_op.drop_column("spam_score")
|
||||
batch_op.drop_column("sender_normalized")
|
||||
batch_op.drop_column("path_prefix")
|
||||
Reference in new issue
Block a user