Discord bot, profile picture endpoint, channel management ux, streaming apis, refactor region logic

2026-03-28 17:42:58 +01:00 · 2025-09-13 02:57:52 +02:00
parent 2afbe80c1b
commit 8ac7d5eece
19 changed files with 2150 additions and 247 deletions
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -0,0 +1,205 @@
+# MeshCore Discord Bot
+
+This directory contains scripts for running long-running background processes alongside the Next.js server.
+
+## Discord Bot
+
+The Discord bot (`discord-bot.ts`) subscribes to the ClickHouse message stream with decryption enabled for the Seattle region and posts new messages to Discord via webhook.
+
+### Features
+
+- **Real-time streaming**: Subscribes to ClickHouse chat message stream
+- **Message decryption**: Automatically decrypts messages using known keys
+- **Discord integration**: Posts messages to Discord via webhook
+- **Message updates**: Messages with the same ID update existing Discord messages instead of posting new ones
+- **Error handling**: Graceful error handling with Discord error notifications
+- **Skip initial messages**: Only processes new messages, not historical ones
+
+### Configuration
+
+The bot is configured via environment variables:
+
+| Variable | Description | Default | Required |
+|----------|-------------|---------|----------|
+| `DISCORD_WEBHOOK_URL` | Discord webhook URL | - | Yes |
+| `MESH_REGION` | Mesh region to monitor | `seattle` | No |
+| `POLL_INTERVAL` | Polling interval in milliseconds | `1000` | No |
+| `MAX_ROWS_PER_POLL` | Maximum rows to fetch per poll | `50` | No |
+| `PRIVATE_KEYS` | Comma-separated list of private keys | - | No |
+
+### Usage
+
+#### Development
+
+```bash
+# Install dependencies
+npm install
+
+# Run with hot reload
+npm run discord-bot:dev
+```
+
+#### Production
+
+```bash
+# Run the bot
+npm run discord-bot
+```
+
+#### Environment Setup
+
+Create a `.env.local` file in the project root:
+
+```bash
+DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/YOUR_WEBHOOK_URL
+MESH_REGION=seattle
+POLL_INTERVAL=1000
+MAX_ROWS_PER_POLL=50
+PRIVATE_KEYS=key1,key2,key3
+```
+
+### Discord Webhook Setup
+
+1. Go to your Discord server settings
+2. Navigate to Integrations > Webhooks
+3. Create a new webhook
+4. Copy the webhook URL
+5. Set it as the `DISCORD_WEBHOOK_URL` environment variable
+
+### Message Format
+
+Messages are posted to Discord with the following format:
+
+- **Username**: MeshCore Chat
+- **Avatar**: Meshtastic logo
+- **Embed**: Rich embed with message details including:
+  - Sender and message text
+  - Channel hash
+  - Message ID
+  - Region
+  - Timestamps
+
+### Error Handling
+
+- Decryption failures are logged and skipped
+- Network errors are retried automatically
+- Critical errors are posted to Discord
+- Graceful shutdown on SIGINT/SIGTERM
+
+### Architecture
+
+The bot consists of several components:
+
+- **`discord-bot.ts`**: Main bot script with message processing logic
+- **`lib/discord.ts`**: Discord webhook client and message formatting utilities
+- **ClickHouse streaming**: Uses existing streaming infrastructure from the main app
+- **Message decryption**: Leverages existing meshcore decryption utilities
+
+### Docker Deployment
+
+The project includes Docker support for easy deployment with both the Next.js server and Discord bot.
+
+#### Prerequisites
+
+- Docker and Docker Compose installed
+- ClickHouse running on the Docker host (default port 8123)
+- External Docker network `shared-network` must exist
+
+#### Setup
+
+1. **Create the required external network:**
+   ```bash
+   docker network create shared-network
+   ```
+
+2. **Set up environment variables:**
+   ```bash
+   # Copy the example file
+   cp scripts/docker.env.example .env
+   
+   # Edit .env with your configuration
+   nano .env
+   ```
+
+3. **Build and start services:**
+   ```bash
+   # Start both Next.js server and Discord bot
+   docker-compose up --build
+   
+   # Or run in background
+   docker-compose up -d --build
+   ```
+
+4. **Access the application:**
+   - Next.js server: http://localhost:3001
+   - Discord bot: Runs in background, check logs with `docker-compose logs discord-bot`
+
+#### Docker Services
+
+- **meshexplorer**: Next.js web application (uses `Dockerfile`)
+- **discord-bot**: Discord bot for chat message streaming (uses `Dockerfile.bot`)
+
+#### Docker Images
+
+The project uses two separate Dockerfiles for optimal image sizes:
+
+- **`Dockerfile`**: Optimized for Next.js standalone output (smaller, faster)
+- **`Dockerfile.bot`**: Optimized for TypeScript execution with tsx (includes source files)
+
+#### Environment Variables
+
+All configuration is loaded from environment variables:
+
+| Variable | Description | Default | Required |
+|----------|-------------|---------|----------|
+| `CLICKHOUSE_HOST` | ClickHouse server hostname | `clickhouse` | No |
+| `CLICKHOUSE_PORT` | ClickHouse server port | `8123` | No |
+| `CLICKHOUSE_USER` | ClickHouse username | `default` | No |
+| `CLICKHOUSE_PASSWORD` | ClickHouse password | `password` | No |
+| `NEXT_PUBLIC_API_URL` | Override API base URL | - | No |
+| `DISCORD_WEBHOOK_URL` | Discord webhook URL | - | Yes |
+| `MESH_REGION` | Mesh region to monitor | `seattle` | No |
+| `POLL_INTERVAL` | Polling interval in milliseconds | `1000` | No |
+| `MAX_ROWS_PER_POLL` | Maximum rows to fetch per poll | `50` | No |
+| `PRIVATE_KEYS` | Comma-separated private keys | - | No |
+
+#### Docker Commands
+
+```bash
+# View logs
+docker-compose logs -f discord-bot
+docker-compose logs -f meshexplorer
+
+# Restart services
+docker-compose restart discord-bot
+docker-compose restart meshexplorer
+
+# Stop services
+docker-compose down
+
+# Rebuild and restart
+docker-compose up --build --force-recreate
+
+# Build individual services
+docker-compose build meshexplorer
+docker-compose build discord-bot
+
+# Run only specific service
+docker-compose up meshexplorer
+docker-compose up discord-bot
+
+# Build and run Discord bot only
+docker build -f Dockerfile.bot -t meshexplorer-bot .
+docker run --env-file .env meshexplorer-bot
+```
+
+### Monitoring
+
+The bot logs important events:
+
+- Message processing status
+- Decryption success/failure
+- Discord API calls
+- Error conditions
+
+Check the console output for real-time monitoring of bot activity.
--- a/scripts/discord-bot.ts
+++ b/scripts/discord-bot.ts
@@ -0,0 +1,215 @@
+#!/usr/bin/env node
+
+/**
+ * Discord Bot for MeshCore Chat Messages
+ * 
+ * This script subscribes to the ClickHouse message stream with decryption enabled
+ * for the Seattle region and posts new messages to Discord via webhook.
+ * 
+ * Messages with the same ID will update the existing Discord message instead of
+ * posting a new one.
+ */
+
+import { createClickHouseStreamer, createChatMessagesStreamerConfig } from '../src/lib/clickhouse/streaming';
+import { decryptMeshcoreGroupMessage } from '../src/lib/meshcore';
+import { DiscordWebhookClient, formatMeshcoreMessageForDiscord } from './lib/discord';
+
+interface BotConfig {
+  webhookUrl: string;
+  region: string;
+  pollInterval: number;
+  maxRowsPerPoll: number;
+  privateKeys: string[];
+}
+
+class MeshCoreDiscordBot {
+  private config: BotConfig;
+  private discordClient: DiscordWebhookClient;
+  private isRunning = false;
+  private streamer: any;
+
+  constructor(config: BotConfig) {
+    this.config = config;
+    this.discordClient = new DiscordWebhookClient(config.webhookUrl);
+  }
+
+  async start() {
+    if (this.isRunning) {
+      console.log('Bot is already running');
+      return;
+    }
+
+    this.isRunning = true;
+    console.log('Starting MeshCore Discord Bot...');
+    console.log(`Region: ${this.config.region}`);
+    console.log(`Poll interval: ${this.config.pollInterval}ms`);
+    console.log(`Max rows per poll: ${this.config.maxRowsPerPoll}`);
+
+    // Create streaming configuration
+    const streamerConfig = createChatMessagesStreamerConfig(undefined, this.config.region);
+    streamerConfig.pollInterval = this.config.pollInterval;
+    streamerConfig.maxRowsPerPoll = this.config.maxRowsPerPoll;
+    streamerConfig.skipInitialMessages = true; // Skip initial messages, only get new ones
+
+    this.streamer = createClickHouseStreamer(streamerConfig);
+
+    try {
+      // Start streaming
+      for await (const result of this.streamer({})) {
+        await this.processMessage(result.row);
+      }
+    } catch (error) {
+      console.error('Streaming error:', error);
+      throw error;
+    }
+  }
+
+  private async processMessage(message: any) {
+    try {
+      console.log(`Processing message ${message.message_id} from channel ${message.channel_hash}`);
+
+      // Decrypt the message
+      const decrypted = await this.decryptMessage(message);
+      
+      if (!decrypted) {
+        console.log(`Failed to decrypt message ${message.message_id}, skipping Discord post`);
+        return;
+      }
+
+      // Format message for Discord
+      const discordMessage = formatMeshcoreMessageForDiscord(message, decrypted);
+
+      // Post or update message in Discord
+      await this.discordClient.postOrUpdateMessage(message.message_id, discordMessage);
+
+      console.log(`Successfully processed message ${message.message_id}: ${decrypted.text}`);
+
+    } catch (error) {
+      console.error(`Error processing message ${message.message_id}:`, error);
+      
+      // Don't send error messages to Discord for processing errors
+      // Just log them for monitoring
+    }
+  }
+
+  private async decryptMessage(message: any): Promise<any> {
+    const PUBLIC_MESHCORE_KEY = "izOH6cXN6mrJ5e26oRXNcg==";
+    const allKeys = [PUBLIC_MESHCORE_KEY, ...this.config.privateKeys];
+
+    try {
+      const decrypted = await decryptMeshcoreGroupMessage({
+        encrypted_message: message.encrypted_message,
+        mac: message.mac,
+        channel_hash: message.channel_hash,
+        knownKeys: allKeys,
+        parse: true
+      });
+
+      return decrypted;
+    } catch (error) {
+      console.warn(`Decryption failed for message ${message.message_id}:`, error);
+      return null;
+    }
+  }
+
+  stop() {
+    this.isRunning = false;
+    console.log('Stopping MeshCore Discord Bot...');
+  }
+
+  getStatus() {
+    return {
+      isRunning: this.isRunning,
+      region: this.config.region,
+      messageMappings: this.discordClient.getAllMappings().size
+    };
+  }
+}
+
+// Main execution
+async function main() {
+  // Get configuration from environment variables
+  const webhookUrl = process.env.DISCORD_WEBHOOK_URL;
+  const region = process.env.MESH_REGION || 'seattle';
+  const pollInterval = parseInt(process.env.POLL_INTERVAL || '1000', 10);
+  const maxRowsPerPoll = parseInt(process.env.MAX_ROWS_PER_POLL || '50', 10);
+  const privateKeys = process.env.PRIVATE_KEYS ? process.env.PRIVATE_KEYS.split(',').filter(key => key.trim()) : [];
+
+  // Validate required configuration
+  if (!webhookUrl) {
+    console.error('Error: DISCORD_WEBHOOK_URL environment variable is required');
+    process.exit(1);
+  }
+
+  // Validate webhook URL format
+  if (!webhookUrl.startsWith('https://discord.com/api/webhooks/')) {
+    console.error('Error: DISCORD_WEBHOOK_URL must be a valid Discord webhook URL');
+    process.exit(1);
+  }
+
+  // Validate region
+  const allowedRegions = ['seattle', 'portland', 'boston'];
+  if (!allowedRegions.includes(region)) {
+    console.error(`Error: MESH_REGION must be one of: ${allowedRegions.join(', ')}`);
+    process.exit(1);
+  }
+
+  console.log('Configuration:');
+  console.log(`  Webhook URL: ${webhookUrl.substring(0, 50)}...`);
+  console.log(`  Region: ${region}`);
+  console.log(`  Poll interval: ${pollInterval}ms`);
+  console.log(`  Max rows per poll: ${maxRowsPerPoll}`);
+  console.log(`  Private keys: ${privateKeys.length}`);
+
+  // Create and start the bot
+  const bot = new MeshCoreDiscordBot({
+    webhookUrl,
+    region,
+    pollInterval,
+    maxRowsPerPoll,
+    privateKeys
+  });
+
+  // Handle graceful shutdown
+  process.on('SIGINT', async () => {
+    console.log('\nReceived SIGINT, shutting down gracefully...');
+    bot.stop();
+    process.exit(0);
+  });
+
+  process.on('SIGTERM', async () => {
+    console.log('\nReceived SIGTERM, shutting down gracefully...');
+    bot.stop();
+    process.exit(0);
+  });
+
+  // Handle uncaught exceptions
+  process.on('uncaughtException', (error) => {
+    console.error('Uncaught Exception:', error);
+    bot.stop();
+    process.exit(1);
+  });
+
+  process.on('unhandledRejection', (reason, promise) => {
+    console.error('Unhandled Rejection at:', promise, 'reason:', reason);
+    bot.stop();
+    process.exit(1);
+  });
+
+  try {
+    await bot.start();
+  } catch (error) {
+    console.error('Bot failed to start:', error);
+    process.exit(1);
+  }
+}
+
+// Run the bot
+if (require.main === module) {
+  main().catch((error) => {
+    console.error('Fatal error:', error);
+    process.exit(1);
+  });
+}
+
+export { MeshCoreDiscordBot };
--- a/scripts/lib/discord.ts
+++ b/scripts/lib/discord.ts
@@ -0,0 +1,190 @@
+/**
+ * Discord webhook integration utilities for posting and updating messages
+ */
+
+export interface DiscordWebhookMessage {
+  content?: string;
+  username?: string;
+  avatar_url?: string;
+  embeds?: DiscordEmbed[];
+  flags?: number;
+}
+
+export interface DiscordEmbed {
+  title?: string;
+  description?: string;
+  color?: number;
+  fields?: DiscordEmbedField[];
+  timestamp?: string;
+  footer?: DiscordEmbedFooter;
+}
+
+export interface DiscordEmbedField {
+  name: string;
+  value: string;
+  inline?: boolean;
+}
+
+export interface DiscordEmbedFooter {
+  text: string;
+  icon_url?: string;
+}
+
+export interface DiscordWebhookResponse {
+  id: string;
+  channel_id: string;
+  content: string;
+  timestamp: string;
+  edited_timestamp?: string;
+}
+
+export class DiscordWebhookClient {
+  private webhookUrl: string;
+  private messageIdMap: Map<string, string> = new Map(); // message_id -> discord_message_id
+
+  constructor(webhookUrl: string) {
+    this.webhookUrl = webhookUrl;
+  }
+
+  /**
+   * Post a new message to Discord
+   */
+  async postMessage(message: DiscordWebhookMessage): Promise<DiscordWebhookResponse> {
+    // Add wait=true to get the message ID in response
+    const url = new URL(this.webhookUrl);
+    url.searchParams.set('wait', 'true');
+    
+    const response = await fetch(url.toString(), {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+      },
+      body: JSON.stringify(message),
+    });
+
+    if (!response.ok) {
+      throw new Error(`Discord webhook failed: ${response.status} ${response.statusText}`);
+    }
+
+    return await response.json();
+  }
+
+  /**
+   * Update an existing Discord message
+   */
+  async updateMessage(discordMessageId: string, message: DiscordWebhookMessage): Promise<DiscordWebhookResponse> {
+    // Use the correct URL format for updating messages
+    const updateUrl = `${this.webhookUrl}/messages/${discordMessageId}`;
+    
+    const response = await fetch(updateUrl, {
+      method: 'PATCH',
+      headers: {
+        'Content-Type': 'application/json',
+      },
+      body: JSON.stringify(message),
+    });
+
+    if (!response.ok) {
+      const errorText = await response.text();
+      throw new Error(`Discord webhook update failed: ${response.status} ${response.statusText} - ${errorText}`);
+    }
+
+    return await response.json();
+  }
+
+  /**
+   * Post or update a message based on message ID mapping
+   */
+  async postOrUpdateMessage(
+    messageId: string, 
+    message: DiscordWebhookMessage
+  ): Promise<DiscordWebhookResponse> {
+    const existingDiscordId = this.messageIdMap.get(messageId);
+    
+    if (existingDiscordId) {
+      // Update existing message
+      try {
+        console.log(`Updating Discord message ${existingDiscordId} for meshcore message ${messageId}`);
+        const result = await this.updateMessage(existingDiscordId, message);
+        console.log(`Successfully updated Discord message ${existingDiscordId} for meshcore message ${messageId}`);
+        return result;
+      } catch (error) {
+        console.warn(`Failed to update Discord message ${existingDiscordId} for meshcore message ${messageId}, posting new message:`, error);
+        // If update fails, remove the old mapping and post a new message
+        this.messageIdMap.delete(messageId);
+        const result = await this.postMessage(message);
+        this.messageIdMap.set(messageId, result.id);
+        console.log(`Posted new Discord message ${result.id} for meshcore message ${messageId} (after update failure)`);
+        return result;
+      }
+    } else {
+      // Post new message
+      console.log(`Posting new Discord message for meshcore message ${messageId}`);
+      const result = await this.postMessage(message);
+      this.messageIdMap.set(messageId, result.id);
+      console.log(`Posted new Discord message ${result.id} for meshcore message ${messageId}`);
+      return result;
+    }
+  }
+
+  /**
+   * Get the Discord message ID for a given meshcore message ID
+   */
+  getDiscordMessageId(messageId: string): string | undefined {
+    return this.messageIdMap.get(messageId);
+  }
+
+  /**
+   * Remove a message ID mapping (useful for cleanup)
+   */
+  removeMessageMapping(messageId: string): boolean {
+    return this.messageIdMap.delete(messageId);
+  }
+
+  /**
+   * Get all message ID mappings
+   */
+  getAllMappings(): Map<string, string> {
+    return new Map(this.messageIdMap);
+  }
+
+  /**
+   * Clear all message ID mappings
+   */
+  clearMappings(): void {
+    this.messageIdMap.clear();
+  }
+}
+
+/**
+ * Format a meshcore chat message for Discord
+ */
+export function formatMeshcoreMessageForDiscord(
+  message: any,
+  decrypted?: {
+    timestamp: number;
+    msgType: number;
+    sender: string;
+    text: string;
+    rawText: string;
+  }
+): DiscordWebhookMessage {
+  const sender = decrypted?.sender || 'Unknown';
+  const text = decrypted?.text || '[Encrypted Message]';
+  
+  // Calculate how many times the message was heard
+  const heardCount = message.origin_path_info ? message.origin_path_info.length : 0;
+  
+  // Format the message content with the requested format
+  const content = `${text}\n-# _Heard ${heardCount} times by [MeshExplorer](https://map.w0z.is/messages)_`;
+
+  // Generate profile picture URL using the new API
+  const profilePictureUrl = `https://map.w0z.is/api/meshcore/profilepicture.png?name=${encodeURIComponent(sender)}`;
+
+  return {
+    username: sender,
+    avatar_url: profilePictureUrl,
+    content: content,
+    flags: 4 // SUPPRESS_EMBEDS flag
+  };
+}