mirror of
https://github.com/MarekWo/mc-webui.git
synced 2026-03-28 17:42:45 +01:00
Feature: Add message archiving system with browse-by-date selector
Implements automatic daily archiving of messages to improve performance
and enable browsing historical chat by date.
Backend changes:
- Add APScheduler for daily archiving at midnight (00:00 UTC)
- Create app/archiver/manager.py with archive logic and scheduler
- Extend parser.py to read from archive files and filter by days
- Add archive configuration to config.py (MC_ARCHIVE_*)
API changes:
- Extend GET /api/messages with archive_date and days parameters
- Add GET /api/archives endpoint to list available archives
- Add POST /api/archive/trigger for manual archiving
Frontend changes:
- Add date selector dropdown in navbar for archive browsing
- Implement archive list loading and date selection
- Update formatTime() to show full dates in archive view
- Live view now shows only last 7 days (configurable)
Docker & Config:
- Add archive volume mount in docker-compose.yml
- Add MC_ARCHIVE_DIR, MC_ARCHIVE_ENABLED, MC_ARCHIVE_RETENTION_DAYS env vars
- Update .env.example with archive configuration section
Documentation:
- Update README.md with archive feature and usage instructions
- Update .claude/instructions.md with archive endpoints
Key features:
- Automatic daily archiving (midnight UTC)
- Live view filtered to last 7 days for better performance
- Browse historical messages by date via dropdown selector
- Archives stored as dated files: {device}.YYYY-MM-DD.msgs
- Original .msgs file never modified (safe, read-only approach)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -45,12 +45,15 @@ Location: `~/.config/meshcore/<device_name>.msgs` (JSON Lines)
|
||||
## Environment Variables
|
||||
|
||||
```
|
||||
MC_SERIAL_PORT - Serial device path
|
||||
MC_DEVICE_NAME - Device name (for .msgs file)
|
||||
MC_CONFIG_DIR - meshcore config directory
|
||||
MC_REFRESH_INTERVAL - Auto-refresh seconds (default: 60)
|
||||
MC_INACTIVE_HOURS - Contact cleanup threshold (default: 48)
|
||||
FLASK_PORT - Web server port (default: 5000)
|
||||
MC_SERIAL_PORT - Serial device path
|
||||
MC_DEVICE_NAME - Device name (for .msgs file)
|
||||
MC_CONFIG_DIR - meshcore config directory
|
||||
MC_REFRESH_INTERVAL - Auto-refresh seconds (default: 60)
|
||||
MC_INACTIVE_HOURS - Contact cleanup threshold (default: 48)
|
||||
MC_ARCHIVE_DIR - Archive directory path (default: /root/.archive/meshcore)
|
||||
MC_ARCHIVE_ENABLED - Enable automatic archiving (default: true)
|
||||
MC_ARCHIVE_RETENTION_DAYS - Days to show in live view (default: 7)
|
||||
FLASK_PORT - Web server port (default: 5000)
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
@@ -65,6 +68,8 @@ mc-webui/
|
||||
│ ├── meshcore/
|
||||
│ │ ├── cli.py # meshcli subprocess wrapper
|
||||
│ │ └── parser.py # .msgs file parser
|
||||
│ ├── archiver/
|
||||
│ │ └── manager.py # Archive scheduler and management
|
||||
│ ├── routes/
|
||||
│ │ ├── api.py # REST endpoints
|
||||
│ │ └── views.py # HTML views
|
||||
@@ -77,10 +82,14 @@ mc-webui/
|
||||
## API Endpoints
|
||||
|
||||
```
|
||||
GET /api/messages - List messages
|
||||
POST /api/messages - Send message
|
||||
GET /api/status - Connection status
|
||||
POST /api/contacts/cleanup - Remove inactive contacts
|
||||
GET /api/messages - List messages (supports ?archive_date=YYYY-MM-DD&days=N)
|
||||
POST /api/messages - Send message
|
||||
GET /api/status - Connection status
|
||||
POST /api/contacts/cleanup - Remove inactive contacts
|
||||
GET /api/archives - List available archives
|
||||
POST /api/archive/trigger - Manually trigger archiving
|
||||
GET /api/device/info - Device information
|
||||
POST /api/sync - Trigger message sync
|
||||
```
|
||||
|
||||
## Important Files
|
||||
|
||||
13
.env.example
13
.env.example
@@ -25,6 +25,19 @@ MC_REFRESH_INTERVAL=60
|
||||
# Hours of inactivity before contacts can be cleaned up
|
||||
MC_INACTIVE_HOURS=48
|
||||
|
||||
# ============================================
|
||||
# Archive Configuration
|
||||
# ============================================
|
||||
|
||||
# Directory for storing archived messages
|
||||
MC_ARCHIVE_DIR=/mnt/archive/meshcore
|
||||
|
||||
# Enable automatic daily archiving at midnight
|
||||
MC_ARCHIVE_ENABLED=true
|
||||
|
||||
# Number of days to show in live view (older messages available in archives)
|
||||
MC_ARCHIVE_RETENTION_DAYS=7
|
||||
|
||||
# ============================================
|
||||
# Flask Server Configuration
|
||||
# ============================================
|
||||
|
||||
20
README.md
20
README.md
@@ -6,12 +6,13 @@ A lightweight web interface for meshcore-cli, providing browser-based access to
|
||||
|
||||
**mc-webui** is a Flask-based web application that wraps `meshcore-cli`, eliminating the need for SSH/terminal access when using MeshCore chat on a Heltec V4 device connected to a Debian VM.
|
||||
|
||||
### Key Features (MVP)
|
||||
### Key Features
|
||||
|
||||
- 📱 **View messages** - Display chat history from Public channel with auto-refresh
|
||||
- ✉️ **Send messages** - Publish to Public channel
|
||||
- ✉️ **Send messages** - Publish to Public channel (200 char limit for LoRa)
|
||||
- 💬 **Reply to users** - Quick reply with `@[UserName]` format
|
||||
- 🧹 **Clean contacts** - Remove inactive contacts with configurable threshold
|
||||
- 📦 **Message archiving** - Automatic daily archiving with browse-by-date selector
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -76,6 +77,9 @@ All configuration is done via environment variables in the `.env` file:
|
||||
| `MC_CONFIG_DIR` | meshcore configuration directory | `/root/.config/meshcore` |
|
||||
| `MC_REFRESH_INTERVAL` | Auto-refresh interval (seconds) | `60` |
|
||||
| `MC_INACTIVE_HOURS` | Inactivity threshold for cleanup | `48` |
|
||||
| `MC_ARCHIVE_DIR` | Archive directory path | `/mnt/archive/meshcore` |
|
||||
| `MC_ARCHIVE_ENABLED` | Enable automatic archiving | `true` |
|
||||
| `MC_ARCHIVE_RETENTION_DAYS` | Days to show in live view | `7` |
|
||||
| `FLASK_HOST` | Listen address | `0.0.0.0` |
|
||||
| `FLASK_PORT` | Application port | `5000` |
|
||||
| `FLASK_DEBUG` | Debug mode | `false` |
|
||||
@@ -138,6 +142,18 @@ See [PRD.md](PRD.md) for detailed requirements and implementation plan.
|
||||
|
||||
The main page displays chat history from the Public channel (channel 0). Messages auto-refresh every 60 seconds by default.
|
||||
|
||||
By default, the live view shows messages from the last 7 days. Older messages are automatically archived and can be accessed via the date selector.
|
||||
|
||||
### Viewing Message Archives
|
||||
|
||||
Access historical messages using the date selector in the navbar:
|
||||
|
||||
1. Click the date dropdown in the navbar (next to Refresh button)
|
||||
2. Select a date to view archived messages for that day
|
||||
3. Select "Today (Live)" to return to live view
|
||||
|
||||
Archives are created automatically at midnight (00:00 UTC) each day. The live view always shows the most recent messages (last 7 days by default).
|
||||
|
||||
### Sending Messages
|
||||
|
||||
1. Type your message in the text field at the bottom
|
||||
|
||||
17
app/archiver/__init__.py
Normal file
17
app/archiver/__init__.py
Normal file
@@ -0,0 +1,17 @@
|
||||
"""
|
||||
Archive module - handles message archiving and management
|
||||
"""
|
||||
|
||||
from app.archiver.manager import (
|
||||
archive_messages,
|
||||
list_archives,
|
||||
get_archive_path,
|
||||
schedule_daily_archiving
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
'archive_messages',
|
||||
'list_archives',
|
||||
'get_archive_path',
|
||||
'schedule_daily_archiving'
|
||||
]
|
||||
282
app/archiver/manager.py
Normal file
282
app/archiver/manager.py
Normal file
@@ -0,0 +1,282 @@
|
||||
"""
|
||||
Archive manager - handles message archiving and scheduling
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from datetime import datetime, time
|
||||
from typing import List, Dict, Optional
|
||||
from apscheduler.schedulers.background import BackgroundScheduler
|
||||
from apscheduler.triggers.cron import CronTrigger
|
||||
|
||||
from app.config import config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Global scheduler instance
|
||||
_scheduler: Optional[BackgroundScheduler] = None
|
||||
|
||||
|
||||
def get_archive_path(archive_date: str) -> Path:
|
||||
"""
|
||||
Get the path to an archive file for a specific date.
|
||||
|
||||
Args:
|
||||
archive_date: Date in YYYY-MM-DD format
|
||||
|
||||
Returns:
|
||||
Path to archive file
|
||||
"""
|
||||
archive_dir = config.archive_dir_path
|
||||
filename = f"{config.MC_DEVICE_NAME}.{archive_date}.msgs"
|
||||
return archive_dir / filename
|
||||
|
||||
|
||||
def archive_messages(archive_date: Optional[str] = None) -> Dict[str, any]:
|
||||
"""
|
||||
Archive messages for a specific date by copying the .msgs file.
|
||||
|
||||
Args:
|
||||
archive_date: Date to archive in YYYY-MM-DD format.
|
||||
If None, uses yesterday's date.
|
||||
|
||||
Returns:
|
||||
Dict with success status and details
|
||||
"""
|
||||
try:
|
||||
# Determine date to archive
|
||||
if archive_date is None:
|
||||
from datetime import date, timedelta
|
||||
yesterday = date.today() - timedelta(days=1)
|
||||
archive_date = yesterday.strftime('%Y-%m-%d')
|
||||
|
||||
# Validate date format
|
||||
try:
|
||||
datetime.strptime(archive_date, '%Y-%m-%d')
|
||||
except ValueError:
|
||||
return {
|
||||
'success': False,
|
||||
'error': f'Invalid date format: {archive_date}. Expected YYYY-MM-DD'
|
||||
}
|
||||
|
||||
# Ensure archive directory exists
|
||||
archive_dir = config.archive_dir_path
|
||||
archive_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Get source .msgs file
|
||||
source_file = config.msgs_file_path
|
||||
if not source_file.exists():
|
||||
logger.warning(f"Source messages file not found: {source_file}")
|
||||
return {
|
||||
'success': False,
|
||||
'error': f'Messages file not found: {source_file}'
|
||||
}
|
||||
|
||||
# Get destination archive file
|
||||
dest_file = get_archive_path(archive_date)
|
||||
|
||||
# Check if archive already exists
|
||||
if dest_file.exists():
|
||||
logger.info(f"Archive already exists: {dest_file}")
|
||||
return {
|
||||
'success': True,
|
||||
'message': f'Archive already exists for {archive_date}',
|
||||
'archive_file': str(dest_file),
|
||||
'exists': True
|
||||
}
|
||||
|
||||
# Copy the file
|
||||
shutil.copy2(source_file, dest_file)
|
||||
|
||||
# Get file size
|
||||
file_size = dest_file.stat().st_size
|
||||
|
||||
logger.info(f"Archived messages to {dest_file} ({file_size} bytes)")
|
||||
|
||||
return {
|
||||
'success': True,
|
||||
'message': f'Successfully archived messages for {archive_date}',
|
||||
'archive_file': str(dest_file),
|
||||
'file_size': file_size,
|
||||
'archive_date': archive_date
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error archiving messages: {e}", exc_info=True)
|
||||
return {
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}
|
||||
|
||||
|
||||
def list_archives() -> List[Dict]:
|
||||
"""
|
||||
List all available archive files with metadata.
|
||||
|
||||
Returns:
|
||||
List of archive info dicts, sorted by date (newest first)
|
||||
"""
|
||||
archives = []
|
||||
|
||||
try:
|
||||
archive_dir = config.archive_dir_path
|
||||
|
||||
# Check if archive directory exists
|
||||
if not archive_dir.exists():
|
||||
logger.info(f"Archive directory does not exist: {archive_dir}")
|
||||
return []
|
||||
|
||||
# Pattern: {device_name}.YYYY-MM-DD.msgs
|
||||
pattern = f"{config.MC_DEVICE_NAME}.*.msgs"
|
||||
|
||||
for archive_file in archive_dir.glob(pattern):
|
||||
try:
|
||||
# Extract date from filename
|
||||
# Format: DeviceName.YYYY-MM-DD.msgs
|
||||
filename = archive_file.name
|
||||
date_part = filename.replace(f"{config.MC_DEVICE_NAME}.", "").replace(".msgs", "")
|
||||
|
||||
# Validate date format
|
||||
try:
|
||||
datetime.strptime(date_part, '%Y-%m-%d')
|
||||
except ValueError:
|
||||
logger.warning(f"Invalid archive filename format: {filename}")
|
||||
continue
|
||||
|
||||
# Get file stats
|
||||
stats = archive_file.stat()
|
||||
file_size = stats.st_size
|
||||
|
||||
# Count messages (read file)
|
||||
message_count = _count_messages_in_file(archive_file)
|
||||
|
||||
archives.append({
|
||||
'date': date_part,
|
||||
'file_size': file_size,
|
||||
'message_count': message_count,
|
||||
'file_path': str(archive_file)
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Error processing archive file {archive_file}: {e}")
|
||||
continue
|
||||
|
||||
# Sort by date, newest first
|
||||
archives.sort(key=lambda x: x['date'], reverse=True)
|
||||
|
||||
logger.info(f"Found {len(archives)} archive files")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing archives: {e}", exc_info=True)
|
||||
|
||||
return archives
|
||||
|
||||
|
||||
def _count_messages_in_file(file_path: Path) -> int:
|
||||
"""
|
||||
Count the number of valid message lines in a file.
|
||||
|
||||
Args:
|
||||
file_path: Path to the .msgs file
|
||||
|
||||
Returns:
|
||||
Number of messages
|
||||
"""
|
||||
import json
|
||||
|
||||
count = 0
|
||||
try:
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
data = json.loads(line)
|
||||
# Only count Public channel messages
|
||||
if data.get('channel_idx', 0) == 0 and data.get('type') in ['CHAN', 'SENT_CHAN']:
|
||||
count += 1
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.warning(f"Error counting messages in {file_path}: {e}")
|
||||
|
||||
return count
|
||||
|
||||
|
||||
def _archive_job():
|
||||
"""
|
||||
Background job that runs daily to archive messages.
|
||||
This is called by the scheduler at midnight.
|
||||
"""
|
||||
logger.info("Running daily archive job...")
|
||||
|
||||
if not config.MC_ARCHIVE_ENABLED:
|
||||
logger.info("Archiving is disabled, skipping")
|
||||
return
|
||||
|
||||
result = archive_messages()
|
||||
|
||||
if result['success']:
|
||||
logger.info(f"Archive job completed: {result.get('message', 'Success')}")
|
||||
else:
|
||||
logger.error(f"Archive job failed: {result.get('error', 'Unknown error')}")
|
||||
|
||||
|
||||
def schedule_daily_archiving():
|
||||
"""
|
||||
Initialize and start the background scheduler for daily archiving.
|
||||
Runs at midnight (00:00) local time.
|
||||
"""
|
||||
global _scheduler
|
||||
|
||||
if not config.MC_ARCHIVE_ENABLED:
|
||||
logger.info("Archiving is disabled in configuration")
|
||||
return
|
||||
|
||||
if _scheduler is not None:
|
||||
logger.warning("Scheduler already initialized")
|
||||
return
|
||||
|
||||
try:
|
||||
_scheduler = BackgroundScheduler(
|
||||
daemon=True,
|
||||
timezone='UTC' # Use UTC for consistency
|
||||
)
|
||||
|
||||
# Schedule job for midnight every day
|
||||
trigger = CronTrigger(hour=0, minute=0)
|
||||
|
||||
_scheduler.add_job(
|
||||
func=_archive_job,
|
||||
trigger=trigger,
|
||||
id='daily_archive',
|
||||
name='Daily Message Archive',
|
||||
replace_existing=True
|
||||
)
|
||||
|
||||
_scheduler.start()
|
||||
|
||||
logger.info("Archive scheduler started - will run daily at 00:00 UTC")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start archive scheduler: {e}", exc_info=True)
|
||||
|
||||
|
||||
def stop_scheduler():
|
||||
"""
|
||||
Stop the background scheduler.
|
||||
Called during application shutdown.
|
||||
"""
|
||||
global _scheduler
|
||||
|
||||
if _scheduler is not None:
|
||||
try:
|
||||
_scheduler.shutdown(wait=False)
|
||||
logger.info("Archive scheduler stopped")
|
||||
except Exception as e:
|
||||
logger.error(f"Error stopping scheduler: {e}")
|
||||
finally:
|
||||
_scheduler = None
|
||||
@@ -18,6 +18,11 @@ class Config:
|
||||
MC_REFRESH_INTERVAL = int(os.getenv('MC_REFRESH_INTERVAL', '60'))
|
||||
MC_INACTIVE_HOURS = int(os.getenv('MC_INACTIVE_HOURS', '48'))
|
||||
|
||||
# Archive configuration
|
||||
MC_ARCHIVE_DIR = os.getenv('MC_ARCHIVE_DIR', '/root/.archive/meshcore')
|
||||
MC_ARCHIVE_ENABLED = os.getenv('MC_ARCHIVE_ENABLED', 'true').lower() == 'true'
|
||||
MC_ARCHIVE_RETENTION_DAYS = int(os.getenv('MC_ARCHIVE_RETENTION_DAYS', '7'))
|
||||
|
||||
# Flask server configuration
|
||||
FLASK_HOST = os.getenv('FLASK_HOST', '0.0.0.0')
|
||||
FLASK_PORT = int(os.getenv('FLASK_PORT', '5000'))
|
||||
@@ -34,6 +39,11 @@ class Config:
|
||||
"""Get the base meshcli command with serial port"""
|
||||
return ['meshcli', '-s', self.MC_SERIAL_PORT]
|
||||
|
||||
@property
|
||||
def archive_dir_path(self) -> Path:
|
||||
"""Get the full path to archive directory"""
|
||||
return Path(self.MC_ARCHIVE_DIR)
|
||||
|
||||
def __repr__(self):
|
||||
return (
|
||||
f"Config(device={self.MC_DEVICE_NAME}, "
|
||||
|
||||
@@ -7,6 +7,7 @@ from flask import Flask
|
||||
from app.config import config
|
||||
from app.routes.views import views_bp
|
||||
from app.routes.api import api_bp
|
||||
from app.archiver.manager import schedule_daily_archiving
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
@@ -29,6 +30,13 @@ def create_app():
|
||||
app.register_blueprint(views_bp)
|
||||
app.register_blueprint(api_bp)
|
||||
|
||||
# Initialize archive scheduler if enabled
|
||||
if config.MC_ARCHIVE_ENABLED:
|
||||
schedule_daily_archiving()
|
||||
logger.info(f"Archive scheduler enabled - directory: {config.MC_ARCHIVE_DIR}")
|
||||
else:
|
||||
logger.info("Archive scheduler disabled")
|
||||
|
||||
logger.info(f"mc-webui started - device: {config.MC_DEVICE_NAME}")
|
||||
logger.info(f"Messages file: {config.msgs_file_path}")
|
||||
logger.info(f"Serial port: {config.MC_SERIAL_PORT}")
|
||||
|
||||
@@ -6,7 +6,7 @@ import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Optional
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timedelta
|
||||
from app.config import config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -69,17 +69,23 @@ def parse_message(line: Dict) -> Optional[Dict]:
|
||||
}
|
||||
|
||||
|
||||
def read_messages(limit: Optional[int] = None, offset: int = 0) -> List[Dict]:
|
||||
def read_messages(limit: Optional[int] = None, offset: int = 0, archive_date: Optional[str] = None, days: Optional[int] = None) -> List[Dict]:
|
||||
"""
|
||||
Read and parse messages from .msgs file.
|
||||
Read and parse messages from .msgs file or archive file.
|
||||
|
||||
Args:
|
||||
limit: Maximum number of messages to return (None = all)
|
||||
offset: Number of messages to skip from the end
|
||||
archive_date: If provided, read from archive file for this date (YYYY-MM-DD)
|
||||
days: If provided, filter messages from the last N days (only for live .msgs)
|
||||
|
||||
Returns:
|
||||
List of parsed message dictionaries, sorted by timestamp (oldest first)
|
||||
"""
|
||||
# If archive_date is provided, read from archive
|
||||
if archive_date:
|
||||
return read_archive_messages(archive_date, limit, offset)
|
||||
|
||||
msgs_file = config.msgs_file_path
|
||||
|
||||
if not msgs_file.exists():
|
||||
@@ -117,6 +123,10 @@ def read_messages(limit: Optional[int] = None, offset: int = 0) -> List[Dict]:
|
||||
# Sort by timestamp (oldest first)
|
||||
messages.sort(key=lambda m: m['timestamp'])
|
||||
|
||||
# Filter by days if specified
|
||||
if days is not None and days > 0:
|
||||
messages = filter_messages_by_days(messages, days)
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
messages = messages[:-offset] if offset < len(messages) else []
|
||||
@@ -147,3 +157,90 @@ def count_messages() -> int:
|
||||
Message count
|
||||
"""
|
||||
return len(read_messages())
|
||||
|
||||
|
||||
def read_archive_messages(archive_date: str, limit: Optional[int] = None, offset: int = 0) -> List[Dict]:
|
||||
"""
|
||||
Read messages from an archive file.
|
||||
|
||||
Args:
|
||||
archive_date: Archive date in YYYY-MM-DD format
|
||||
limit: Maximum number of messages to return (None = all)
|
||||
offset: Number of messages to skip from the end
|
||||
|
||||
Returns:
|
||||
List of parsed message dictionaries, sorted by timestamp (oldest first)
|
||||
"""
|
||||
from app.archiver.manager import get_archive_path
|
||||
|
||||
archive_file = get_archive_path(archive_date)
|
||||
|
||||
if not archive_file.exists():
|
||||
logger.warning(f"Archive file not found: {archive_file}")
|
||||
return []
|
||||
|
||||
messages = []
|
||||
|
||||
try:
|
||||
with open(archive_file, 'r', encoding='utf-8') as f:
|
||||
for line_num, line in enumerate(f, 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
try:
|
||||
data = json.loads(line)
|
||||
parsed = parse_message(data)
|
||||
if parsed:
|
||||
messages.append(parsed)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.warning(f"Invalid JSON at line {line_num} in archive: {e}")
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.error(f"Error parsing line {line_num} in archive: {e}")
|
||||
continue
|
||||
|
||||
except FileNotFoundError:
|
||||
logger.error(f"Archive file not found: {archive_file}")
|
||||
return []
|
||||
except Exception as e:
|
||||
logger.error(f"Error reading archive file: {e}")
|
||||
return []
|
||||
|
||||
# Sort by timestamp (oldest first)
|
||||
messages.sort(key=lambda m: m['timestamp'])
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
messages = messages[:-offset] if offset < len(messages) else []
|
||||
|
||||
if limit is not None and limit > 0:
|
||||
messages = messages[-limit:]
|
||||
|
||||
logger.info(f"Loaded {len(messages)} messages from archive {archive_date}")
|
||||
return messages
|
||||
|
||||
|
||||
def filter_messages_by_days(messages: List[Dict], days: int) -> List[Dict]:
|
||||
"""
|
||||
Filter messages to include only those from the last N days.
|
||||
|
||||
Args:
|
||||
messages: List of message dicts
|
||||
days: Number of days to include (from now)
|
||||
|
||||
Returns:
|
||||
Filtered list of messages
|
||||
"""
|
||||
if not messages:
|
||||
return []
|
||||
|
||||
# Calculate cutoff timestamp
|
||||
cutoff_date = datetime.now() - timedelta(days=days)
|
||||
cutoff_timestamp = cutoff_date.timestamp()
|
||||
|
||||
# Filter messages
|
||||
filtered = [msg for msg in messages if msg['timestamp'] >= cutoff_timestamp]
|
||||
|
||||
logger.info(f"Filtered {len(filtered)} messages from last {days} days (out of {len(messages)} total)")
|
||||
return filtered
|
||||
|
||||
@@ -3,9 +3,11 @@ REST API endpoints for mc-webui
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from flask import Blueprint, jsonify, request
|
||||
from app.meshcore import cli, parser
|
||||
from app.config import config
|
||||
from app.archiver import manager as archive_manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -15,11 +17,13 @@ api_bp = Blueprint('api', __name__, url_prefix='/api')
|
||||
@api_bp.route('/messages', methods=['GET'])
|
||||
def get_messages():
|
||||
"""
|
||||
Get list of messages from Public channel.
|
||||
Get list of messages from Public channel or archive.
|
||||
|
||||
Query parameters:
|
||||
limit (int): Maximum number of messages to return
|
||||
offset (int): Number of messages to skip from the end
|
||||
archive_date (str): View archive for specific date (YYYY-MM-DD format)
|
||||
days (int): Show only messages from last N days (live view only)
|
||||
|
||||
Returns:
|
||||
JSON with messages list
|
||||
@@ -27,13 +31,32 @@ def get_messages():
|
||||
try:
|
||||
limit = request.args.get('limit', type=int)
|
||||
offset = request.args.get('offset', default=0, type=int)
|
||||
archive_date = request.args.get('archive_date', type=str)
|
||||
days = request.args.get('days', type=int)
|
||||
|
||||
messages = parser.read_messages(limit=limit, offset=offset)
|
||||
# Validate archive_date format if provided
|
||||
if archive_date:
|
||||
try:
|
||||
datetime.strptime(archive_date, '%Y-%m-%d')
|
||||
except ValueError:
|
||||
return jsonify({
|
||||
'success': False,
|
||||
'error': f'Invalid date format: {archive_date}. Expected YYYY-MM-DD'
|
||||
}), 400
|
||||
|
||||
# Read messages (from archive or live .msgs file)
|
||||
messages = parser.read_messages(
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
archive_date=archive_date,
|
||||
days=days
|
||||
)
|
||||
|
||||
return jsonify({
|
||||
'success': True,
|
||||
'count': len(messages),
|
||||
'messages': messages
|
||||
'messages': messages,
|
||||
'archive_date': archive_date if archive_date else None
|
||||
}), 200
|
||||
|
||||
except Exception as e:
|
||||
@@ -242,3 +265,72 @@ def sync_messages():
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}), 500
|
||||
|
||||
|
||||
@api_bp.route('/archives', methods=['GET'])
|
||||
def get_archives():
|
||||
"""
|
||||
Get list of available message archives.
|
||||
|
||||
Returns:
|
||||
JSON with list of archives, each with:
|
||||
- date (str): Archive date in YYYY-MM-DD format
|
||||
- message_count (int): Number of messages in archive
|
||||
- file_size (int): Archive file size in bytes
|
||||
"""
|
||||
try:
|
||||
archives = archive_manager.list_archives()
|
||||
|
||||
return jsonify({
|
||||
'success': True,
|
||||
'archives': archives,
|
||||
'count': len(archives)
|
||||
}), 200
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing archives: {e}")
|
||||
return jsonify({
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}), 500
|
||||
|
||||
|
||||
@api_bp.route('/archive/trigger', methods=['POST'])
|
||||
def trigger_archive():
|
||||
"""
|
||||
Manually trigger message archiving.
|
||||
|
||||
JSON body:
|
||||
date (str): Date to archive in YYYY-MM-DD format (optional, defaults to yesterday)
|
||||
|
||||
Returns:
|
||||
JSON with archive operation result
|
||||
"""
|
||||
try:
|
||||
data = request.get_json() or {}
|
||||
archive_date = data.get('date')
|
||||
|
||||
# Validate date format if provided
|
||||
if archive_date:
|
||||
try:
|
||||
datetime.strptime(archive_date, '%Y-%m-%d')
|
||||
except ValueError:
|
||||
return jsonify({
|
||||
'success': False,
|
||||
'error': f'Invalid date format: {archive_date}. Expected YYYY-MM-DD'
|
||||
}), 400
|
||||
|
||||
# Trigger archiving
|
||||
result = archive_manager.archive_messages(archive_date)
|
||||
|
||||
if result['success']:
|
||||
return jsonify(result), 200
|
||||
else:
|
||||
return jsonify(result), 500
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error triggering archive: {e}")
|
||||
return jsonify({
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}), 500
|
||||
|
||||
@@ -6,11 +6,15 @@
|
||||
let lastMessageCount = 0;
|
||||
let autoRefreshInterval = null;
|
||||
let isUserScrolling = false;
|
||||
let currentArchiveDate = null; // Current selected archive date (null = live)
|
||||
|
||||
// Initialize on page load
|
||||
document.addEventListener('DOMContentLoaded', function() {
|
||||
console.log('mc-webui initialized');
|
||||
|
||||
// Load archive list
|
||||
loadArchiveList();
|
||||
|
||||
// Load initial messages
|
||||
loadMessages();
|
||||
|
||||
@@ -55,6 +59,12 @@ function setupEventListeners() {
|
||||
loadMessages();
|
||||
});
|
||||
|
||||
// Date selector (archive selection)
|
||||
document.getElementById('dateSelector').addEventListener('change', function(e) {
|
||||
currentArchiveDate = e.target.value || null;
|
||||
loadMessages();
|
||||
});
|
||||
|
||||
// Cleanup contacts button
|
||||
document.getElementById('cleanupBtn').addEventListener('click', function() {
|
||||
cleanupContacts();
|
||||
@@ -79,7 +89,18 @@ function setupEventListeners() {
|
||||
*/
|
||||
async function loadMessages() {
|
||||
try {
|
||||
const response = await fetch('/api/messages?limit=100');
|
||||
// Build URL with appropriate parameters
|
||||
let url = '/api/messages?limit=500';
|
||||
|
||||
if (currentArchiveDate) {
|
||||
// Loading archive
|
||||
url += `&archive_date=${currentArchiveDate}`;
|
||||
} else {
|
||||
// Loading live messages - show last 7 days only
|
||||
url += '&days=7';
|
||||
}
|
||||
|
||||
const response = await fetch(url);
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
@@ -354,6 +375,13 @@ function scrollToBottom() {
|
||||
*/
|
||||
function formatTime(timestamp) {
|
||||
const date = new Date(timestamp * 1000);
|
||||
|
||||
// When viewing archive, always show full date + time
|
||||
if (currentArchiveDate) {
|
||||
return date.toLocaleDateString() + ' ' + date.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
|
||||
}
|
||||
|
||||
// When viewing live messages, use relative time
|
||||
const now = new Date();
|
||||
const diffDays = Math.floor((now - date) / (1000 * 60 * 60 * 24));
|
||||
|
||||
@@ -393,6 +421,47 @@ function updateCharCounter() {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Load list of available archives
|
||||
*/
|
||||
async function loadArchiveList() {
|
||||
try {
|
||||
const response = await fetch('/api/archives');
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
populateDateSelector(data.archives);
|
||||
} else {
|
||||
console.error('Error loading archives:', data.error);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error loading archive list:', error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Populate the date selector dropdown with archive dates
|
||||
*/
|
||||
function populateDateSelector(archives) {
|
||||
const selector = document.getElementById('dateSelector');
|
||||
|
||||
// Keep the "Today (Live)" option
|
||||
// Remove all other options
|
||||
while (selector.options.length > 1) {
|
||||
selector.remove(1);
|
||||
}
|
||||
|
||||
// Add archive dates
|
||||
archives.forEach(archive => {
|
||||
const option = document.createElement('option');
|
||||
option.value = archive.date;
|
||||
option.textContent = `${archive.date} (${archive.message_count} msgs)`;
|
||||
selector.appendChild(option);
|
||||
});
|
||||
|
||||
console.log(`Loaded ${archives.length} archives`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Escape HTML to prevent XSS
|
||||
*/
|
||||
|
||||
@@ -25,10 +25,14 @@
|
||||
<small class="text-white-50">- {{ device_name }}</small>
|
||||
{% endif %}
|
||||
</span>
|
||||
<div class="d-flex align-items-center">
|
||||
<button class="btn btn-outline-light btn-sm me-2" id="refreshBtn" title="Refresh messages">
|
||||
<div class="d-flex align-items-center gap-2">
|
||||
<button class="btn btn-outline-light btn-sm" id="refreshBtn" title="Refresh messages">
|
||||
<i class="bi bi-arrow-clockwise"></i> Refresh
|
||||
</button>
|
||||
<select id="dateSelector" class="form-select form-select-sm" style="width: auto; min-width: 150px;" title="Select date">
|
||||
<option value="">Today (Live)</option>
|
||||
<!-- Archive dates loaded dynamically via JavaScript -->
|
||||
</select>
|
||||
<button class="btn btn-outline-light btn-sm" data-bs-toggle="modal" data-bs-target="#settingsModal" title="Settings">
|
||||
<i class="bi bi-gear"></i>
|
||||
</button>
|
||||
|
||||
@@ -13,12 +13,16 @@ services:
|
||||
- "${MC_SERIAL_PORT}:${MC_SERIAL_PORT}"
|
||||
volumes:
|
||||
- "${MC_CONFIG_DIR}:/root/.config/meshcore:rw"
|
||||
- "${MC_ARCHIVE_DIR:-./archive}:/root/.archive/meshcore:rw"
|
||||
environment:
|
||||
- MC_SERIAL_PORT=${MC_SERIAL_PORT}
|
||||
- MC_DEVICE_NAME=${MC_DEVICE_NAME}
|
||||
- MC_CONFIG_DIR=/root/.config/meshcore
|
||||
- MC_REFRESH_INTERVAL=${MC_REFRESH_INTERVAL:-60}
|
||||
- MC_INACTIVE_HOURS=${MC_INACTIVE_HOURS:-48}
|
||||
- MC_ARCHIVE_DIR=/root/.archive/meshcore
|
||||
- MC_ARCHIVE_ENABLED=${MC_ARCHIVE_ENABLED:-true}
|
||||
- MC_ARCHIVE_RETENTION_DAYS=${MC_ARCHIVE_RETENTION_DAYS:-7}
|
||||
- FLASK_HOST=${FLASK_HOST:-0.0.0.0}
|
||||
- FLASK_PORT=${FLASK_PORT:-5000}
|
||||
- FLASK_DEBUG=${FLASK_DEBUG:-false}
|
||||
|
||||
@@ -10,5 +10,11 @@ gunicorn==21.2.0
|
||||
# Configuration
|
||||
python-dotenv==1.0.0
|
||||
|
||||
# Scheduled Tasks
|
||||
APScheduler==3.10.4
|
||||
|
||||
# Date/Time Utilities
|
||||
python-dateutil==2.8.2
|
||||
|
||||
# Additional utilities (if needed later)
|
||||
# requests==2.31.0 # For future API integrations
|
||||
|
||||
Reference in New Issue
Block a user