adam/meshview
1
0
Fork 0
mirror of https://github.com/pablorevilla-meshtastic/meshview.git synced 2026-07-05 17:31:26 +02:00
Code Issues Packages Projects Releases Wiki Activity

Version 3.0.0 Feature Release - Target Before Thanksgiving! (#96)

Browse Source 
* Add alembic DB schema management (#86)

* Use alembic
* add creation helper
* example migration tool

* Store UTC int time in DB (#81)

* use UTC int time

* Remove old index notes script -- no longer needed

* modify alembic to support cleaner migrations

* add /version json endpoint

* move technical docs

* remove old migrate script

* add readme in docs:

* more doc tidy

* rm

* update api docs

* ignore other database files

* health endpoint

* alembic log format

* break out api calls in to their own file to reduce footprint

* ruff and docs

* vuln

* Improves arguments in mvrun.py

* Set dbcleanup.log location configurable

* mvrun work

* fallback if missing config

* remove unused loop

* improve migrations and fix logging problem with mqtt

* Container using slim/uv

* auto build containers

* symlink

* fix symlink

* checkout and containerfile

* make /app owned by ap0p

* Traceroute Return Path logged and displayed (#97)


* traceroute returns are now logged and /packetlist now graphs the correct data for a return route
* now using alembic to update schema
* HOWTO - Alembic

---------

Co-authored-by: Joel Krauska <jkrauska@gmail.com>

* DB Backups

* backups and cleanups are different

* ruff

* Docker Docs

* setup-dev

* graphviz for dot in Container

* Summary of 3.0.0 stuff

* Alembic was blocking mqtt logs

* Add us first/last timestamps to node table too

* Worked on /api/packet. Needed to modify
- Store.py to read the new time data
- api.py to present the new time data
- firehose.html chat.html and map.html now use the new apis and the time is the browser local time

* Worked on /api/packet. Needed to modify
- Store.py to read the new time data
- api.py to present the new time data
- firehose.html chat.html and map.html now use the new apis and the time is the browser local time

* Improves container build (#94)

* Worked on /api/packet. Needed to modify
- Store.py to read the new time data
- api.py to present the new time data
- firehose.html chat.html and map.html now use the new apis and the time is the browser local time

* Worked on /api/packet. Needed to modify
- Store.py to read the new time data
- api.py to present the new time data
- firehose.html chat.html and map.html now use the new apis and the time is the browser local time

* Worked on /api/packet. Needed to modify
- Added new api endpoint /api/packets_seen
- Modified web.py and store.py to support changes to APIs.
- Started to work on new_node.html and new_packet.html for presentation of data.

* Worked on /api/packet. Needed to modify
- Added new api endpoint /api/packets_seen
- Modified web.py and store.py to support changes to APIs.
- Started to work on new_node.html and new_packet.html for presentation of data.

* Finishing up all the pages for the 3.0 release.

Now all pages are functional.

* Finishing up all the pages for the 3.0 release.

Now all pages are functional.

* fix ruff format

* more ruff

* Finishing up all the pages for the 3.0 release.

Now all pages are functional.

* Finishing up all the pages for the 3.0 release.

Now all pages are functional.

* pyproject.toml requirements

* use sys.executable

* fix 0 epoch dates in /chat

* Make the robots do our bidding

* another compatibility fix when _us is empty and we need to sort by BOTH old and new

* Finishing up all the pages for the 3.0 release.

Now all pages are functional.

* Finishing up all the pages for the 3.0 release.

Now all pages are functional.

* Remamed new_node to node. shorter and descriptive.

* Remamed new_node to node. shorter and descriptive.

* Remamed new_node to node. shorter and descriptive.

* Remamed new_node to node. shorter and descriptive.

* Remamed new_node to node. shorter and descriptive.

* Remamed new_node to node. shorter and descriptive.

* More changes... almost ready for release.

Ranamed 2 pages for easy or reading.

* Fix the net page as it was not showing the date information

* Fix the net page as it was not showing the date information

* Fix the net page as it was not showing the date information

* Fix the net page as it was not showing the date information

* ruff

---------

Co-authored-by: Óscar García Amor <ogarcia@connectical.com>
Co-authored-by: Jim Schrempp <jschrempp@users.noreply.github.com>
Co-authored-by: Pablo Revilla <pablorevilla@gmail.com>
This commit is contained in:
Joel Krauska
2025-11-28 11:17:20 -08:00
committed by GitHub
parent e68cdf8cc1
commit e77428661c
70 changed files with 7027 additions and 4069 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ALEMBIC_SETUP.md
+361
View File
@@ -0,0 +1,361 @@
# Alembic Database Migration Setup
This document describes the automatic database migration system implemented for MeshView using Alembic.
## Overview
The system provides automatic database schema migrations with coordination between the writer app (startdb.py) and reader app (web.py):
- **Writer App**: Automatically runs pending migrations on startup
- **Reader App**: Waits for migrations to complete before starting
## Architecture
### Key Components
1. **`meshview/migrations.py`** - Migration management utilities
- `run_migrations()` - Runs pending migrations (writer app)
- `wait_for_migrations()` - Waits for schema to be current (reader app)
- `is_database_up_to_date()` - Checks schema version
- Migration status tracking table
2. **`alembic/`** - Alembic migration directory
- `env.py` - Configured for async SQLAlchemy support
- `versions/` - Migration scripts directory
- `alembic.ini` - Alembic configuration
3. **Modified Apps**:
- `startdb.py` - Writer app that runs migrations before MQTT ingestion
- `meshview/web.py` - Reader app that waits for schema updates
## How It Works - Automatic In-Place Updates
### ✨ Fully Automatic Operation
**No manual migration commands needed!** The database schema updates automatically when you:
1. Deploy new code with migration files
2. Restart the applications
### Writer App (startdb.py) Startup Sequence
1. Initialize database connection
2. Create migration status tracking table
3. Set "migration in progress" flag
4. **🔄 Automatically run any pending Alembic migrations** (synchronously)
- Detects current schema version
- Compares to latest available migration
- Runs all pending migrations in sequence
- Updates database schema in place
5. Clear "migration in progress" flag
6. Start MQTT ingestion and other tasks
### Reader App (web.py) Startup Sequence
1. Initialize database connection
2. **Check database schema version**
3. If not up to date:
- Wait up to 60 seconds (30 retries × 2 seconds)
- Check every 2 seconds for schema updates
- Automatically proceeds once writer completes migrations
4. Once schema is current, start web server
### 🎯 Key Point: Zero Manual Steps
When you deploy new code with migrations:
```bash
# Just start the apps - migrations happen automatically!
./env/bin/python startdb.py # Migrations run here automatically
./env/bin/python main.py # Waits for migrations, then starts
```
**The database updates itself!** No need to run `alembic upgrade` manually.
### Coordination
The apps coordinate using:
- **Alembic version table** (`alembic_version`) - Tracks current schema version
- **Migration status table** (`migration_status`) - Optional flag for "in progress" state
## Creating New Migrations
### Using the helper script:
```bash
./env/bin/python create_migration.py
```
### Manual creation:
```bash
./env/bin/alembic revision --autogenerate -m "Description of changes"
```
This will:
1. Compare current database schema with SQLAlchemy models
2. Generate a migration script in `alembic/versions/`
3. Automatically detect most schema changes
### Manual migration (advanced):
```bash
./env/bin/alembic revision -m "Manual migration"
```
Then edit the generated file to add custom migration logic.
## Running Migrations
### Automatic (Recommended)
Migrations run automatically when the writer app starts:
```bash
./env/bin/python startdb.py
```
### Manual
To run migrations manually:
```bash
./env/bin/alembic upgrade head
```
To downgrade:
```bash
./env/bin/alembic downgrade -1 # Go back one version
./env/bin/alembic downgrade base # Go back to beginning
```
## Checking Migration Status
Check current database version:
```bash
./env/bin/alembic current
```
View migration history:
```bash
./env/bin/alembic history
```
## Benefits
1. **Zero Manual Intervention**: Migrations run automatically on startup
2. **Safe Coordination**: Reader won't connect to incompatible schema
3. **Version Control**: All schema changes tracked in git
4. **Rollback Capability**: Can downgrade if needed
5. **Auto-generation**: Most migrations created automatically from model changes
## Migration Workflow
### Development Process
1. **Modify SQLAlchemy models** in `meshview/models.py`
2. **Create migration**:
```bash
./env/bin/python create_migration.py
```
3. **Review generated migration** in `alembic/versions/`
4. **Test migration**:
- Stop all apps
- Start writer app (migrations run automatically)
- Start reader app (waits for schema to be current)
5. **Commit migration** to version control
### Production Deployment
1. **Deploy new code** with migration scripts
2. **Start writer app** - Migrations run automatically
3. **Start reader app** - Waits for migrations, then starts
4. **Monitor logs** for migration success
## Troubleshooting
### Migration fails
Check logs in writer app for error details. To manually fix:
```bash
./env/bin/alembic current # Check current version
./env/bin/alembic history # View available versions
./env/bin/alembic upgrade head # Try manual upgrade
```
### Reader app won't start (timeout)
Check if writer app is running and has completed migrations:
```bash
./env/bin/alembic current
```
### Reset to clean state
⚠️ **Warning: This will lose all data**
```bash
rm packets.db # Or your database file
./env/bin/alembic upgrade head # Create fresh schema
```
## File Structure
```
meshview/
├── alembic.ini # Alembic configuration
├── alembic/
│ ├── env.py # Async-enabled migration runner
│ ├── script.py.mako # Migration template
│ └── versions/ # Migration scripts
│ └── c88468b7ab0b_initial_migration.py
├── meshview/
│ ├── models.py # SQLAlchemy models (source of truth)
│ ├── migrations.py # Migration utilities
│ ├── mqtt_database.py # Writer database connection
│ └── database.py # Reader database connection
├── startdb.py # Writer app (runs migrations)
├── main.py # Entry point for reader app
└── create_migration.py # Helper script for creating migrations
```
## Configuration
Database URL is read from `config.ini`:
```ini
[database]
connection_string = sqlite+aiosqlite:///packets.db
```
Alembic automatically uses this configuration through `meshview/migrations.py`.
## Important Notes
1. **Always test migrations** in development before deploying to production
2. **Backup database** before running migrations in production
3. **Check for data loss** - Some migrations may require data migration logic
4. **Coordinate deployments** - Start writer before readers in multi-instance setups
5. **Monitor logs** during first startup after deployment
## Example Migrations
### Example 1: Generated Initial Migration
Here's what an auto-generated migration looks like (from comparing models to database):
```python
"""Initial migration
Revision ID: c88468b7ab0b
Revises:
Create Date: 2025-01-26 20:56:50.123456
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers
revision = 'c88468b7ab0b'
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
# Upgrade operations
op.create_table('node',
sa.Column('id', sa.String(), nullable=False),
sa.Column('node_id', sa.BigInteger(), nullable=True),
# ... more columns
sa.PrimaryKeyConstraint('id')
)
def downgrade() -> None:
# Downgrade operations
op.drop_table('node')
```
### Example 2: Manual Migration Adding a New Table
We've included an example migration (`1717fa5c6545_add_example_table.py`) that demonstrates how to manually create a new table:
```python
"""Add example table
Revision ID: 1717fa5c6545
Revises: c88468b7ab0b
Create Date: 2025-10-26 20:59:04.347066
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
def upgrade() -> None:
"""Create example table with sample columns."""
op.create_table(
'example',
sa.Column('id', sa.Integer(), nullable=False, primary_key=True, autoincrement=True),
sa.Column('name', sa.String(length=100), nullable=False),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('value', sa.Float(), nullable=True),
sa.Column('is_active', sa.Boolean(), nullable=False, server_default='1'),
sa.Column('created_at', sa.DateTime(), nullable=False,
server_default=sa.text('CURRENT_TIMESTAMP')),
sa.Column('updated_at', sa.DateTime(), nullable=True),
sa.PrimaryKeyConstraint('id')
)
# Create an index on the name column for faster lookups
op.create_index('idx_example_name', 'example', ['name'])
def downgrade() -> None:
"""Remove example table."""
op.drop_index('idx_example_name', table_name='example')
op.drop_table('example')
```
**Key features demonstrated:**
- Various column types (Integer, String, Text, Float, Boolean, DateTime)
- Primary key with autoincrement
- Nullable and non-nullable columns
- Server defaults (for timestamps and booleans)
- Creating indexes
- Proper downgrade that reverses all changes
**To test this migration:**
```bash
# Apply the migration
./env/bin/alembic upgrade head
# Check it was applied
./env/bin/alembic current
# Verify table was created
sqlite3 packetsPL.db "SELECT sql FROM sqlite_master WHERE type='table' AND name='example';"
# Roll back the migration
./env/bin/alembic downgrade -1
# Verify table was removed
sqlite3 packetsPL.db "SELECT name FROM sqlite_master WHERE type='table' AND name='example';"
```
**To remove this example migration** (after testing):
```bash
# First make sure you're not on this revision
./env/bin/alembic downgrade c88468b7ab0b
# Then delete the migration file
rm alembic/versions/1717fa5c6545_add_example_table.py
```
## References
- [Alembic Documentation](https://alembic.sqlalchemy.org/)
- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/)
- [Async SQLAlchemy](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html)
docs/API_Documentation.md
+344
View File
@@ -0,0 +1,344 @@
# API Documentation
## 1. Chat API
### GET `/api/chat`
Returns the most recent chat messages.
**Query Parameters**
- `limit` (optional, int): Maximum number of messages to return. Default: `100`.
**Response Example**
```json
{
"packets": [
{
"id": 123,
"import_time": "2025-07-22T12:45:00",
"from_node_id": 987654,
"from_node": "Alice",
"channel": "main",
"payload": "Hello, world!"
}
]
}
```
---
### GET `/api/chat/updates`
Returns chat messages imported after a given timestamp.
**Query Parameters**
- `last_time` (optional, ISO timestamp): Only messages imported after this time are returned.
**Response Example**
```json
{
"packets": [
{
"id": 124,
"import_time": "2025-07-22T12:50:00",
"from_node_id": 987654,
"from_node": "Alice",
"channel": "main",
"payload": "New message!"
}
],
"latest_import_time": "2025-07-22T12:50:00"
}
```
---
## 2. Nodes API
### GET `/api/nodes`
Returns a list of all nodes, with optional filtering by last seen.
**Query Parameters**
- `hours` (optional, int): Return nodes seen in the last N hours.
- `days` (optional, int): Return nodes seen in the last N days.
- `last_seen_after` (optional, ISO timestamp): Return nodes seen after this time.
**Response Example**
```json
{
"nodes": [
{
"node_id": 1234,
"long_name": "Alice",
"short_name": "A",
"channel": "main",
"last_seen": "2025-07-22T12:40:00",
"hardware": "T-Beam",
"firmware": "1.2.3",
"role": "client",
"last_lat": 37.7749,
"last_long": -122.4194
}
]
}
```
---
## 3. Packets API
### GET `/api/packets`
Returns a list of packets with optional filters.
**Query Parameters**
- `limit` (optional, int): Maximum number of packets to return. Default: `200`.
- `since` (optional, ISO timestamp): Only packets imported after this timestamp are returned.
**Response Example**
```json
{
"packets": [
{
"id": 123,
"from_node_id": 5678,
"to_node_id": 91011,
"portnum": 1,
"import_time": "2025-07-22T12:45:00",
"payload": "Hello, Bob!"
}
]
}
```
---
---
## 4. Channels API
### GET `/api/channels`
Returns a list of channels seen in a given time period.
**Query Parameters**
- `period_type` (optional, string): Time granularity (`hour` or `day`). Default: `hour`.
- `length` (optional, int): Number of periods to look back. Default: `24`.
**Response Example**
```json
{
"channels": ["LongFast", "MediumFast", "ShortFast"]
}
```
---
## 5. Statistics API
### GET `/api/stats`
Retrieve packet statistics aggregated by time periods, with optional filtering.
---
## Query Parameters
| Parameter | Type | Required | Default | Description |
|--------------|---------|----------|----------|-------------------------------------------------------------------------------------------------|
| `period_type` | string | No | `hour` | Time granularity of the stats. Allowed values: `hour`, `day`. |
| `length` | integer | No | 24 | Number of periods to include (hours or days). |
| `channel` | string | No | — | Filter results by channel name (case-insensitive). |
| `portnum` | integer | No | — | Filter results by port number. |
| `to_node` | integer | No | — | Filter results to packets sent **to** this node ID. |
| `from_node` | integer | No | — | Filter results to packets sent **from** this node ID. |
---
## Response
```json
{
"period_type": "hour",
"length": 24,
"channel": "LongFast",
"portnum": 1,
"to_node": 12345678,
"from_node": 87654321,
"data": [
{
"period": "2025-08-08 14:00",
"count": 10
},
{
"period": "2025-08-08 15:00",
"count": 7
}
// more entries...
]
}
```
---
## 6. Edges API
### GET `/api/edges`
Returns network edges (connections between nodes) based on traceroutes and neighbor info.
**Query Parameters**
- `type` (optional, string): Filter by edge type (`traceroute` or `neighbor`). If omitted, returns both types.
**Response Example**
```json
{
"edges": [
{
"from": 12345678,
"to": 87654321,
"type": "traceroute"
},
{
"from": 11111111,
"to": 22222222,
"type": "neighbor"
}
]
}
```
---
## 7. Configuration API
### GET `/api/config`
Returns the current site configuration (safe subset exposed to clients).
**Response Example**
```json
{
"site": {
"domain": "meshview.example.com",
"language": "en",
"title": "Bay Area Mesh",
"message": "Real time data from around the bay area",
"starting": "/chat",
"nodes": "true",
"conversations": "true",
"everything": "true",
"graphs": "true",
"stats": "true",
"net": "true",
"map": "true",
"top": "true",
"map_top_left_lat": 39.0,
"map_top_left_lon": -123.0,
"map_bottom_right_lat": 36.0,
"map_bottom_right_lon": -121.0,
"map_interval": 3,
"firehose_interval": 3,
"weekly_net_message": "Weekly Mesh check-in message.",
"net_tag": "#BayMeshNet",
"version": "2.0.8 ~ 10-22-25"
},
"mqtt": {
"server": "mqtt.bayme.sh",
"topics": ["msh/US/bayarea/#"]
},
"cleanup": {
"enabled": "false",
"days_to_keep": "14",
"hour": "2",
"minute": "0",
"vacuum": "false"
}
}
```
---
## 8. Language/Translations API
### GET `/api/lang`
Returns translation strings for the UI.
**Query Parameters**
- `lang` (optional, string): Language code (e.g., `en`, `es`). Defaults to site language setting.
- `section` (optional, string): Specific section to retrieve translations for.
**Response Example (full)**
```json
{
"chat": {
"title": "Chat",
"send": "Send"
},
"map": {
"title": "Map",
"zoom_in": "Zoom In"
}
}
```
**Response Example (section-specific)**
Request: `/api/lang?section=chat`
```json
{
"title": "Chat",
"send": "Send"
}
```
---
## 9. Health Check API
### GET `/health`
Health check endpoint for monitoring, load balancers, and orchestration systems.
**Response Example (Healthy)**
```json
{
"status": "healthy",
"timestamp": "2025-11-03T14:30:00.123456Z",
"version": "3.0.0",
"git_revision": "6416978",
"database": "connected",
"database_size": "853.03 MB",
"database_size_bytes": 894468096
}
```
**Response Example (Unhealthy)**
Status Code: `503 Service Unavailable`
```json
{
"status": "unhealthy",
"timestamp": "2025-11-03T14:30:00.123456Z",
"version": "2.0.8",
"git_revision": "6416978",
"database": "disconnected"
}
```
---
## 10. Version API
### GET `/version`
Returns detailed version information including semver, release date, and git revision.
**Response Example**
```json
{
"version": "2.0.8",
"release_date": "2025-10-22",
"git_revision": "6416978a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q",
"git_revision_short": "6416978"
}
```
---
## Notes
- All timestamps (`import_time`, `last_seen`) are returned in ISO 8601 format.
- `portnum` is an integer representing the packet type.
- `payload` is always a UTF-8 decoded string.
- Node IDs are integers (e.g., `12345678`).
docs/Database-Changes-With-Alembic.md
+146
View File
@@ -0,0 +1,146 @@
# Database Changes With Alembic
This guide explains how to make database schema changes in MeshView using Alembic migrations.
## Overview
When you need to add, modify, or remove columns from database tables, you must:
1. Update the SQLAlchemy model
2. Create an Alembic migration
3. Let the system automatically apply the migration
## Step-by-Step Process
### 1. Update the Model
Edit `meshview/models.py` to add/modify the column in the appropriate model class:
```python
class Traceroute(Base):
__tablename__ = "traceroute"
id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
# ... existing columns ...
route_return: Mapped[bytes] = mapped_column(nullable=True) # New column
```
### 2. Create an Alembic Migration
Generate a new migration file with a descriptive message:
```bash
./env/bin/alembic revision -m "add route_return to traceroute"
```
This creates a new file in `alembic/versions/` with a unique revision ID.
### 3. Fill in the Migration
Edit the generated migration file to implement the actual database changes:
```python
def upgrade() -> None:
# Add route_return column to traceroute table
with op.batch_alter_table('traceroute', schema=None) as batch_op:
batch_op.add_column(sa.Column('route_return', sa.LargeBinary(), nullable=True))
def downgrade() -> None:
# Remove route_return column from traceroute table
with op.batch_alter_table('traceroute', schema=None) as batch_op:
batch_op.drop_column('route_return')
```
### 4. Migration Runs Automatically
When you restart the application with `mvrun.py`:
1. The writer process (`startdb.py`) starts up
2. It checks if the database schema is up to date
3. If new migrations are pending, it runs them automatically
4. The reader process (web server) waits for migrations to complete before starting
**No manual migration command is needed** - the application handles this automatically on startup.
### 5. Commit Both Files
Add both files to git:
```bash
git add meshview/models.py
git add alembic/versions/ac311b3782a1_add_route_return_to_traceroute.py
git commit -m "Add route_return column to traceroute table"
```
## Important Notes
### SQLite Compatibility
Always use `batch_alter_table` for SQLite compatibility:
```python
with op.batch_alter_table('table_name', schema=None) as batch_op:
batch_op.add_column(...)
```
SQLite has limited ALTER TABLE support, and `batch_alter_table` works around these limitations.
### Migration Process
- **Writer process** (`startdb.py`): Runs migrations on startup
- **Reader process** (web server in `main.py`): Waits for migrations to complete
- Migrations are checked and applied every time the application starts
- The system uses a migration status table to coordinate between processes
### Common Column Types
```python
# Integer
column: Mapped[int] = mapped_column(BigInteger, nullable=True)
# String
column: Mapped[str] = mapped_column(nullable=True)
# Bytes/Binary
column: Mapped[bytes] = mapped_column(nullable=True)
# DateTime
column: Mapped[datetime] = mapped_column(nullable=True)
# Boolean
column: Mapped[bool] = mapped_column(nullable=True)
# Float
column: Mapped[float] = mapped_column(nullable=True)
```
### Migration File Location
Migrations are stored in: `alembic/versions/`
Each migration file includes:
- Revision ID (unique identifier)
- Down revision (previous migration in chain)
- Create date
- `upgrade()` function (applies changes)
- `downgrade()` function (reverts changes)
## Troubleshooting
### Migration Not Running
If migrations don't run automatically:
1. Check that the database is writable
2. Look for errors in the startup logs
3. Verify the migration chain is correct (each migration references the previous one)
### Manual Migration (Not Recommended)
If you need to manually run migrations for debugging:
```bash
./env/bin/alembic upgrade head
```
However, the application normally handles this automatically.
docs/README.md
+14
View File
@@ -0,0 +1,14 @@
# Technical Documentation
This directory contains technical documentation for MeshView that goes beyond initial setup and basic usage.
These documents are intended for developers, contributors, and advanced users who need deeper insight into the system's architecture, database migrations, API endpoints, and internal workings.
## Contents
- [ALEMBIC_SETUP.md](ALEMBIC_SETUP.md) - Database migration setup and management
- [TIMESTAMP_MIGRATION.md](TIMESTAMP_MIGRATION.md) - Details on timestamp schema changes
- [API_Documentation.md](API_Documentation.md) - REST API endpoints and usage
- [CODE_IMPROVEMENTS.md](CODE_IMPROVEMENTS.md) - Suggested code improvements and refactoring ideas
For initial setup and basic usage instructions, please see the main [README.md](../README.md) in the root directory.
docs/TIMESTAMP_MIGRATION.md
+193
View File
@@ -0,0 +1,193 @@
# High-Resolution Timestamp Migration
This document describes the implementation of GitHub issue #55: storing high-resolution timestamps as integers in the database for improved performance and query efficiency.
## Overview
The meshview database now stores timestamps in two formats:
1. **TEXT format** (`import_time`): Human-readable ISO8601 format with microseconds (e.g., `2025-03-12 04:15:56.058038`)
2. **INTEGER format** (`import_time_us`): Microseconds since Unix epoch (1970-01-01 00:00:00 UTC)
The dual format approach provides:
- **Backward compatibility**: Existing `import_time` TEXT columns remain unchanged
- **Performance**: Fast integer comparisons and math operations
- **Precision**: Microsecond resolution for accurate timing
- **Efficiency**: Compact storage and fast indexed lookups
## Database Changes
### New Columns Added
Three tables have new `import_time_us` columns:
1. **packet.import_time_us** (INTEGER)
- Stores when the packet was imported into the database
- Indexed for fast queries
2. **packet_seen.import_time_us** (INTEGER)
- Stores when the packet_seen record was imported
- Indexed for performance
3. **traceroute.import_time_us** (INTEGER)
- Stores when the traceroute was imported
- Indexed for fast lookups
### New Indexes
The following indexes were created for optimal query performance:
```sql
CREATE INDEX idx_packet_import_time_us ON packet(import_time_us DESC);
CREATE INDEX idx_packet_from_node_time_us ON packet(from_node_id, import_time_us DESC);
CREATE INDEX idx_packet_seen_import_time_us ON packet_seen(import_time_us);
CREATE INDEX idx_traceroute_import_time_us ON traceroute(import_time_us);
```
## Migration Process
### For Existing Databases
Run the migration script to add the new columns and populate them from existing data:
```bash
python migrate_add_timestamp_us.py [database_path]
```
If no path is provided, it defaults to `packets.db` in the current directory.
The migration script:
1. Checks if migration is needed (idempotent)
2. Adds `import_time_us` columns to the three tables
3. Populates the new columns from existing `import_time` values
4. Creates indexes for optimal performance
5. Verifies the migration completed successfully
### For New Databases
New databases created with the updated schema will automatically include the `import_time_us` columns. The MQTT store module populates both columns when inserting new records.
## Code Changes
### Models (meshview/models.py)
The ORM models now include the new `import_time_us` fields:
```python
class Packet(Base):
import_time: Mapped[datetime] = mapped_column(nullable=True)
import_time_us: Mapped[int] = mapped_column(BigInteger, nullable=True)
```
### MQTT Store (meshview/mqtt_store.py)
The data ingestion logic now populates both timestamp columns using UTC time:
```python
now = datetime.datetime.now(datetime.timezone.utc)
now_us = int(now.timestamp() * 1_000_000)
# Both columns are populated
import_time=now,
import_time_us=now_us,
```
**Important**: All new timestamps use UTC (Coordinated Universal Time) for consistency across time zones.
## Using the New Timestamps
### Example Queries
**Query packets from the last 7 days:**
```sql
-- Old way (slower)
SELECT * FROM packet
WHERE import_time >= datetime('now', '-7 days');
-- New way (faster)
SELECT * FROM packet
WHERE import_time_us >= (strftime('%s', 'now', '-7 days') * 1000000);
```
**Query packets in a specific time range:**
```sql
SELECT * FROM packet
WHERE import_time_us BETWEEN 1759254380000000 AND 1759254390000000;
```
**Calculate time differences (in microseconds):**
```sql
SELECT
id,
(import_time_us - LAG(import_time_us) OVER (ORDER BY import_time_us)) / 1000000.0 as seconds_since_last
FROM packet
LIMIT 10;
```
### Converting Timestamps
**From datetime to microseconds (UTC):**
```python
import datetime
now = datetime.datetime.now(datetime.timezone.utc)
now_us = int(now.timestamp() * 1_000_000)
```
**From microseconds to datetime:**
```python
import datetime
timestamp_us = 1759254380813451
dt = datetime.datetime.fromtimestamp(timestamp_us / 1_000_000)
```
**In SQL queries:**
```sql
-- Datetime to microseconds
SELECT CAST((strftime('%s', import_time) || substr(import_time, 21, 6)) AS INTEGER);
-- Microseconds to datetime (approximate)
SELECT datetime(import_time_us / 1000000, 'unixepoch');
```
## Performance Benefits
The integer timestamp columns provide significant performance improvements:
1. **Faster comparisons**: Integer comparisons are much faster than string/datetime comparisons
2. **Smaller index size**: Integer indexes are more compact than datetime indexes
3. **Range queries**: BETWEEN operations on integers are highly optimized
4. **Math operations**: Easy to calculate time differences, averages, etc.
5. **Sorting**: Integer sorting is faster than datetime sorting
## Backward Compatibility
The original `import_time` TEXT columns remain unchanged:
- Existing code continues to work
- Human-readable timestamps still available
- Gradual migration to new columns possible
- No breaking changes for existing queries
## Future Work
Future improvements could include:
- Migrating queries to use `import_time_us` columns
- Deprecating the TEXT `import_time` columns (after transition period)
- Adding helper functions for timestamp conversion
- Creating views that expose both formats
## Testing
The migration was tested on a production database with:
- 132,466 packet records
- 1,385,659 packet_seen records
- 28,414 traceroute records
All records were successfully migrated with microsecond precision preserved.
## References
- GitHub Issue: #55 - Storing High-Resolution Timestamps in SQLite
- SQLite datetime functions: https://www.sqlite.org/lang_datefunc.html
- Python datetime module: https://docs.python.org/3/library/datetime.html