adam/pyMC_Repeater
1
0 Fork 0
mirror of https://github.com/pyMC-dev/pyMC_Repeater.git synced 2026-10-30 19:51:59 +01:00
Code Issues 41 Packages Projects Releases Wiki Activity

OTA updates do not take effect at runtime due to PYTHONPATH shadowing in service file #54

New Issue
Closed
opened 2026-10-27 09:30:28 +01:00 by adam · 3 comments
adam commented 2026-10-27 09:30:28 +01:00
Owner
Copy link
Copy source

Summary

After performing an OTA update via the web dashboard (/api/update/install), the pip package is correctly updated but the running service continues to execute the old code. The version displayed on LetsMesh, /api/stats, and the frontend can all disagree depending on which source they read.

Root Cause

The systemd service file (/etc/systemd/system/pymc-repeater.service) deployed by manage.sh on older branches contains:

WorkingDirectory=/opt/pymc_repeater
Environment="PYTHONPATH=/opt/pymc_repeater"

When the service restarts after an OTA update:

  1. pip install --force-reinstall correctly installs the new version into site-packages
  2. systemctl restart pymc-repeater starts a new process
  3. But PYTHONPATH=/opt/pymc_repeater causes Python to find repeater/ in the old source checkout before the pip-installed package in site-packages
  4. The process loads old code, old _version.py, and broadcasts the old version to LetsMesh

The _get_installed_version() function in update_endpoints.py reads from pip dist-info (correct), while repeater.__version__ reads from the shadowed _version.py (stale). This creates a split where the update system thinks the upgrade succeeded, but the runtime is unchanged.

How We Discovered It

  1. Performed OTA update from the pymc_console frontend — reported success
  2. Hard-refreshed the browser — sidebar showed v1.0.8.dev22 (from update API's dist-info scan)
  3. LetsMesh observer still reported 1.0.8.dev9+g664f061bb (from runtime __version__)
  4. /api/stats confirmed runtime version was still dev9 with git hash local segment
  5. /api/update/status confirmed dist-info version was dev22
  6. Inspected service file on device — still had old PYTHONPATH and WorkingDirectory

The feat/companion Fix (Partial)

The feat/companion branch already addresses this in two ways:

  1. Fixed service file — WorkingDirectory=/var/lib/pymc_repeater, no PYTHONPATH
  2. pymc-do-upgrade wrapper — contains migration logic that strips PYTHONPATH and fixes WorkingDirectory via sed before each OTA install

However, there are gaps:

Gap 1: manage.sh service file copy can fail silently

In the install/upgrade functions, the service file copy uses || true:

cp "$SCRIPT_DIR/pymc-repeater.service" /etc/systemd/system/ 2>/dev/null || true

If this fails (permissions, path issues), the old service file remains and nothing reports an error.

Gap 2: The migration only runs via the non-root pymc-do-upgrade path

If the service runs as root (e.g. some Docker/Proxmox setups), _do_install() takes the direct pip install path which has no service file migration:

if is_root:
    cmd = ["python3", "-m", "pip", "install", ...]

Gap 3: No post-install verification

After restart_service(), there is no check that the new process is actually running the expected version. The finish_install() call happens before the restart, so it always reports success even if the runtime stays on old code.

Suggested Improvements

  1. Remove || true from service file copy in manage.sh, or at minimum log a warning if the copy fails
  2. Add service file migration to the root code path in _do_install() (not just the wrapper)
  3. Post-restart version verification — after restart_service(), poll /api/stats or re-read __version__ to confirm the new version is actually running
  4. Consider removing /opt/pymc_repeater/repeater/ during OTA — if the source checkout is only needed for manage.sh, the repeater/ subdirectory should be cleaned up to prevent any possible shadowing

Workaround

For existing deployments with the old service file, run on the device:

sudo sed -i '/^Environment=.*PYTHONPATH/d' /etc/systemd/system/pymc-repeater.service
sudo sed -i 's|WorkingDirectory=/opt/pymc_repeater|WorkingDirectory=/var/lib/pymc_repeater|' /etc/systemd/system/pymc-repeater.service
sudo systemctl daemon-reload
sudo systemctl restart pymc-repeater

Environment

  • Device: Raspberry Pi (aarch64), Debian/Bookworm
  • Branch: feat/companion
  • pyMC_Repeater version: 1.0.8.dev22
  • Python: 3.13
  • Installation method: manage.sh install followed by OTA via web dashboard
## Summary After performing an OTA update via the web dashboard (`/api/update/install`), the pip package is correctly updated but the running service continues to execute the **old code**. The version displayed on LetsMesh, `/api/stats`, and the frontend can all disagree depending on which source they read. ## Root Cause The systemd service file (`/etc/systemd/system/pymc-repeater.service`) deployed by `manage.sh` on older branches contains: ```ini WorkingDirectory=/opt/pymc_repeater Environment="PYTHONPATH=/opt/pymc_repeater" ``` When the service restarts after an OTA update: 1. `pip install --force-reinstall` correctly installs the new version into site-packages 2. `systemctl restart pymc-repeater` starts a new process 3. But `PYTHONPATH=/opt/pymc_repeater` causes Python to find `repeater/` in the old source checkout **before** the pip-installed package in site-packages 4. The process loads old code, old `_version.py`, and broadcasts the old version to LetsMesh The `_get_installed_version()` function in `update_endpoints.py` reads from pip dist-info (correct), while `repeater.__version__` reads from the shadowed `_version.py` (stale). This creates a split where the update system thinks the upgrade succeeded, but the runtime is unchanged. ## How We Discovered It 1. Performed OTA update from the pymc_console frontend — reported success 2. Hard-refreshed the browser — sidebar showed `v1.0.8.dev22` (from update API's dist-info scan) 3. LetsMesh observer still reported `1.0.8.dev9+g664f061bb` (from runtime `__version__`) 4. `/api/stats` confirmed runtime version was still `dev9` with git hash local segment 5. `/api/update/status` confirmed dist-info version was `dev22` 6. Inspected service file on device — still had old `PYTHONPATH` and `WorkingDirectory` ## The `feat/companion` Fix (Partial) The `feat/companion` branch already addresses this in two ways: 1. **Fixed service file** — `WorkingDirectory=/var/lib/pymc_repeater`, no `PYTHONPATH` 2. **`pymc-do-upgrade` wrapper** — contains migration logic that strips `PYTHONPATH` and fixes `WorkingDirectory` via `sed` before each OTA install However, there are gaps: ### Gap 1: `manage.sh` service file copy can fail silently In the install/upgrade functions, the service file copy uses `|| true`: ```bash cp "$SCRIPT_DIR/pymc-repeater.service" /etc/systemd/system/ 2>/dev/null || true ``` If this fails (permissions, path issues), the old service file remains and nothing reports an error. ### Gap 2: The migration only runs via the non-root `pymc-do-upgrade` path If the service runs as root (e.g. some Docker/Proxmox setups), `_do_install()` takes the direct pip install path which has **no** service file migration: ```python if is_root: cmd = ["python3", "-m", "pip", "install", ...] ``` ### Gap 3: No post-install verification After `restart_service()`, there is no check that the new process is actually running the expected version. The `finish_install()` call happens before the restart, so it always reports success even if the runtime stays on old code. ## Suggested Improvements 1. **Remove `|| true` from service file copy** in `manage.sh`, or at minimum log a warning if the copy fails 2. **Add service file migration to the root code path** in `_do_install()` (not just the wrapper) 3. **Post-restart version verification** — after `restart_service()`, poll `/api/stats` or re-read `__version__` to confirm the new version is actually running 4. **Consider removing `/opt/pymc_repeater/repeater/`** during OTA — if the source checkout is only needed for `manage.sh`, the `repeater/` subdirectory should be cleaned up to prevent any possible shadowing ## Workaround For existing deployments with the old service file, run on the device: ```bash sudo sed -i '/^Environment=.*PYTHONPATH/d' /etc/systemd/system/pymc-repeater.service sudo sed -i 's|WorkingDirectory=/opt/pymc_repeater|WorkingDirectory=/var/lib/pymc_repeater|' /etc/systemd/system/pymc-repeater.service sudo systemctl daemon-reload sudo systemctl restart pymc-repeater ``` ## Environment - Device: Raspberry Pi (aarch64), Debian/Bookworm - Branch: `feat/companion` - pyMC_Repeater version: 1.0.8.dev22 - Python: 3.13 - Installation method: `manage.sh install` followed by OTA via web dashboard <!-- github-sync-id:4055418186 -->
adam closed this issue 2026-10-27 09:30:28 +01:00
adam commented 2026-10-27 09:30:29 +01:00
Author
Owner
Copy link
Copy source

The remove was intentional, now the pymc_repeater runs from the dist location saving that sync process to /opt/

Have you installed the latest companion branch sudo manage.py upgrade

I will take a look at the rest of what the robot is saying. 😃

The remove was intentional, now the pymc_repeater runs from the dist location saving that sync process to /opt/ Have you installed the latest companion branch sudo manage.py upgrade I will take a look at the rest of what the robot is saying. 😃 <!-- github-comment-id:4037207149 -->
adam commented 2026-10-27 09:30:30 +01:00
Author
Owner
Copy link
Copy source

yep! all of this was done after a fresh pull / upgrade on the pyMC_Repeater side. The workaround cured it.

Could have been a local environment issue, but wanted to flag in case it was worth sniffing around.

yep! all of this was done after a fresh pull / upgrade on the pyMC_Repeater side. The workaround cured it. Could have been a local environment issue, but wanted to flag in case it was worth sniffing around. <!-- github-comment-id:4037455245 -->
adam commented 2026-10-27 09:30:30 +01:00
Author
Owner
Copy link
Copy source

Have committed some changes to tighten this up, d701342951

Have committed some changes to tighten this up, d7013429510b3676e13d0665c14c3e9cdab69325 <!-- github-comment-id:4038156268 -->
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adam/pyMC_Repeater#54