SIGTERM did not stop the daemon. Observed: the process stayed alive 18+
minutes with all three companion listen sockets still bound and the serial
port still held, so a restart could not reopen the radio. Two defects
compounded.
Cleanup never ran. The signal handler cancelled run(), whose finally then
awaited _shutdown() from inside an already-cancelled task -- so the first
await raised CancelledError, run() returned, and asyncio.run() tore the loop
down before a single shutdown step executed. Not one step logged. Running
cleanup in a sibling task instead does not fix it either: run() returns as
soon as the dispatcher stops and asyncio.run() cancels every leftover task
on the way out, which showed up as 'frame server :5050' being cancelled
mid-stop. The handler now unwinds run() cooperatively by stopping the
dispatcher, so run_forever() returns on its own and cleanup runs in a task
that is not being cancelled. Cancelling is kept only as a fallback for a
failure before the dispatcher exists.
Nothing bounded the steps. Frame servers, bridges, router and Glass had no
timeout, so one stuck step stranded every step after it -- including
releasing the radio. Each step is now bounded by SHUTDOWN_STEP_TIMEOUT_S and
logged by name, so a hang is both survivable and diagnosable, and the sync
steps (HTTP stop, sensor manager, GPS, radio cleanup) run off-loop so a
blocking close cannot stall the sequence. The dispatcher is stopped first so
RX ends before its radio is released.
Even with cleanup fixed, a single non-daemon thread that never returns hangs
SIGTERM forever: interpreter finalization joins them with no timeout, which
is where the original 18-minute hang sat (main thread parked in
Py_FinalizeEx -> wait_for_thread_shutdown). Report any that are still alive
by name so the offender can be fixed at the source, excluding asyncio's own
executor workers since asyncio.run() joins those under its own timeout. Then
arm a daemon watchdog that forces the process down SHUTDOWN_EXIT_GRACE_S
after cleanup finishes, so a future stray thread costs a delayed exit rather
than a stuck service.
Verified on hardware: SIGTERM and SIGINT both exit in 1s with zero warnings,
all ports and the serial device released, and MQTT publishing its offline
status before disconnecting.
Startup config mistakes were handled inconsistently: an identity collision
exited cleanly, but a missing or invalid config file (FileNotFoundError /
RuntimeError from load_config) and a missing identity key (RuntimeError) dumped
a full traceback -- and load_config ran outside main()'s try, so those errors
escaped the fatal handler entirely.
Add repeater/exceptions.py with ConfigurationError(RuntimeError) and re-parent
IdentityConfigurationError onto it (it stays a RuntimeError, so existing
except-sites are unaffected). Raise ConfigurationError for the missing/invalid
config file, the config-load failure, and the missing identity key. Move
load_config and daemon construction inside main()'s try and catch
ConfigurationError there, logging just the message and exiting 1; unexpected
failures still log with a traceback.
A configured-identity collision (IdentityConfigurationError from the startup
preflight) is an actionable config problem, not a crash, but the top-level
fatal handler logged every exception with exc_info=True, burying the message
under a full traceback. Catch IdentityConfigurationError in main() and log just
the message before exiting 1; unexpected errors still get the traceback.
Local identities occupy two routing/persistence namespaces: companions
(companion_bridges[hash] plus the companion_* tables keyed by the hash byte)
and server-side identities (the repeater and every room server, which share
the login/text/protocol helper handlers[hash] slot and the room_* tables). A
one-byte prefix collision is only unrepresentable when both identities share a
namespace; a companion and a server-side identity live in physically separate
stores, and the packet router (_consume_via_local_candidates) already offers a
colliding packet to both and lets HMAC pick the owner.
Key IdentityManager state by (hash_byte, namespace) instead of the bare hash so
the guard rejects only same-namespace collisions. This keeps blocking the pairs
that actually break -- companion<->companion (bridge overwrite plus
companion_prefs PRIMARY KEY corruption) and server<->server, i.e.
repeater<->room-server and room-server<->room-server (handlers[hash] overwrite
plus room_* corruption) -- while allowing a companion to share a prefix with the
repeater or a room server, which was previously rejected despite being only a
cosmetic label clash.
Build a core RegionMap from the node's served regions and wire it into
the dispatcher and every companion bridge, so a flood reply is re-scoped
to the region its request arrived under (or left plain for a wildcard /
direct request) -- matching firmware simple_repeater::sendFloodReply.
Previously replies went out plain, so a reply to a request in region B
was dropped by B-only repeaters.
The map is built once from the node-wide transport_keys table (each named
region -> RegionEntry, flags=REGION_DENY_FLOOD for deny-flood regions; the
'*' wildcard is deliberately not an entry so plain floods reply plain). A
single shared instance reaches the dispatcher and all bridges. Public
regions rely on name-hashing for their key; a stored key is carried only
when it is genuinely custom material the name would not reproduce, keeping
reply-matching aligned with the forwarding transport-code check.
Region edits at runtime (CLI, web API, Glass sync) all funnel through the
transport_keys CRUD methods, which now fire a post-commit change callback;
the daemon rebuilds the map and reassigns a fresh instance to the
dispatcher and every live bridge (atomic rebind, safe against an in-flight
find_match on the RX thread).
Requires openhop_core with Dispatcher.region_map / CompanionBridge.region_map.
The PATH and protocol-response paths recorded the companion dedupe
unconditionally after the bridge fan-out, so a delivery where every
bridge raised was still suppressed for the full dedupe TTL — the client
lost the packet even though later mesh copies arrived. Have the fan-out
report delivery (at least one bridge completed without raising) next to
authentication, and mark the dedupe only on delivery, which is the
behaviour the marking helper's contract already documented. One healthy
bridge still counts as delivered, so duplicate suppression of repeated
mesh copies is unchanged.
_consume_via_local_candidates awaited the targeted companion bridge with
no exception guard, so a raising bridge aborted the whole candidate loop:
the hash-colliding room-server or repeater identity registered at the
same one-byte dest hash was never offered the packet, and an unclaimed
packet never fell through to the forwarding engine. Wrap the bridge call
with the same log-and-continue handling the fan-out path already uses.
Firmware only relays a TRACE at an intermediate hop when
allowPacketForward passes, so disabling forwarding stops trace relay.
The trace helper forwarded via packet injection, which is gated only by
the local-TX check — a repeater in monitor mode kept repeating traces
while reporting repeat off. Consult the repeater mode before relaying;
locally originated pings are injected directly and keep working in
monitor mode, and ping-response matching still runs before the relay
decision.
Firmware rejects rxdelay outside 0-20 and txdelay/direct.txdelay outside
0-2.0; the CLI only rejected negatives, so a remote admin could set
delay factors far beyond what any firmware node would accept. Apply the
firmware ranges with the firmware error strings and state the ranges in
the help text. Existing configs with out-of-range values are untouched —
only new CLI sets are gated. Also adds an end-to-end regression test
running every set command against the real ConfigManager on a temp
config file.
set flood.advert.interval wrote flood_advert_interval_hours, a key
nothing reads, and get reported it with a default of 24 — the engine's
flood-advert timer (and the web API) consume send_advert_interval_hours
with a default of 10. Read and write the consumed key so the remote
command actually reschedules the timer; the orphan key is left in place
and ignored.
set freq and set radio stored the CLI's MHz/kHz inputs directly into
radio.frequency and radio.bandwidth, which the rest of the stack treats
as Hz — a freq change tuned the radio to a few hundred hertz. Convert to
Hz on write and validate set radio with the firmware gate (freq 150-2500,
bw 7-500, sf 5-12, cr 5-8, same error string; set freq stays unvalidated
like firmware). Radio changes are now saved without a live apply so the
reply's restart-to-apply contract is real: a live retune would cut off
the remote admin mid-session, and the all-or-nothing live radio path
would also have dragged staged frequency changes along with a tx tweak.
The password, guest.password, and allow.read.only commands wrote a
top-level security section with a stale key name, while LoginHelper
authenticates from repeater.security.admin_password/.guest_password/
.allow_read_only — so remote password changes never took effect. Point
the set and get commands at the real subtree, and push repeater.security
onto the live repeater ACL during a repeater-section live update: the
ACL captures its passwords at registration, so without the refresh a
saved change would still only apply after a restart. Room-server ACLs
keep their per-identity settings passwords.
ConfigManager.save_to_file returns a bare bool, but every mesh CLI set
command (and the password command) still tuple-unpacked it, so each one
wrote the YAML and then raised, skipping the live update and returning an
error to the remote admin. Route every save through one helper that
checks the bool, reports save failures honestly, and only live-applies
after a successful write. The test fixture mocked the stale tuple shape,
which is why the suite stayed green while the CLI was broken on the real
manager.
Updated the _persist_companion_message method to accept a queue_entry parameter for more precise removal of messages from the bridge queue. Introduced a new _remove_queue_entry method to ensure messages are removed by identity, preventing potential message loss during concurrent operations. Adjusted related tests to reflect these changes and verify correct behavior.
Added a new method to the AirtimeManager class to refresh modulation parameters without clearing transmission history. Updated ConfigManager to call this method after applying live radio configurations. Enhanced tests to verify that modulation updates occur correctly during live updates.
Introduced tests to verify that the repeater's bridge does not enable client-repeat forwarding in Core's Dispatcher. The tests confirm that the repeater maintains the expected inert behavior regarding client-repeat capabilities and preferences.
The pushed frame serialized the delivery wall-clock time instead of the
post's own stored timestamp, and left the flag byte's low 2 attempt bits
always zero. Firmware sends the stored post timestamp and ORs in a fresh
random attempt so a retried post produces a different packet hash and
ACK; with a fixed timestamp two retries within the same second would be
byte-identical and dropped as mesh duplicates. The serialized timestamp
now also equals the push_post_timestamp sync watermark, matching the
firmware invariant.
GlobalRateLimiter.acquire() enforced the inter-message gap inside an
async-with that released the lock before returning, so the caller's
radio transmission ran with no lock held and every room's sync loop
could push concurrently. acquire() now returns with the lock held
(released via release() in a finally, including on cancellation during
the gap wait), uses a monotonic clock so wall-clock jumps cannot skip
or stretch the gap, and push_post_to_client acquires it only around the
sync-state write plus the blocking send, so the ACK-timeout clock
starts when the transmission actually begins. Regression tests cover
two rooms sharing one limiter (no concurrent sends, gap enforced) and
lock release when the send raises.
Firmware's Dispatcher::tryParsePacket drops any payload version above 0
at parse, and core's Packet.read_from now enforces the same rule, so a
version-1 routed ACK can never reach the relay; the wire vector
exercising that unreachable input is removed (header regeneration is
still covered by the remaining vectors, including transport-direct).
The advert fixture also OR'd the version-1 bit into its header; it now
builds the version-0 header that PacketBuilder.create_advert emits.
The room post limit was 160 Python characters; firmware caps post text
at MAX_POST_TEXT_LEN = 160-9 = 151 bytes (the 160-byte encrypted text
budget minus the timestamp/flags/author-prefix header). Store posts
truncated to 151 UTF-8 bytes at a codepoint boundary, and clamp again
when pushing so previously stored oversized posts cannot produce
oversized frames.
An ACK at an intermediate direct hop is now regenerated as a plain
DIRECT ACK (transport codes dropped, version bits cleared) and relayed
with zero retransmit delay, matching routeDirectRecvAcks, instead of
taking the generic delayed direct forward. The repeater.multi_acks
preference (already exposed by the mesh CLI) is now consumed by the
engine: when enabled, both ACK relay sites precede the plain ACK with
a MULTIPART-wrapped redundancy copy spaced by the direct retransmit
delay + 300 ms. The multipart relay seen-key now keeps the MULTIPART
payload type over the unwrapped payload, matching firmware hasSeen so
a redundancy pair survives each hop.
The daemon registers its own default identity so companion and
room-server hash collisions against the repeater are caught, which
adds a repeater:repeater entry to the endpoint's raw registered list
and counts it in total_registered. The web UI only renders the
per-entry registered boolean on configured room servers and
companions (matched by prefixed name), so it is unaffected; pin the
payload shape for external API consumers, and pin that the bare name
"repeater" is reserved — a room server configured with that name is
rejected by the collision rules and reported unregistered.
Keep both migration 12 (companion message signal/channel data) and
migration 13 (packet upstream hash for neighbour links), and retain
ACK/MULTIPART plus TRACE imports in engine tests.
The daemon's max-TX-power lookup duplicated the core fallback chain and
ended in a radio_type string match to know the SX1262's 22 dBm limit --
a driver constant that every new backend name would silently lose. The
SX1262 driver class now declares max_tx_power_dbm itself (the CH341
variant instantiates the same class, only the transport differs), so
the lookup collapses to the shared resolver over the radio object and
the validated deployment settings.
The TX delay factors changed from absolute seconds to firmware-matching
airtime multipliers (random delay in [0, 5 * airtime * factor], per
MeshCore getRetransmitDelay), but deployed configs tuned under the old
seconds interpretation get no signal on upgrade. Log the interpretation
and both factors at startup, and fix the stale wording in
config.yaml.example and the web API docstring. Defaults are unchanged
(flood 1.0, direct 0.5; firmware ships 0.5/0.3).
The mesh CLI's get/set txdelay and direct.txdelay read and wrote
repeater.tx_delay_factor / repeater.direct_tx_delay_factor, but the
engine and web API use the delays section, so the CLI knobs were
silently dead: set appeared to succeed while changing nothing, and get
echoed the unused value back. Same defect and same fix as the recent
rxdelay repoint.
The companion queue's capacity path ran a Python loop of single-row
SELECT+DELETE pairs ordered by created_at. Wall-clock ordering meant a
backwards clock step (NTP correction) could make the just-inserted
channel row sort as oldest and wrongly reject the incoming message while
older channel rows remained evictable.
Replace the loop with one set-based DELETE ordered by id (AUTOINCREMENT,
i.e. insertion order, immune to clock steps), with an evictable-count
pre-check preserving the all-or-nothing rejection rule: never displace a
direct message, never evict the incoming row to make room for itself,
roll back the insert entirely when channel rows cannot make room. The
queue load and pop queries move to id ordering for the same reason.
The delays.rx_delay_base config value was stored and reported but never
applied. Now that the core dispatcher implements MeshCore's flood
reception-quality delay, feed it through:
- set dispatcher.rx_delay_base from delays.rx_delay_base at startup and
re-apply it on live config updates of the delays section (web API
saves already pass that section)
- point the mesh CLI get/set rxdelay at the delays section; it read and
wrote repeater.rx_delay_base, which nothing consumes, so the CLI knob
was silently dead
- delegate RepeaterHandler.calculate_packet_score to the shared core
packet_score (same firmware packetScoreInt formula, one impl)
Suppression works through the existing engine seen-cache because it
dedupes at process time, after the dispatcher hold. Default remains 0
(delay disabled), matching firmware.
AirtimeManager.calculate_airtime now uses openhop_core's
calculate_lora_airtime_ms, which matches RadioLib's getTimeOnAir. This
adopts the radio driver's symbol-time auto-LDRO rule in place of the
SF/BW shorthand, which diverged from hardware at 16.384 ms symbol
settings such as SF12 @ 250 kHz and SF10 @ 62.5 kHz. Test oracles
follow the same rule, with literal RadioLib-derived regression values
for the divergent settings.
Match MeshCore's forwardMultipartDirect: a multipart packet is never
relayed as an ordinary directed packet. At an intermediate direct hop a
multipart ACK is re-emitted as a plain DIRECT ACK with the wrapper byte
stripped and this node removed from the path, deduped on the regenerated
ACK form and spaced by (remaining + 1) * 300ms. Flood routing, final
hops, and non-ACK embedded types are dropped rather than forwarded.
Coverage includes literal firmware-format wire vectors for one- and
two-byte hash widths and transport-direct input, end-to-end receive-to-
transmit relaying, and the drop cases.
Use openhop_core's new single-argument event registrations for the
frame server's SQLite persistence callbacks instead of the deprecated
positional on_*_received forms.
Rightup