A duty cycle is a limit on a channel, not on a node, and one AirtimeManager
charging every radio at the top-level radio block's modulation is wrong twice
over on a bridge. The same packet occupies a 62.5 kHz channel about eight times
longer than a 500 kHz one, so whichever bandwidth that block names, the other
radio is mis-charged by that ratio; and the two share one allowance, so a busy
local radio spends the backhaul's airtime on a band it never transmits on.
There is now one budget per channel, built from each radio's own profile and
its own band's limit -- 868.0-868.6 MHz allows 1% where 869.4-869.65 allows
10%, and one number cannot describe both. Two radios tuned alike share a
budget, because they contend for the same spectrum. duty_cycle.shared_budget
restores a single node-wide allowance for anyone who wants the conservative
reading, and a single-radio node is metered exactly as it was.
Metering a channel means knowing which radio will carry a packet, and the
engine used to work that out by restating openhop_core's dispatch rules. It
asks now. openhop_core passes a packet's ingress radio to the TX policy, so the
choice is a function of the packet rather than of whatever arrived most
recently; the repeater asks before scheduling and gets the answer the send will
reach. That also closes a race that predates this: sticky and bridge read the
node's last RX at send time, which after a retransmit delay on a busy node is
another packet's arrival, and moves a reply onto the wrong band.
The core change is optional. The keyword in resolve_tx_radio_id's signature is
the version check, and against a core without it the tx_mode selectors are
registered in their original one-argument form and behave exactly as they did.
What changes there is the gate: with no way to know which radio will transmit,
every radio the fabric could pick is held to its own budget, so a node cannot
transmit past a legal limit by guessing the quiet channel. Which mode is in
force is logged at startup.
Also fixed, each reproduced by an adversarial review of the earlier attempt:
- A node with one radios[] entry had its modulation and limit overwritten by
the stale top-level block on every radio save from the web UI. The refresh
path now branches on whether profiles were read, not on how many radios
there are.
- build_radio_profiles is all-or-nothing by design -- it feeds the UI -- so one
radio_type: none collapsed metering onto the top-level block for the radio
still transmitting. Metering has its own builder that degrades per radio.
- Carrying the whole node's spend onto an unrecognised channel compounded over
repeated rebuilds (100 -> 400 -> 1600 ms) and wedged transmission for the
rest of the window. Carry-over is keyed by channel, which makes an unchanged
rebuild idempotent, and an unrecognised one starts from the busiest single
channel rather than the sum of them.
- A rebuild worked the previous default manager out from the config again,
which by then already named the new default -- preserving the wrong object
and orphaning the one NeighborScopeHelper holds for the life of the process.
It is remembered rather than recomputed.
- Every caller that reports one airtime figure was reporting the default
radio's channel: the MeshCore wire field total_air_time_secs, companion
stats, the RRD and SQLite history, and /stats. They read a node total again
-- totals summed across channels, utilisation the highest of them, because a
node at its limit on one channel is at a limit however quiet the other is.
tests/test_fabric_fanout.py's rig never bound a tx_mode selector, so all 49 of
those tests ran against a fabric with no TX policy at all. It binds the real
one now; that gap is why an on-air change went unnoticed through two reviews.
A bridge sends one logical packet through both radios, and the two do not fare
alike: 500 kHz and 62.5 kHz are different channels with different neighbours and
different contention. The packet row holds one set of LBT figures, from the
primary egress, so the other radio's attempts and backoffs were dropped on the
floor -- and an egress that failed was never stored at all, leaving the half of
the story an operator most wants to see only in the log.
A new packet_egress table keeps one row per physical send: packet id, timestamp,
radio, whether it went out, and its LBT attempts, total backoff and channel-busy
flag. Written only on a node with two or more radios; with one radio the packet
row already says all of this, and nothing about single-radio behaviour changes.
The rows are written on the storage writer thread beside the packet, addressed
by the id the insert returns, and are kept off packet_record itself so they do
not ride along on every websocket, Glass and MQTT publish. Retention ages them
out on the packet cutoff, and purging packets takes them along.
/lbt_diagnostics gains radios[]: the same summary and bucket shape per radio,
read from the new table over a covering index, so one radio's contention is
never read as the other's. The combined figures above it are unchanged and still
count each packet once. Per-radio entries carry no rf block -- the RRD series
behind it is node-wide, and hanging it off one radio would read as that radio's
own RSSI and loss -- and no packet-type breakdown, which would cost a heap
lookup per egress.
The fan-out now reads a send's LBT metadata whether or not it succeeded. The
single-egress path has always done so, and the figures on a failed send are
exactly the ones that explain it -- as far as they reach us: a radio that raises
instead of returning metadata (an SX1262 giving up on a busy channel) reports
nothing to read, so that row records the failure with no attempts rather than
inventing them.
Redundancy copies are recorded too. A multi-ack burst is real airtime on both
radios, and leaving it out would undercount exactly the traffic multi-ack adds
to a channel already busy enough to need it.
/db_stats lists packet_egress only once something has been written to it, so a
single-radio node reports the tables it always reported and never pays for
counting rows it does not have.
The bucket and summary maths moved out of get_lbt_diagnostics into three module
functions so the per-radio series is the same code as the combined one rather
than a second copy of 150 lines. The combined answer is unchanged.
A bridge has one receiver per band, and until now one of them stood for the
node: the background timer read the default radio and stored the answer with no
radio attached. On the 910.100/910.525 pair that reported the wide radio's noise
floor as the whole node's, and charged its CRC counter with both radios' errors.
Every physical radio is now sampled, one after another, and stored against its
own id. Migration 19 adds a nullable radio_id to noise_floor and crc_errors with
a covering index each, so a per-radio history reads only its own rows. Rows
written before this stay NULL: they were never measured per radio, and
labelling them would claim an attribution nobody made.
CRC baselines are per radio. A shared one charges every radio's errors to
whichever was sampled last, which reports swings that never happened.
Reads take an optional radio_id: /noise_floor_history, /noise_floor_stats,
/crc_error_count and /crc_error_history. Naming a radio also makes offset paging
stable, which the RF Health page depends on -- paging the merged series shifts a
page's contents every time the other radio samples. History rows name their
radio only on a node with two or more radios, so a single-radio response is
byte-for-byte what it was.
The default radio is sampled first and is the only one published. /stats, the
MQTT status message and companion stats read its cached value, and publishing
per radio doubles the observer-feed cadence -- its own change, not this one.
The StorageCollector facade carries the new arguments too. The web endpoints
reach the database through it, so the tests that drive them autospec it: a bare
mock accepts any keyword and would have let a facade that had not kept up
answer happily in tests and raise TypeError on a real node.
The self-echo guard registered a packet's hash and forwarded path when the
engine decided to forward it and kept it for 30 s. Two things could then
hide a real neighbour from the link tracker:
- A neighbour whose 1-byte path hash collides with ours, repeating the
same packet from the same upstream path, carries exactly the path we
would have sent.
- The registration happened before the duty-cycle gate, so a packet that
was never transmitted still suppressed matching receptions.
Registration now happens at the physical send, inside the TX lock after
the duty-cycle gate, and records which radio is sending. A send that
fails is forgotten. A reception counts as our echo only when it arrives on
a radio other than the one that sent it, and while that send is on the
air or within 2 s of it finishing, which covers a remote modem's delivery
delay. Single-radio nodes keep no records at all, since a lone radio
cannot hear itself.
A colliding neighbour is now mistaken for our echo only if it repeats the
same packet on the same path, on the other radio, inside that window.
The engine now passes each reception's radio to the neighbour link
tracker, from both the forwarding path and the duplicate-recording path,
so /neighbor_links can report how each radio hears a neighbour.
Two radios sharing one frequency hear each other transmit. The echo of a
relayed flood ends in this node's own path hash, so the tracker recorded
the node as its own neighbour, at near-zero-dB signal. The engine now
remembers the packet hash and exact path of each transmission for 30 s,
registered before the first send because an echo can arrive while a
later egress of the same fan-out is still on the air, and skips tracker
observation for a reception that matches. A neighbour repeating our copy
appends its own hash and still counts. The echo is still stored and
published as the reception it is; only the neighbour table ignores it.
Matching the exact path rather than the last hop's hash keeps a real
neighbour whose 1-byte hash collides with ours.
openhop-dev/openhop_core#123 settled on fabric.origin_tx: default | all
as the name for sending locally originated packets out on every radio,
and openhop-dev/openhop_core#124 with openhop-dev/openhop_repeater#412
implement that knob in core. Use the same key here so a Fabric config
means one thing whichever implementation lands.
local_tx_mode is still read, so a config written against the first
version of this branch keeps its behaviour. If both keys are set to
different modes, startup rejects the config before any hardware is
opened, since there is no way to tell which one the operator meant.
While running, the engine reads the option through the same helper and
lets origin_tx win. Validation errors name the key the operator actually
wrote.
The radio stack meta and the startup log report origin_tx.
Add two Fabric options for an exactly-two-radio node:
- fabric.repeat_on_ingress (tx_mode: bridge): a relayed packet is sent
on the other radio first, then back out on the radio that heard it.
- fabric.local_tx_mode: all: locally originated packets (companions,
adverts, room/protocol replies) go out once on every radio, default
radio first.
MeshCore processing (policy, dedupe, path mutation) still runs once per
logical packet; only the physical TX is duplicated, each egress with its
own Packet, serialised through the TX lock and gated independently for
duty cycle and link health. One successful egress is a successful
forward. Multi-ack extras share the egress set.
inject_packet registers its single ACK waiter before the first TX so an
ACK heard mid-fan-out outlives the dispatcher's ACK cache. Fan-out fixes
flood scope and path-hash mode once before copying and metering, so all
radios carry identical bytes and the duty-cycle gates see them.
Packet records gain tx_radio_ids (JSON column + migration); tx_radio_id
remains the primary successful egress. Invalid option combinations are
rejected before hardware is opened. Defaults preserve existing behaviour.
After mesh.default_region was wired to the core dispatcher's send-time
scope resolver, a relayed plain flood was indistinguishable from a
locally originated one, so Dispatcher._apply_flood_scope() converted it
to the repeater's default transport scope and sent it out-of-region.
Successful flood forwards now mark their existing scope decision as
authoritative before entering Dispatcher.send_packet(): relayed plain
floods stay plain and relayed transport floods retain the originator's
codes. Locally originated floods and REPLY_SCOPE_DEFAULT fallback
replies still pick up the configured default scope.
Marking only on the flood_forward() success path leaves packets rejected
by validation, policy, duplicate, or loop detection untouched, and
leaves locally injected transmissions (which bypass forwarding) alone.
A local transmit retried after a flat 1s backoff. When the failure was the
link itself -- a KISS modem that dropped off USB and is re-enumerating, which
takes about 2s to come back and re-handshake -- that retry landed inside the
outage and failed immediately, turning one modem hiccup into a failed message
even though the companion client was still waiting.
A transient failure with the link up keeps its 1s backoff outside _tx_lock. A
dropped link is now waited for instead, polled at 100ms and bounded by
delays.local_tx_link_wait_seconds: default 5s, non-finite values rejected and
everything capped at 30s, because the client's own command timeout is already
part-spent on the TX delay by the time we get here. Relayed packets never
wait -- a repeat held back that long is worse than one dropped.
Link state is read from the radio the send will actually use. Attribute
access on a FabricRadio passes through to its default endpoint, so a send
carrying preferred_tx_radio_id would otherwise be judged by the wrong radio,
suppressing a healthy one or waving through a dead one. A multi-radio fabric
with no explicit target picks its endpoint at send time, so nothing is
inspected and the send proceeds as before. Radios that track no link state
(a directly driven SX1262) answer nothing and keep their exact timing.
- Added tests for multi-radio stack functionality, including merging radio entries and building radio stacks.
- Implemented policy context tests to ensure correct handling of radio IDs in multi-radio scenarios.
Added a new method to the AirtimeManager class to refresh modulation parameters without clearing transmission history. Updated ConfigManager to call this method after applying live radio configurations. Enhanced tests to verify that modulation updates occur correctly during live updates.
An ACK at an intermediate direct hop is now regenerated as a plain
DIRECT ACK (transport codes dropped, version bits cleared) and relayed
with zero retransmit delay, matching routeDirectRecvAcks, instead of
taking the generic delayed direct forward. The repeater.multi_acks
preference (already exposed by the mesh CLI) is now consumed by the
engine: when enabled, both ACK relay sites precede the plain ACK with
a MULTIPART-wrapped redundancy copy spaced by the direct retransmit
delay + 300 ms. The multipart relay seen-key now keeps the MULTIPART
payload type over the unwrapped payload, matching firmware hasSeen so
a redundancy pair survives each hop.
Replace the eight hand-rolled companion-bridge delivery loops in the
packet router with a single _fan_out_to_bridges helper that reads
HandlerResult.authenticated directly, so a broken handler contract
surfaces instead of being hidden by getattr hedges.
Introduce a DropReason str-Enum in the engine for the seventeen fixed
drop reasons (with __str__ returning the value, since str() of a
str-Enum changed on Python 3.11+) and derive the router's
expected-drop check from it, retiring the string-prefix tuple.
Detailed variants keep their suffixes embedded in the reason string.
Split the companion-delivery dedupe cache into a pure check and an
explicit mark so PATH and protocol responses record delivery only
after the fan-out runs — a copy where every bridge raised is retried
on the next copy instead of being suppressed for the full TTL.
Keep both migration 12 (companion message signal/channel data) and
migration 13 (packet upstream hash for neighbour links), and retain
ACK/MULTIPART plus TRACE imports in engine tests.
The TX delay factors changed from absolute seconds to firmware-matching
airtime multipliers (random delay in [0, 5 * airtime * factor], per
MeshCore getRetransmitDelay), but deployed configs tuned under the old
seconds interpretation get no signal on upgrade. Log the interpretation
and both factors at startup, and fix the stale wording in
config.yaml.example and the web API docstring. Defaults are unchanged
(flood 1.0, direct 0.5; firmware ships 0.5/0.3).
The delays.rx_delay_base config value was stored and reported but never
applied. Now that the core dispatcher implements MeshCore's flood
reception-quality delay, feed it through:
- set dispatcher.rx_delay_base from delays.rx_delay_base at startup and
re-apply it on live config updates of the delays section (web API
saves already pass that section)
- point the mesh CLI get/set rxdelay at the delays section; it read and
wrote repeater.rx_delay_base, which nothing consumes, so the CLI knob
was silently dead
- delegate RepeaterHandler.calculate_packet_score to the shared core
packet_score (same firmware packetScoreInt formula, one impl)
Suppression works through the existing engine seen-cache because it
dedupes at process time, after the dispatcher hold. Default remains 0
(delay disabled), matching firmware.
Match MeshCore's forwardMultipartDirect: a multipart packet is never
relayed as an ordinary directed packet. At an intermediate direct hop a
multipart ACK is re-emitted as a plain DIRECT ACK with the wrapper byte
stripped and this node removed from the path, deduped on the regenerated
ACK form and spaced by (remaining + 1) * 300ms. Flood routing, final
hops, and non-ACK embedded types are dropped rather than forwarded.
Coverage includes literal firmware-format wire vectors for one- and
two-byte hash widths and transport-direct input, end-to-end receive-to-
transmit relaying, and the drop cases.
Extend the authenticated-ownership model to the PATH and RESPONSE
routing branches so a packet is consumed (do-not-retransmit) only when
a local identity MAC-verifies it. Prefix-only collisions and forged
traffic stay eligible for the forwarding engine.
- PathHelper marks do-not-retransmit and reports authenticated only
after a successful MAC decrypt with a valid path envelope; invalid or
truncated envelopes remain forwardable.
- PacketRouter aggregates authenticated results across the path helper
and companion bridges for PATH and RESPONSE, skipping the engine only
on authenticated ownership while preserving empty-path DIRECT release
hygiene.
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
- Updated the __call__ method to return a boolean indicating transmission success.
- Adjusted handling of transmission failures to increment dropped_count appropriately.
- Enhanced logging for transmission failures in PacketRouter to ensure visibility of issues.
- Added tests to verify correct behavior during transmission exceptions and failures.
Every MQTT-published packet has shipped with duration="0" since the
PacketRecord factory was introduced. The repeater already computes LoRa
time-on-air via AirtimeManager.calculate_airtime() (the canonical
Semtech reference formula) for duty-cycle gating and TX delay, but the
result was thrown away after each packet - never stored on the
packet_record dict that flows to MQTT/SQLite/Glass/websocket.
What changes
- engine.py: RepeaterHandler._build_packet_record() now computes
airtime_ms once per packet (Semtech formula via AirtimeManager) and
stores it as packet_record['airtime_ms']. Single source of truth for
every downstream consumer.
- storage_utils.py: PacketRecord.from_packet_record() reads the new
airtime_ms field and serializes it as a rounded integer in the
'duration' field of the published JSON. Falls back to 0 if the field
is missing (backward compatibility for any older code path).
- storage_collector.py: _publish_packet_to_mqtt() simplified - no
recomputation, no helper. The publish path is now a passthrough.
Why
MQTT consumers (firmware-compatible analyzers, dashboards, the upstream
meshcoretomqtt project) expect the same time-on-air value the firmware
emits. Hard-coded "0" makes airtime/utilization charts derived from the
mqtt stream useless and silently diverges from firmware behavior.
Plumbing the value through packet_record (instead of recomputing in the
publish path) means any future consumer - SQLite schema, web UI charts,
Glass telemetry - reads the same number without separate calculations.
Tests
tests/test_packet_duration.py - 5 tests covering:
- backward compat (legacy packet_record without airtime_ms => '0')
- airtime_ms field flows through to duration as rounded integer string
- explicit zero stays '0'
- AirtimeManager output matches an independently-implemented Semtech
reference for typical MeshCore EU settings (SF8/62.5kHz/CR4-8)
- low-data-rate optimization branch (SF12/125kHz triggers DE=1)
Co-Authored-By: Oz <oz-agent@warp.dev>
Reviewer concern (PR 190):
The 1-second backoff sleep for local_transmission retry happened inside
`async with self._tx_lock`, blocking all other queued TX tasks for the
full second — hurting latency and throughput under load.
Fix — tighten lock scope to one attempt per acquisition:
Before: acquire lock → [attempt 0 → sleep(1) → attempt 1] → release
After: for each attempt:
[sleep(1) if retry] ← OUTSIDE the lock
acquire lock
re-check can_transmit ← fresh check every acquisition
attempt single send
record_tx on success
release lock
The duty-cycle gate now runs on every lock acquisition (not just the first),
which is correct: airtime state may change during the backoff sleep.
Tests added (tests/test_tx_lock.py):
1. test_concurrent_sends_do_not_interleave — two tasks racing to the same
delay timer must never overlap inside send_packet.
2. test_duty_cycle_toctou_is_fixed — second packet is dropped when the
first consumes the budget inside the lock.
3. test_local_retry_releases_lock_during_backoff — a concurrent relayed
packet fires at ~0.1s while local retry sleeps 1s; confirms it is not
blocked by the backoff.
4. test_non_local_failure_propagates — relayed send failure raises
immediately with exactly one attempt.
5. test_duty_cycle_rechecked_on_retry — if the budget is exhausted during
backoff, the retry is dropped by the in-lock gate (not sent).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merged this now. It’s a safe change with no behavioural impact, and it removes unnecessary work in the hot paths when DEBUG logging is off. Happy to revisit if we want to standardise on lazy formatting later, but this gives us an immediate win.
Python evaluates f-string arguments before calling logger.debug(), so in
production (INFO level) every debug log call in the hot path still paid the
cost of string formatting even though the output was discarded.
The most expensive sites are in __call__ (runs on every received packet):
- "RX packet: header=0x{...}, payload_len=..., path_len=..., rssi=..., snr=..."
- "Packet header=0x{...}, type=..., route=..."
And in _calculate_tx_delay (runs on every forwarded packet):
- "Route=FLOOD/DIRECT, len=...B, airtime=...ms, delay=...s"
- "Congestion detected, score=..., delay multiplier=..."
Plus transport code and local-TX debug logs (less frequent but same issue).
Fix: wrap each f-string logger.debug() call with
if logger.isEnabledFor(logging.DEBUG):
so the f-string is never constructed when debug logging is disabled.
logger.isEnabledFor() is a pure in-memory integer comparison — essentially
free at runtime. In production at INFO level this eliminates string
concatenation, attribute lookups (packet.header, len(packet.payload), etc.),
and format operations on every forwarded packet.
Eight call sites guarded; no logic changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before this change, calculate_packet_hash() (SHA-256 + hex + upper) was called
3 times per forwarded packet and 4 times per dropped packet:
__call__ → pkt_hash_full = packet.calculate_packet_hash() #1
→ flood/direct_forward → is_duplicate → calculate_packet_hash() #2
→ flood/direct_forward → mark_seen → calculate_packet_hash() #3
(drop) → _get_drop_reason → is_duplicate → calculate_packet_hash() #4
pkt_hash_full was computed in __call__ but never threaded down into
process_packet, flood_forward, direct_forward, is_duplicate, or _get_drop_reason.
Each method recomputed it independently.
Fix: add optional packet_hash: Optional[str] = None to is_duplicate,
_get_drop_reason, flood_forward, direct_forward, and process_packet. Pass
pkt_hash_full from __call__ through the chain. Each method uses the provided
hash or falls back to computing it — preserving backward compatibility for
external callers (TraceHelper, etc.) that have no pre-computed hash.
Result: 1 SHA-256 computation per packet in the hot path regardless of whether
the packet is forwarded or dropped.
Also adds explicit INVARIANT docstrings to flood_forward, direct_forward, and
is_duplicate documenting that these methods must remain synchronous (no await).
The is_duplicate + mark_seen pair is atomic within the asyncio event loop; adding
an await between them would allow two concurrent tasks to both pass the duplicate
check for the same packet — forwarding it twice.
Docs: docs/pr_hash_once.md — problem analysis, call-chain diagram, per-method
diffs, quantification (~3-8 µs saved per packet), test plan (including hash-count
assertion), and proof that passing the original's hash to the deep-copied packet
is correct.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add self._tx_lock (asyncio.Lock) to RepeaterHandler and acquire it inside
delayed_send after the per-packet sleep completes.
Problem 1 — radio interleave: concurrent delayed_send coroutines (one per
queued packet) could both exit their sleep at nearly the same moment and call
dispatcher.send_packet simultaneously, interleaving SPI/serial register writes
to the half-duplex LoRa radio.
Problem 2 — TOCTOU gap: the upfront can_transmit() check in __call__ and the
record_tx() call in delayed_send are separated by the entire TX delay (up to
several seconds). Under burst conditions two tasks both pass the check before
either has recorded its airtime, causing both to transmit and the duty-cycle
budget to be exceeded.
Fix: acquire _tx_lock after the sleep so delay timers still run concurrently
(matching firmware behaviour), then immediately re-check can_transmit() inside
the lock before sending. Because only one task holds the lock at a time,
airtime state is stable; check and record_tx() are effectively atomic — no
TOCTOU window. Airtime is recorded only on a successful send, so a radio
failure never inflates the budget.
Also move `import random` from inside _calculate_tx_delay to module level
(stdlib imports belong at the top; the lazy-import pattern is unnecessary here).
Docs: docs/pr_tx_serialization.md — problem statement, root-cause analysis,
alternative approaches considered, invariant table, full unit + field test plan,
and proof of correctness for the asyncio.Lock approach.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
agessaman