A duty cycle is a limit on a channel, not on a node, and one AirtimeManager
charging every radio at the top-level radio block's modulation is wrong twice
over on a bridge. The same packet occupies a 62.5 kHz channel about eight times
longer than a 500 kHz one, so whichever bandwidth that block names, the other
radio is mis-charged by that ratio; and the two share one allowance, so a busy
local radio spends the backhaul's airtime on a band it never transmits on.
There is now one budget per channel, built from each radio's own profile and
its own band's limit -- 868.0-868.6 MHz allows 1% where 869.4-869.65 allows
10%, and one number cannot describe both. Two radios tuned alike share a
budget, because they contend for the same spectrum. duty_cycle.shared_budget
restores a single node-wide allowance for anyone who wants the conservative
reading, and a single-radio node is metered exactly as it was.
Metering a channel means knowing which radio will carry a packet, and the
engine used to work that out by restating openhop_core's dispatch rules. It
asks now. openhop_core passes a packet's ingress radio to the TX policy, so the
choice is a function of the packet rather than of whatever arrived most
recently; the repeater asks before scheduling and gets the answer the send will
reach. That also closes a race that predates this: sticky and bridge read the
node's last RX at send time, which after a retransmit delay on a busy node is
another packet's arrival, and moves a reply onto the wrong band.
The core change is optional. The keyword in resolve_tx_radio_id's signature is
the version check, and against a core without it the tx_mode selectors are
registered in their original one-argument form and behave exactly as they did.
What changes there is the gate: with no way to know which radio will transmit,
every radio the fabric could pick is held to its own budget, so a node cannot
transmit past a legal limit by guessing the quiet channel. Which mode is in
force is logged at startup.
Also fixed, each reproduced by an adversarial review of the earlier attempt:
- A node with one radios[] entry had its modulation and limit overwritten by
the stale top-level block on every radio save from the web UI. The refresh
path now branches on whether profiles were read, not on how many radios
there are.
- build_radio_profiles is all-or-nothing by design -- it feeds the UI -- so one
radio_type: none collapsed metering onto the top-level block for the radio
still transmitting. Metering has its own builder that degrades per radio.
- Carrying the whole node's spend onto an unrecognised channel compounded over
repeated rebuilds (100 -> 400 -> 1600 ms) and wedged transmission for the
rest of the window. Carry-over is keyed by channel, which makes an unchanged
rebuild idempotent, and an unrecognised one starts from the busiest single
channel rather than the sum of them.
- A rebuild worked the previous default manager out from the config again,
which by then already named the new default -- preserving the wrong object
and orphaning the one NeighborScopeHelper holds for the life of the process.
It is remembered rather than recomputed.
- Every caller that reports one airtime figure was reporting the default
radio's channel: the MeshCore wire field total_air_time_secs, companion
stats, the RRD and SQLite history, and /stats. They read a node total again
-- totals summed across channels, utilisation the highest of them, because a
node at its limit on one channel is at a limit however quiet the other is.
tests/test_fabric_fanout.py's rig never bound a tx_mode selector, so all 49 of
those tests ran against a fabric with no TX policy at all. It binds the real
one now; that gap is why an on-air change went unnoticed through two reviews.
The self-echo guard registered a packet's hash and forwarded path when the
engine decided to forward it and kept it for 30 s. Two things could then
hide a real neighbour from the link tracker:
- A neighbour whose 1-byte path hash collides with ours, repeating the
same packet from the same upstream path, carries exactly the path we
would have sent.
- The registration happened before the duty-cycle gate, so a packet that
was never transmitted still suppressed matching receptions.
Registration now happens at the physical send, inside the TX lock after
the duty-cycle gate, and records which radio is sending. A send that
fails is forgotten. A reception counts as our echo only when it arrives on
a radio other than the one that sent it, and while that send is on the
air or within 2 s of it finishing, which covers a remote modem's delivery
delay. Single-radio nodes keep no records at all, since a lone radio
cannot hear itself.
A colliding neighbour is now mistaken for our echo only if it repeats the
same packet on the same path, on the other radio, inside that window.
Add two Fabric options for an exactly-two-radio node:
- fabric.repeat_on_ingress (tx_mode: bridge): a relayed packet is sent
on the other radio first, then back out on the radio that heard it.
- fabric.local_tx_mode: all: locally originated packets (companions,
adverts, room/protocol replies) go out once on every radio, default
radio first.
MeshCore processing (policy, dedupe, path mutation) still runs once per
logical packet; only the physical TX is duplicated, each egress with its
own Packet, serialised through the TX lock and gated independently for
duty cycle and link health. One successful egress is a successful
forward. Multi-ack extras share the egress set.
inject_packet registers its single ACK waiter before the first TX so an
ACK heard mid-fan-out outlives the dispatcher's ACK cache. Fan-out fixes
flood scope and path-hash mode once before copying and metering, so all
radios carry identical bytes and the duty-cycle gates see them.
Packet records gain tx_radio_ids (JSON column + migration); tx_radio_id
remains the primary successful egress. Invalid option combinations are
rejected before hardware is opened. Defaults preserve existing behaviour.
A local transmit retried after a flat 1s backoff. When the failure was the
link itself -- a KISS modem that dropped off USB and is re-enumerating, which
takes about 2s to come back and re-handshake -- that retry landed inside the
outage and failed immediately, turning one modem hiccup into a failed message
even though the companion client was still waiting.
A transient failure with the link up keeps its 1s backoff outside _tx_lock. A
dropped link is now waited for instead, polled at 100ms and bounded by
delays.local_tx_link_wait_seconds: default 5s, non-finite values rejected and
everything capped at 30s, because the client's own command timeout is already
part-spent on the TX delay by the time we get here. Relayed packets never
wait -- a repeat held back that long is worse than one dropped.
Link state is read from the radio the send will actually use. Attribute
access on a FabricRadio passes through to its default endpoint, so a send
carrying preferred_tx_radio_id would otherwise be judged by the wrong radio,
suppressing a healthy one or waving through a dead one. A multi-radio fabric
with no explicit target picks its endpoint at send time, so nothing is
inspected and the send proceeds as before. Radios that track no link state
(a directly driven SX1262) answer nothing and keep their exact timing.
- Added tests for multi-radio stack functionality, including merging radio entries and building radio stacks.
- Implemented policy context tests to ensure correct handling of radio IDs in multi-radio scenarios.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
- Updated the __call__ method to return a boolean indicating transmission success.
- Adjusted handling of transmission failures to increment dropped_count appropriately.
- Enhanced logging for transmission failures in PacketRouter to ensure visibility of issues.
- Added tests to verify correct behavior during transmission exceptions and failures.
Reviewer concern (PR 190):
The 1-second backoff sleep for local_transmission retry happened inside
`async with self._tx_lock`, blocking all other queued TX tasks for the
full second — hurting latency and throughput under load.
Fix — tighten lock scope to one attempt per acquisition:
Before: acquire lock → [attempt 0 → sleep(1) → attempt 1] → release
After: for each attempt:
[sleep(1) if retry] ← OUTSIDE the lock
acquire lock
re-check can_transmit ← fresh check every acquisition
attempt single send
record_tx on success
release lock
The duty-cycle gate now runs on every lock acquisition (not just the first),
which is correct: airtime state may change during the backoff sleep.
Tests added (tests/test_tx_lock.py):
1. test_concurrent_sends_do_not_interleave — two tasks racing to the same
delay timer must never overlap inside send_packet.
2. test_duty_cycle_toctou_is_fixed — second packet is dropped when the
first consumes the budget inside the lock.
3. test_local_retry_releases_lock_during_backoff — a concurrent relayed
packet fires at ~0.1s while local retry sleeps 1s; confirms it is not
blocked by the backoff.
4. test_non_local_failure_propagates — relayed send failure raises
immediately with exactly one attempt.
5. test_duty_cycle_rechecked_on_retry — if the budget is exhausted during
backoff, the retry is dropped by the in-lock gate (not sent).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
agessaman