A duty cycle is a limit on a channel, not on a node, and one AirtimeManager
charging every radio at the top-level radio block's modulation is wrong twice
over on a bridge. The same packet occupies a 62.5 kHz channel about eight times
longer than a 500 kHz one, so whichever bandwidth that block names, the other
radio is mis-charged by that ratio; and the two share one allowance, so a busy
local radio spends the backhaul's airtime on a band it never transmits on.
There is now one budget per channel, built from each radio's own profile and
its own band's limit -- 868.0-868.6 MHz allows 1% where 869.4-869.65 allows
10%, and one number cannot describe both. Two radios tuned alike share a
budget, because they contend for the same spectrum. duty_cycle.shared_budget
restores a single node-wide allowance for anyone who wants the conservative
reading, and a single-radio node is metered exactly as it was.
Metering a channel means knowing which radio will carry a packet, and the
engine used to work that out by restating openhop_core's dispatch rules. It
asks now. openhop_core passes a packet's ingress radio to the TX policy, so the
choice is a function of the packet rather than of whatever arrived most
recently; the repeater asks before scheduling and gets the answer the send will
reach. That also closes a race that predates this: sticky and bridge read the
node's last RX at send time, which after a retransmit delay on a busy node is
another packet's arrival, and moves a reply onto the wrong band.
The core change is optional. The keyword in resolve_tx_radio_id's signature is
the version check, and against a core without it the tx_mode selectors are
registered in their original one-argument form and behave exactly as they did.
What changes there is the gate: with no way to know which radio will transmit,
every radio the fabric could pick is held to its own budget, so a node cannot
transmit past a legal limit by guessing the quiet channel. Which mode is in
force is logged at startup.
Also fixed, each reproduced by an adversarial review of the earlier attempt:
- A node with one radios[] entry had its modulation and limit overwritten by
the stale top-level block on every radio save from the web UI. The refresh
path now branches on whether profiles were read, not on how many radios
there are.
- build_radio_profiles is all-or-nothing by design -- it feeds the UI -- so one
radio_type: none collapsed metering onto the top-level block for the radio
still transmitting. Metering has its own builder that degrades per radio.
- Carrying the whole node's spend onto an unrecognised channel compounded over
repeated rebuilds (100 -> 400 -> 1600 ms) and wedged transmission for the
rest of the window. Carry-over is keyed by channel, which makes an unchanged
rebuild idempotent, and an unrecognised one starts from the busiest single
channel rather than the sum of them.
- A rebuild worked the previous default manager out from the config again,
which by then already named the new default -- preserving the wrong object
and orphaning the one NeighborScopeHelper holds for the life of the process.
It is remembered rather than recomputed.
- Every caller that reports one airtime figure was reporting the default
radio's channel: the MeshCore wire field total_air_time_secs, companion
stats, the RRD and SQLite history, and /stats. They read a node total again
-- totals summed across channels, utilisation the highest of them, because a
node at its limit on one channel is at a limit however quiet the other is.
tests/test_fabric_fanout.py's rig never bound a tx_mode selector, so all 49 of
those tests ran against a fabric with no TX policy at all. It binds the real
one now; that gap is why an on-air change went unnoticed through two reviews.
A bridge sends one logical packet through both radios, and the two do not fare
alike: 500 kHz and 62.5 kHz are different channels with different neighbours and
different contention. The packet row holds one set of LBT figures, from the
primary egress, so the other radio's attempts and backoffs were dropped on the
floor -- and an egress that failed was never stored at all, leaving the half of
the story an operator most wants to see only in the log.
A new packet_egress table keeps one row per physical send: packet id, timestamp,
radio, whether it went out, and its LBT attempts, total backoff and channel-busy
flag. Written only on a node with two or more radios; with one radio the packet
row already says all of this, and nothing about single-radio behaviour changes.
The rows are written on the storage writer thread beside the packet, addressed
by the id the insert returns, and are kept off packet_record itself so they do
not ride along on every websocket, Glass and MQTT publish. Retention ages them
out on the packet cutoff, and purging packets takes them along.
/lbt_diagnostics gains radios[]: the same summary and bucket shape per radio,
read from the new table over a covering index, so one radio's contention is
never read as the other's. The combined figures above it are unchanged and still
count each packet once. Per-radio entries carry no rf block -- the RRD series
behind it is node-wide, and hanging it off one radio would read as that radio's
own RSSI and loss -- and no packet-type breakdown, which would cost a heap
lookup per egress.
The fan-out now reads a send's LBT metadata whether or not it succeeded. The
single-egress path has always done so, and the figures on a failed send are
exactly the ones that explain it -- as far as they reach us: a radio that raises
instead of returning metadata (an SX1262 giving up on a busy channel) reports
nothing to read, so that row records the failure with no attempts rather than
inventing them.
Redundancy copies are recorded too. A multi-ack burst is real airtime on both
radios, and leaving it out would undercount exactly the traffic multi-ack adds
to a channel already busy enough to need it.
/db_stats lists packet_egress only once something has been written to it, so a
single-radio node reports the tables it always reported and never pays for
counting rows it does not have.
The bucket and summary maths moved out of get_lbt_diagnostics into three module
functions so the per-radio series is the same code as the combined one rather
than a second copy of 150 lines. The combined answer is unchanged.
A bridge has one receiver per band, and until now one of them stood for the
node: the background timer read the default radio and stored the answer with no
radio attached. On the 910.100/910.525 pair that reported the wide radio's noise
floor as the whole node's, and charged its CRC counter with both radios' errors.
Every physical radio is now sampled, one after another, and stored against its
own id. Migration 19 adds a nullable radio_id to noise_floor and crc_errors with
a covering index each, so a per-radio history reads only its own rows. Rows
written before this stay NULL: they were never measured per radio, and
labelling them would claim an attribution nobody made.
CRC baselines are per radio. A shared one charges every radio's errors to
whichever was sampled last, which reports swings that never happened.
Reads take an optional radio_id: /noise_floor_history, /noise_floor_stats,
/crc_error_count and /crc_error_history. Naming a radio also makes offset paging
stable, which the RF Health page depends on -- paging the merged series shifts a
page's contents every time the other radio samples. History rows name their
radio only on a node with two or more radios, so a single-radio response is
byte-for-byte what it was.
The default radio is sampled first and is the only one published. /stats, the
MQTT status message and companion stats read its cached value, and publishing
per radio doubles the observer-feed cadence -- its own change, not this one.
The StorageCollector facade carries the new arguments too. The web endpoints
reach the database through it, so the tests that drive them autospec it: a bare
mock accepts any keyword and would have let a facade that had not kept up
answer happily in tests and raise TypeError on a real node.
Adding the radio filter and per-radio split by concatenating SQL fragments tripped bandit B608, failing the pre-commit check. The fragments were constants and every value was bound, but the queries are now three fixed strings: a NULL radio parameter disables the rx_radio_id filter, and the per-radio split selects its own query. Results are unchanged.
Picks up the per-radio analytics fixes from openHop_RepeaterUI feat/per-radio-airtime: settled and generation-checked loads, whole-minute chart buckets, per-radio duplicate series, and the radio selection kept in ?radio=.
The self-echo guard registered a packet's hash and forwarded path when the
engine decided to forward it and kept it for 30 s. Two things could then
hide a real neighbour from the link tracker:
- A neighbour whose 1-byte path hash collides with ours, repeating the
same packet from the same upstream path, carries exactly the path we
would have sent.
- The registration happened before the duty-cycle gate, so a packet that
was never transmitted still suppressed matching receptions.
Registration now happens at the physical send, inside the TX lock after
the duty-cycle gate, and records which radio is sending. A send that
fails is forgotten. A reception counts as our echo only when it arrives on
a radio other than the one that sent it, and while that send is on the
air or within 2 s of it finishing, which covers a remote modem's delivery
delay. Single-radio nodes keep no records at all, since a lone radio
cannot hear itself.
A colliding neighbour is now mistaken for our echo only if it repeats the
same packet on the same path, on the other radio, inside that window.
The engine now passes each reception's radio to the neighbour link
tracker, from both the forwarding path and the duplicate-recording path,
so /neighbor_links can report how each radio hears a neighbour.
Two radios sharing one frequency hear each other transmit. The echo of a
relayed flood ends in this node's own path hash, so the tracker recorded
the node as its own neighbour, at near-zero-dB signal. The engine now
remembers the packet hash and exact path of each transmission for 30 s,
registered before the first send because an echo can arrive while a
later egress of the same fan-out is still on the air, and skips tracker
observation for a reception that matches. A neighbour repeating our copy
appends its own hash and still counts. The echo is still stored and
published as the reception it is; only the neighbour table ignores it.
Matching the exact path rather than the last hop's hash keeps a real
neighbour whose 1-byte hash collides with ours.
On a Fabric node the analytics endpoints merged every radio together, with
one transmission per logical packet and nothing to tell a reception on one
radio from the other. Every packet row already records its ingress radio
and each radio that sent it, so the split needs no new data.
The attribution rules get_airtime_buckets introduced move into shared
helpers: packet_carriers() says which stored radio ids a row occupied, and
RadioResolver maps them onto the configured radios (one radio owns every
row; with two, an unknown id is unattributed, never guessed).
get_airtime_buckets now uses them with no change in output.
With two or more radio profiles:
- /packet_stats adds radios[] (received, duplicates, dropped, physical
transmissions, average RSSI/SNR) and unattributed counts. The legacy
totals come from the same grouped pass, so the 24 h aggregate the
WebSocket broadcasts does not gain a second scan. Totals still count
one transmission per packet.
- /route_stats adds the route mix of each radio's receptions, plus
originated for packets no radio received.
- /radio_packet_rates (new) returns receptions and transmissions per
radio per bucket, grouped in SQL on columns idx_packets_airtime covers.
A week-long window is an index-only scan instead of a Python loop over
every row: 0.17 s against 1.11 s for the airtime loop on a synthetic
1.55M-row table.
- /neighbor_links gives each link radios[], its stats as each radio heard
it. The tracker keeps them beside the merged figures without changing
its keys or entry count.
- /neighbor_link_history rows carry rx_radio_id, and the endpoint takes a
radio_id filter and a by_radio bucket split.
A single-radio node gets exactly the answers it had: the legacy queries
still run for it and every new field is omitted.
get_airtime_buckets selects rx_radio_id, tx_radio_id and tx_radio_ids to
charge airtime per radio, but idx_packets_airtime only covers timestamp,
length, payload_length and transmitted. SQLite therefore fell back to
idx_packets_timestamp with a heap lookup for every row in the window:
the access pattern the covering index was added to remove, and the one
that pushes the chart past the client timeout on an SD card. The test
meant to guard this asserted the plain timestamp index.
Migration 18 drops idx_packets_airtime and rebuilds it with the three
radio id columns. CREATE INDEX IF NOT EXISTS never reshapes an existing
index, so the base schema cannot do this. It keeps creating the original
four-column shape, which is safe on a database that predates the radio
id columns, and the migration widens it once migrations 16 and 17 have
added them. The rebuild reads the packets table once, at startup.
The bucket query is now a module constant shared with the test, which
asserts COVERING INDEX idx_packets_airtime for both airtime queries, so
a column added to the query cannot slip out of the index unnoticed.
openhop-dev/openhop_core#123 settled on fabric.origin_tx: default | all
as the name for sending locally originated packets out on every radio,
and openhop-dev/openhop_core#124 with openhop-dev/openhop_repeater#412
implement that knob in core. Use the same key here so a Fabric config
means one thing whichever implementation lands.
local_tx_mode is still read, so a config written against the first
version of this branch keeps its behaviour. If both keys are set to
different modes, startup rejects the config before any hardware is
opened, since there is no way to tell which one the operator meant.
While running, the engine reads the option through the same helper and
lets origin_tx win. Validation errors name the key the operator actually
wrote.
The radio stack meta and the startup log report origin_tx.
A two-radio Fabric bridge runs each side at its own SF, so one combined
utilization line computed from a single caller-supplied profile describes
neither radio. Report airtime per radio instead.
build_radio_stack now records the effective air settings of every radio it
builds, in configured order, as radio_stack_meta["radio_profiles"]. The
same builder is exposed as build_radio_profiles() so /stats and the airtime
endpoint can read current settings from the live config rather than a boot
snapshot. Air settings are read the way get_radio_for_board reads them, and
a field that cannot be read is reported as null rather than defaulted, so
nothing is charted from settings that were never on the air.
get_airtime_buckets takes those profiles and charges each stored packet to
the radios that actually carried it: its ingress rx_radio_id and every
successful egress in tx_radio_ids, falling back to the scalar tx_radio_id
for pre-fan-out rows. A relayed packet therefore appears as RX on the radio
that heard it and TX on each radio that sent it, each at that radio's own
time-on-air. Duplicate receptions are included; they occupied the receiver.
With one profile every packet belongs to that radio, so single-radio and
pre-Fabric databases are attributed exactly as before. With two or more, a
packet whose radio cannot be identified is counted as unattributed rather
than guessed onto the default radio.
The response is additive: legacy buckets/rx_total/tx_total remain as the
combined series, and radios[] carries one series per radio alongside
unattributed_rx_count/unattributed_tx_count. Airtime comes from the shared
core estimator instead of a second inline Semtech formula, memoised per
(radio, length). The sf/bw_hz/cr/preamble query parameters are kept only as
a fallback; server profiles are authoritative and are never applied across
both radios of a bridge.
Add two Fabric options for an exactly-two-radio node:
- fabric.repeat_on_ingress (tx_mode: bridge): a relayed packet is sent
on the other radio first, then back out on the radio that heard it.
- fabric.local_tx_mode: all: locally originated packets (companions,
adverts, room/protocol replies) go out once on every radio, default
radio first.
MeshCore processing (policy, dedupe, path mutation) still runs once per
logical packet; only the physical TX is duplicated, each egress with its
own Packet, serialised through the TX lock and gated independently for
duty cycle and link health. One successful egress is a successful
forward. Multi-ack extras share the egress set.
inject_packet registers its single ACK waiter before the first TX so an
ACK heard mid-fan-out outlives the dispatcher's ACK cache. Fan-out fixes
flood scope and path-hash mode once before copying and metering, so all
radios carry identical bytes and the duty-cycle gates see them.
Packet records gain tx_radio_ids (JSON column + migration); tx_radio_id
remains the primary successful egress. Invalid option combinations are
rejected before hardware is opened. Defaults preserve existing behaviour.
The companion frame proxy only accepted an operator JWT in `?token=`,
while `/ws/packets` also accepts a long-lived API token (X-API-Key
header, or the same query parameter). Apply the identical rule to the
frame proxy so a client that already holds a repeater API key can open
the companion link on the HTTP port without keeping a JWT session alive.
No behaviour change for JWT clients. Same fallback order, close codes
and log lines as PacketWebSocket.
After mesh.default_region was wired to the core dispatcher's send-time
scope resolver, a relayed plain flood was indistinguishable from a
locally originated one, so Dispatcher._apply_flood_scope() converted it
to the repeater's default transport scope and sent it out-of-region.
Successful flood forwards now mark their existing scope decision as
authoritative before entering Dispatcher.send_packet(): relayed plain
floods stay plain and relayed transport floods retain the originator's
codes. Locally originated floods and REPLY_SCOPE_DEFAULT fallback
replies still pick up the configured default scope.
Marking only on the flood_forward() success path leaves packets rejected
by validation, policy, duplicate, or loop detection untouched, and
leaves locally injected transmissions (which bypass forwarding) alone.
The SSE endpoints registered their bridge callbacks once behind a
_callbacks_registered latch. Anything that cleared the bridge's push
callbacks therefore killed the stream permanently — the latch stayed True
while the subscriptions were gone, so no later stream open repaired them.
Cache the callback objects instead and re-register them each time a stream
opens. Bridge registration is idempotent, so the repeat costs nothing while
a subscription that went missing is repaired rather than leaving the stream
silently dead until the daemon restarts.
The clear that made this reachable is fixed in openhop_core, where the
frame server now retracts only its own subscriptions instead of clearing
every callback on each companion client connection. This half is the
belt-and-braces: clear_push_callbacks() remains a legitimate full reset,
and the stream should survive one.
Firmware's ``simple_repeater`` resolves a ``default_scope`` TransportKey at boot
from ``region_map.getDefaultRegion()`` and answers the ``REPLY_SCOPE_DEFAULT``
row with ``sendFloodScoped(default_scope, ...)``. The comment on that branch is
the whole point of it::
// un-scoped would be dropped at hop 0 by repeaters running flood.max.unscoped=0
We had the config for it -- ``mesh.default_region``, which the web API already
auto-creates as an allow-flood region -- but nothing carried it to
``Dispatcher.default_flood_transport_key``, so it only ever reached adverts and
the neighbours publisher. Core's send-time resolver therefore found no default
and left the reply a plain flood.
The row is reached whenever the request's scope is *unknowable* rather than
un-scoped: a DIRECT request we hold no return path for, or a TRANSPORT_FLOOD
whose code matched no served region. Both are ordinary traffic, and on a mesh
running ``flood.max.unscoped = 0`` both replies died at hop 0 while firmware
would have delivered them.
``resolve_default_scope_key`` resolves through the built RegionMap rather than
hashing the configured name, for parity with ``getDefaultRegion()``, which can
only return a region the map holds: a ``$private`` default gets its stored key
material, and a default naming an unserved region resolves to nothing rather
than to a scope no local Region would match on the way back in. An unset default
and the ``*`` wildcard resolve to None, which is firmware's
``default_scope.isNull()`` row -- a plain flood.
Re-resolved from the three places that change the answer: boot, the
transport_keys change hook, and a runtime ``mesh.default_region`` edit. That last
one matters because the web API creates the region *before* writing the config,
so the storage hook has already run against the previous value.
Companion bridges need no separate wiring: a deferred reply from a bridge's
login server reaches the shared dispatcher unmarked, so it resolves here, which
is what firmware ``simple_room_server`` does with its own ``default_scope``.
The two end-to-end tests skip on a core that still resolves REPLY_SCOPE_DEFAULT
at RX and marks the reply final, since the assignment is correct but inert
there. They are gated on the behaviour rather than on a version, so they start
running by themselves once that core lands.
A local transmit retried after a flat 1s backoff. When the failure was the
link itself -- a KISS modem that dropped off USB and is re-enumerating, which
takes about 2s to come back and re-handshake -- that retry landed inside the
outage and failed immediately, turning one modem hiccup into a failed message
even though the companion client was still waiting.
A transient failure with the link up keeps its 1s backoff outside _tx_lock. A
dropped link is now waited for instead, polled at 100ms and bounded by
delays.local_tx_link_wait_seconds: default 5s, non-finite values rejected and
everything capped at 30s, because the client's own command timeout is already
part-spent on the TX delay by the time we get here. Relayed packets never
wait -- a repeat held back that long is worse than one dropped.
Link state is read from the radio the send will actually use. Attribute
access on a FabricRadio passes through to its default endpoint, so a send
carrying preferred_tx_radio_id would otherwise be judged by the wrong radio,
suppressing a healthy one or waving through a dead one. A multi-radio fabric
with no explicit target picks its endpoint at send time, so nothing is
inspected and the send proceeds as before. Radios that track no link state
(a directly driven SX1262) answer nothing and keep their exact timing.
pushPostToClient sizes the body with strlen(post.text) for the payload and
the expected ACK alike. The web API will store "a\0b", and pushing it whole
sends text the receiver cannot see -- it stops at the NUL -- while hashing
the pending ACK over a span the receiver never reproduces. The push would
then never be acknowledged and would retry into the failure backoff.
Same defect the core builder had; this is a second, hand-rolled builder.
Also proves MC-R4 end to end at last: a genuinely encrypted guest PLAIN
through a real registered room identity, watching the injector for an ACK
that must not appear. The seam was only covered in three separate halves
before -- core honours a veto, _may_ack computes one, a mocked constructor
receives one -- none of which would have caught the wiring being dropped.
The core text handler answers by BaseChatMesh's rule -- plain and signed
text, never a CLI type -- and it runs before this helper's gates. So a
signed post, a non-admin command or a guest's post was acknowledged and
only then thrown away: the sender saw delivery confirmed for a message
that went nowhere.
_may_ack is firmware's rule for a server, handed to the core handler as
should_ack_fn at register_identity. simple_repeater builds an ACK only
under `if (flags == TXT_TYPE_PLAIN)`, inside a branch already gated on
isAdmin(); simple_room_server sets send_ack only on the non-guest PLAIN
path. Neither answers a CLI type, an unsupported type or a replayed
timestamp -- but both still answer a retry, because firmware suppresses
the work, not the reply. It reads no state it does not also leave alone.
It predicts eligibility, not the outcome: a post that clears the role and
replay checks and is then refused by openHop's own rate limit or a
database error is still ACKed. Those refusals have no firmware
counterpart, so closing that gap needs the store result fed back here.
Guarded by a signature check: passing should_ack_fn to a core that does
not take it raises TypeError out of register_identity, which would leave
the node with no text handler at all. pyproject.toml tracks
openhop_core@dev with no version floor, so that is an ordinary state --
and in it the ACK goes back to being sent, availability over the
invariant.
Two things firmware does that openHop did not.
Dispatch by type, not by text. simple_room_server::onPeerDataRecv routes
CLI_DATA/CLI_COMMAND to the admin CLI and turns PLAIN into a post;
simple_repeater has no chat function, so it runs every accepted type
through handleCommand with no text test at all. openHop asked
_is_cli_command(message_text) for both, and so got both wrong: a PLAIN
"set ..." ran as a command instead of being posted, a command whose text
was not in the prefix list was published to the room, and an admin's
mistyped repeater command vanished silently instead of being answered.
Guard replays. Both servers open the branch with
`sender_timestamp >= client->last_timestamp`, treat equality as a retry
they accept but do not re-run, and drop anything older. openHop checked
neither, so replaying an admin command with the same timestamp and
different attempt bits re-executed it -- packet dedup does not catch that
because the bytes differ. Note ACL._is_replay is not reusable here: its
strict <= rejects the equal case this branch accepts.
Both read from packet.decrypted, which openhop_core now publishes. A core
too old to report them falls back to the previous text-prefix behaviour
and logs a warning rather than dropping every message.
simple_room_server::onPeerDataRecv takes the
`(client->permissions & PERM_ACL_ROLE_MASK) == PERM_ACL_GUEST` branch for a
PLAIN message and stores nothing. openHop called add_post for anyone who
authenticated, so knowing the guest password was enough to write to a room:
add_post caps length and rate-limits per client, but never looks at the role.
Also warns once per message when openhop_core does not publish txt_type,
so the open text-type gate is visible in the log rather than silent.
simple_repeater::onPeerDataRecv and simple_room_server::onPeerDataRecv
both open their TXT_MSG branch by rejecting any flags outside
{PLAIN, CLI_DATA, CLI_COMMAND}. openHop had no such gate: it decided what
a message was purely from whether the text looked like a command, so a
SIGNED_PLAIN -- a room post, whose 4-byte author prefix the core handler
has already stripped, leaving bare text -- ran as an admin CLI command
whenever it happened to start with a command prefix, and was stored as a
post on a room server that firmware would have dropped.
CLI_COMMAND (3) joined the accepted set when firmware split "a CLI
command" out of CLI_DATA (MeshCore 2c0ace25); CLI_DATA stays in it
because that is what every released client still sends.
Reads the type from packet.decrypted, which the core text handler now
publishes. A core too old to report it leaves the gate open rather than
dropping every message.
`pre-commit run --all-files` fails on `dev` at the `ruff-format` hook, so every
open pull request is red on a check that has nothing to do with its own diff.
Both files arrived with 72da7ee (pr-436) formatted by an earlier ruff.
Purely cosmetic, and exactly what `ruff format` produces:
- `sqlite_handler.py` — a generator expression and an f-string concatenation
that both now fit inside the line limit unsplit.
- `test_companion_import_repeater_contacts.py` — the blank line ruff requires
after a module docstring.
agessaman