svc-core/docs/impl/reports/BUS-BENCH.md

type, date

type	date
report	2026-10-12

Raw CAN over MQTT: bus bench

Can mosquitto carry yukart/can/raw at real CAN rates (project-yukart decision 4)? Measured on the car Orin (mosquitto 2.0.11, localhost) with bus-bench: bridge reads a vcan0 and publishes frames, sink subscribes, checks sequence numbers and measures latency from the kernel's reception timestamp to the subscriber. The car's usual services (recorder, orin-status, svc-gw, svc-vision) kept running.

scripts/bus-bench.sh ./bus-bench z27ag-20261009-163736-ign.log 20     # on the Orin

Loads:

• car replay: the in-car capture (z27ag-20261009-163736-ign.log, engine at idle) at its own timing, about 890 frames/s. This is the real bus.
• full bus: cangen -L 8 at about 4,400 frames/s. 500 kbit/s carries roughly 3,700–4,500 8-byte frames/s depending on bit stuffing, so this is a saturated bus.
• flood: cangen -g 0, as fast as vcan goes, to find where it breaks.

Two encodings: one MQTT message per frame, or frames batched over a 10 ms window. Both use the same provisional binary format (24 bytes a frame, 12-byte header; see src/bus-bench/main.rs). QoS 0, not retained.

Results (20 s runs; full bus 15 s)

With TCP_NODELAY on both the clients and the broker (see below):

Load	Encoding	Frames/s delivered	Lost	Latency p50 / p99 / max (ms)	CPU: broker · bridge · sink (% of a core)
car replay	per frame	893	0	0.31 / 0.92 / 1.3	4.7 · 3.5 · 3.8
car replay	10 ms batch	893	0	8.7 / 15.3 / 17.6	0.9 · 1.2 · 0.6
full bus	per frame	4,440	0	0.16 / 0.30 / 3.3	27.5 · 21.7 · 17.0
full bus	10 ms batch	4,562	0	5.3 / 10.4 / 10.8	1.1 · 3.5 · 0.8
flood	per frame	45,700	84 %	~1.8 s, queues full	85 · 86 · 81
flood	10 ms batch	268,800	0	5.3 / 10.2 / 11.1	1.3 · 63 · 0.9

• Real rates hold, with margin. Per frame breaks at about 45,000 messages/s: ten times a saturated bus and fifty times the real one.
• Batching is the clear choice. At a full bus it costs the broker about 1 % of a core instead of 28 %, at a fixed ~100 messages/s whatever the bus load. Frames are never lost: the batched flood (60 times a full bus) still lost none, and the bridge's own CAN reads became the limit. The price is up to one window of latency (the 10 ms here). That is no problem for a dashboard, a recorder or a decoder.
• Per frame stays the option for a consumer that needs sub-millisecond latency. Nothing planned does.

Nagle: the 40 ms tail

The first runs had a p99.9 near 40 ms at every load: Nagle's algorithm holding small writes back until the previous one is acknowledged, against delayed ACKs. Both sides do it by default: rumqttc (NetworkOptions::set_tcp_nodelay, off by default) and mosquitto (set_tcp_nodelay, off by default).

Full bus (~3,000 frames/s)	per frame p99 / p99.9 / max	10 ms batch p99 / max
Nagle on both sides	23.5 / 41.7 / 45	65.6 / 80.9
client no-delay, broker Nagle	0.61 / 38.0 / 47	43.3 / 55.3
no-delay on both	0.33 / 0.67 / 2.8	10.5 / 10.8

The car's broker now has set_tcp_nodelay true (orin-nano-env-setup MOSQUITTO.md). Clients that publish or consume raw CAN set set_tcp_nodelay(true) too. At once a second (orin-status) it does not matter.

Not tested

• A real CAN adapter: vcan has no bus timing, and the PCAN-USB's own reception and timestamps are already covered by the recorder.
• Several subscribers on yukart/can/raw at once (recorder reads SocketCAN directly, so expected consumers are the decoder and perhaps svc-gw).
• The broker under memory pressure, or with the GPU/vision load of later phases.