svc-core/docs/impl/reports/BUS-BENCH.md

type, date

type	date
report	2026-10-12

Raw CAN over MQTT: bus bench

Can mosquitto carry yukart/can/raw at real CAN rates (project-yukart decision 4)? Measured on the car Orin (mosquitto 2.0.11, localhost) with bus-bench: bridge reads a vcan0 and publishes frames, sink subscribes, checks sequence numbers and measures latency from the kernel's reception timestamp to the subscriber. The car's usual services (recorder, orin-status, svc-gw, svc-vision) kept running.

scripts/bus-bench.sh ./bus-bench z27ag-20261009-163736-ign.log 20     # on the Orin
scripts/bus-mix.sh   ./bus-bench z27ag-20261009-163736-ign.log 20     # after bus-bench.sh

Loads:

• car replay: the in-car capture (z27ag-20261009-163736-ign.log, engine at idle) at its own timing, about 890 frames/s. This is the real bus.
• full bus: cangen -L 8 at about 4,400 frames/s. 500 kbit/s carries roughly 3,700–4,500 8-byte frames/s depending on bit stuffing, so this is a saturated bus.
• flood: cangen -g 0, as fast as vcan goes, to find where it breaks.

Two encodings: one MQTT message per frame, or frames batched over a 10 ms window. Both use the same provisional binary format (24 bytes a frame, 12-byte header; see src/bus-bench/main.rs). QoS 0, not retained.

Results (20 s runs; full bus 15 s)

With TCP_NODELAY on both the clients and the broker (see below):

Load	Encoding	Frames/s delivered	Lost	Latency p50 / p99 / max (ms)	CPU: broker · bridge · sink (% of a core)
car replay	per frame	893	0	0.31 / 0.92 / 1.3	4.7 · 3.5 · 3.8
car replay	10 ms batch	893	0	8.7 / 15.3 / 17.6	0.9 · 1.2 · 0.6
full bus	per frame	4,440	0	0.16 / 0.30 / 3.3	27.5 · 21.7 · 17.0
full bus	10 ms batch	4,562	0	5.3 / 10.4 / 10.8	1.1 · 3.5 · 0.8
flood	per frame	45,700	84 %	~1.8 s, queues full	85 · 86 · 81
flood	10 ms batch	268,800	0	5.3 / 10.2 / 11.1	1.3 · 63 · 0.9

• Real rates hold, with margin. Per frame breaks at about 45,000 messages/s: ten times a saturated bus and fifty times the real one.
• Batching is the clear choice. At a full bus it costs the broker about 1 % of a core instead of 28 %, at a fixed ~100 messages/s whatever the bus load. Frames are never lost: the batched flood (60 times a full bus) still lost none, and the bridge's own CAN reads became the limit. The price is up to one window of latency (the 10 ms here). That is no problem for a dashboard, a recorder or a decoder.
• Per frame stays the option for a consumer that needs sub-millisecond latency. Nothing planned does.

Nagle: the 40 ms tail

The first runs had a p99.9 near 40 ms at every load: Nagle's algorithm holding small writes back until the previous one is acknowledged, against delayed ACKs. Both sides do it by default: rumqttc (NetworkOptions::set_tcp_nodelay, off by default) and mosquitto (set_tcp_nodelay, off by default).

Full bus (~3,000 frames/s)	per frame p99 / p99.9 / max	10 ms batch p99 / max
Nagle on both sides	23.5 / 41.7 / 45	65.6 / 80.9
client no-delay, broker Nagle	0.61 / 38.0 / 47	43.3 / 55.3
no-delay on both	0.33 / 0.67 / 2.8	10.5 / 10.8

The car's broker now has set_tcp_nodelay true (orin-nano-env-setup MOSQUITTO.md). Clients that publish or consume raw CAN set set_tcp_nodelay(true) too. At once a second (orin-status) it does not matter.

The bus as the car will use it

scripts/bus-mix.sh: the raw stream (10 ms batches) to four subscribers (the decoder plus three: svc-gw, telemetry-gw, a debug tool), and a stand-in decoder publishing envelopes on one retained topic per signal, to five subscribers that each take every signal (svc-gw, recorder, svc-nav, svc-vision, telemetry-gw). Loads keep the car's 27 CAN IDs and their rates (2–100 Hz): the capture as recorded, and played 5× faster for a full bus. Signals per frame and a per-signal rate cap (latest value wins) stand in for the decoder design nobody has made yet. 20 s runs.

Load	Signals/frame	Cap	Envelopes/s	Deliveries/s (×5)	Broker CPU	Envelope latency p50 / p99 (ms)
car	3	50 Hz	1,600	8,000	22 %	13 / 21
car	3	none	2,550	12,800	35 %	13 / 20
car	8	20 Hz	2,550	12,800	32 %	14 / 27
car	8	50 Hz	4,300	21,600	47 %	16 / 32
car	8	none	6,800	34,000	51 %	14 / 47
full bus	3	50 Hz	3,100	15,500	39 %	12 / 18
full bus	8	20 Hz	3,500	17,300	44 %	22 / 41
full bus	8	50 Hz	8,100	40,600	57 %	15 / 391
full bus	3	none	12,800	63,800	74 %	636 / 1,472
full bus	8	none	16,300 offered	—	75 %	5,380 / 5,994; decoder queue overflowing

• The raw stream never suffered. All four raw subscribers got every frame in every run, p99 ≤ 16 ms, including the runs where the envelopes collapsed.
• Envelopes are the load. The broker's cost follows deliveries (messages × subscribers). Up to ~22,000 deliveries/s it stays under half a core with p99 near 30 ms; around 40,000 the tail goes to hundreds of ms; past ~60,000 queues grow by seconds. Mosquitto is single-threaded, so this is a ceiling, not a share.
• Each subscriber pays too. Parsing JSON envelopes cost each sink 14–38 % of a core at these rates; five of them took two cores. A real subscriber that takes everything (svc-gw) pays the same.
• Envelope loss is not measured exactly (no sequence numbers in the envelope); the decoder's own refused counts what it could not even queue.

What keeps it in the comfortable range, any of which is enough at the car's rate:

1. Cap each signal's rate in the decoder, latest value wins. 20–50 Hz is still 10–50× OBD-II polling, and the raw stream keeps the full rate for anything that needs it.
2. Subscribe narrowly. Only a consumer that shows or stores everything (svc-gw) needs yukart/vehicle/+; svc-nav and svc-vision need a handful of topics.
3. Publish slow signals on change, with a heartbeat; temperatures and states do not need 50 Hz.
4. The recorder need not record decoded signals at full rate. It records raw frames, and decoding them again offline gives the same signals.

Not tested

• A real CAN adapter: vcan has no bus timing, and the PCAN-USB's own reception and timestamps are already covered by the recorder.
• The broker under memory pressure, or with the GPU/vision load of later phases.
• The real decoder: signals per frame, rates and topic layout are stand-ins.