project-yukart/docs/ORIN-RUNTIME.md

type

type
reference

Orin runtime

ARCHITECTURE.md describes services and the bus without saying how they coexist on one 8 GB board. This document covers that: what runs, in what order, under what budget, and what happens when it goes wrong.

Cross-cutting by nature — it spans svc-core, svc-nav, svc-gw and the host repo, so it lives in the hub rather than in any one of them.

Layers

  clients        app-flutter, app-esp32          off the Orin
  ─────────────────────────────────────────────────────────
  serving        svc-gw, svc-nav
  processing     decoder, recorder
  acquisition    can-bridge, obd-poller, gnss
  bus            mosquitto
  ─────────────────────────────────────────────────────────
  host           kernel modules, gpsd, ALSA, network         orin-nano-env-setup

A layer may depend on the one below it and on the bus. Nothing depends sideways: no service waits on another service's process, only on the broker.

The binding constraint: 8 GB, shared

The Orin Nano has 8 GB of LPDDR5 shared between CPU and GPU. There is no separate VRAM to fall back on — a model loaded onto the GPU takes memory away from everything else on the board. Roughly 6–7 GB is left once JetPack and the OS have taken their share.

Consumer	Resident	Note
OS + JetPack	~1–1.5 GB	before anything of ours starts
mosquitto, acquisition, svc-gw	tens of MB	not a factor
Valhalla server	small; tiles are mmap'd page cache	evictable — see below
Whisper	273 MB (tiny) → 3.9 GB (large)	small is 244 M params; large does not fit
GPT-SoVITS	upstream guidance assumes an 8 GB VRAM card	the largest unknown; measure before designing around it
vision (Phase 5)	—	competes directly with voice for the same pool

Four rules follow.

Reasoning runs in the cloud. The local voice stack fits only because the LLM is not resident. An on-device LLM and the voice stack do not coexist in this budget.

Size Whisper by vocabulary, not by benchmark. In-car commands are a narrow, predictable set. large buys accuracy this application cannot spend, at a cost this board cannot pay.

Voice and vision cannot both be resident. Phase 5 has to plan for eviction, not addition. Worth knowing now, while vision is still deferred.

Valhalla's memory-mapped tiles are not a leak. They appear as page cache and are evictable. Under pressure the kernel drops tile pages and re-faults them on next use — that costs routing latency, not an OOM. This is the graceful failure in the system and should be left alone.

Startup tiers

Ordered by what is lost if a tier is late.

Tier	Units	Why here
0	kernel modules, gpsd, network	Host layer; nothing of ours can precede it
1	mosquitto, can-bridge, recorder	Data is lost if late. Must reach ready fastest
2	decoder, gnss, obd-poller	Derive from tier 1; a late start loses nothing
3	svc-gw, svc-nav	Serving; a display can wait
4	voice — ASR, TTS	Interactive; seconds late is harmless

Capture first is the whole principle. A CAN frame missed at tier 1 is gone forever; a map that loads five seconds into the drive costs nothing. Ordering the boot the intuitive way — get the screen up quickly — trades irreplaceable data for a cosmetic gain.

Failure isolation

• Restart=always on every unit. A service dying mid-drive is a normal event.
• Units order themselves After= the broker and nothing else. No service may wait on another service, or a restart cascades.
• The broker is the single point of failure, and that is accepted: retained topics make reconnection cheap, so a client that comes back has state on its first message rather than waiting for the next publish.
• No service holds another's state. Anything worth surviving a restart is either retained on the bus or on disk.

Ignition off

Shutdown is the reverse of startup, so tier 1 dies last.

The only thing that must complete is the recorder's flush. It therefore owns its own flush and shutdown handling rather than trusting the OS to get there in time — a truncated session is the one failure that cannot be recovered by restarting anything.

Storage

Self-built binaries live under /home/mikkeli/infra/ per the host repo's guidelines. Data does not.

Data	Size	Growth
Map artifacts	GB	Replaced wholesale, quarterly at most
Voice models	GB	Static
Recorded sessions	GB per drive	Unbounded

Recordings are the problem. Without a bound the disk fills mid-drive and capture stops — silently, because a full disk is not a crash.

Recordings are bounded with rotation. The recorder enforces its own ceiling and discards oldest-first; it does not rely on an external cleanup job that might not have run. Two properties matter more than the specific size:

• The bound is enforced by the writer, so the failure mode is losing the oldest session, which is recoverable, rather than silently stopping capture, which is not.
• A session in progress is never the one rotated away.

Audio

The tablet owns the car's single aux input. It is the music player, and that role is not moving to the Orin.

  tablet: media (Spotify etc.) ──┐
                                 ├──▶ Android mixer ──▶ aux in
  Orin:   GPT-SoVITS (TTS) ──────┘
          └── utterances sent over the reliable boundary channel

  Orin:   mic ──▶ Whisper (ASR)        stays local, never crosses

The tablet is the media source, so it holds the aux cable whatever is decided about speech. Giving the Orin its own aux line would not free the display unit — it would add a second permanent cable to the car and leave the first one in place.

TTS crosses as chunked utterances

An utterance travels as a sequence of chunks with an explicit terminator, over the reliable channel. Not as one object, and not as a clock-synchronised audio stream.

This is deliberately the shape that keeps both modes open, because whole-utterance is simply a stream of one chunk:

TTS capability	What it sends
Synthesises fully, then returns	one chunk, then terminator
Streams as it synthesises	chunks as produced, then terminator

Same protocol, same receiver. The receiver starts playback on the first chunk, appends as more arrive, and finishes on the terminator — which is what Android's AudioTrack streaming mode does natively.

⚠ Never ship an utterance as a single response body with a "download, then play" player. It works, and it forecloses streaming permanently — the receiver has no way to begin before the last byte. The chunk boundary is what keeps both TTS modes available.

The transport is testable with no real TTS

Audio is the lowest-priority part of the system, but its transport does not have to wait for GPT-SoVITS. A canned WAV, or espeak-ng — instant, tiny, and awful — exercises chunking, ordering, the terminator, ducking and the no-sink path completely. Voice quality is irrelevant to every one of those.

Ducking is the tablet's job

Android's AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK ducks other applications' audio — Spotify included — for the duration of a request, then restores it. Cross- application ducking is native on the tablet.

The microphone stays on the Orin

Whisper runs on the Orin, so its input does too. Audio in on one box and audio out on the other is asymmetric on purpose: each side does the simple thing, and a microphone has no reason to reach the head unit.

The Orin runs headless with no sound server: capture reads the ALSA device directly. If a service ever needs PipeWire, run it for that service's user (lingering), not as a desktop session.

Degrade when there is no sink

With no tablet connected the Orin can still hear — the mic is local — but cannot speak. The voice stack expires or drops pending utterances rather than queueing them unboundedly or blocking on a sink that may never arrive.

The tablet is required for any audio at all — it is both the media source and the speech sink.