project-yukart/docs/ORIN-RUNTIME.md

Orin runtime

ARCHITECTURE.md describes services and the bus without saying how they coexist on one 8 GB board. This document covers that: what runs, in what order, under what budget, and what happens when it goes wrong.

Cross-cutting by nature — it spans svc-core, svc-nav, svc-gw and the host repo, so it lives in the hub rather than in any one of them.

Layers

  clients        app-flutter, app-esp32          off the Orin
  ─────────────────────────────────────────────────────────
  serving        svc-gw, svc-nav
  processing     decoder, recorder
  acquisition    can-bridge, obd-poller, gnss
  bus            mosquitto
  ─────────────────────────────────────────────────────────
  host           kernel modules, gpsd, PipeWire, network     orin-nano-env-setup

A layer may depend on the one below it and on the bus. Nothing depends sideways: no service waits on another service's process, only on the broker.

The binding constraint: 8 GB, shared

The Orin Nano has 8 GB of LPDDR5 shared between CPU and GPU. There is no separate VRAM to fall back on — a model loaded onto the GPU takes memory away from everything else on the board. Roughly 6–7 GB is left once JetPack and the OS have taken their share.

Consumer	Resident	Note
OS + JetPack	~1–1.5 GB	before anything of ours starts
mosquitto, acquisition, svc-gw	tens of MB	not a factor
Valhalla server	small; tiles are mmap'd page cache	evictable — see below
Whisper	273 MB (tiny) → 3.9 GB (large)	small is 244 M params; large does not fit
GPT-SoVITS	upstream guidance assumes an 8 GB VRAM card	the largest unknown; measure before designing around it
vision (Phase 5)	—	competes directly with voice for the same pool

Four rules follow.

The cloud LLM is load-bearing. OVERVIEW.md puts reasoning in the cloud. That decision is what makes a local voice stack fit at all — moving the LLM on-device collapses this budget entirely. It is not a free choice to revisit.

Size Whisper by vocabulary, not by benchmark. In-car commands are a narrow, predictable set. large buys accuracy this application cannot spend, at a cost this board cannot pay.

Voice and vision cannot both be resident. Phase 5 has to plan for eviction, not addition. Worth knowing now, while vision is still deferred.

Valhalla's memory-mapped tiles are not a leak. They appear as page cache and are evictable. Under pressure the kernel drops tile pages and re-faults them on next use — that costs routing latency, not an OOM. This is the graceful failure in the system and should be left alone.

Startup tiers

Ordered by what is lost if a tier is late.

Tier	Units	Why here
0	kernel modules, gpsd, network	Host layer; nothing of ours can precede it
1	mosquitto, can-bridge, recorder	Data is lost if late. Must reach ready fastest
2	decoder, gnss, obd-poller	Derive from tier 1; a late start loses nothing
3	svc-gw, svc-nav	Serving; a display can wait
4	voice — ASR, TTS	Interactive; seconds late is harmless

Capture first is the whole principle. A CAN frame missed at tier 1 is gone forever; a map that loads five seconds into the drive costs nothing. Ordering the boot the intuitive way — get the screen up quickly — trades irreplaceable data for a cosmetic gain.

Failure isolation

• Restart=always on every unit. A service dying mid-drive is a normal event.
• Units order themselves After= the broker and nothing else. No service may wait on another service, or a restart cascades.
• The broker is the single point of failure, and that is accepted: retained topics make reconnection cheap, so a client that comes back has state on its first message rather than waiting for the next publish.
• No service holds another's state. Anything worth surviving a restart is either retained on the bus or on disk.

Ignition off

Shutdown is the reverse of startup, so tier 1 dies last.

The only thing that must complete is the recorder's flush. It therefore owns its own flush and shutdown handling rather than trusting the OS to get there in time — a truncated session is the one failure that cannot be recovered by restarting anything.

Storage

Self-built binaries live under /home/mikkeli/infra/ per the host repo's guidelines. Data does not.

Data	Size	Growth
Map artifacts	GB	Replaced wholesale, quarterly at most
Voice models	GB	Static
Recorded sessions	GB per drive	Unbounded

Recordings are the problem. Without a bound the disk fills mid-drive and capture stops — silently, because a full disk is not a crash.

Recordings are bounded with rotation. The recorder enforces its own ceiling and discards oldest-first; it does not rely on an external cleanup job that might not have run. Two properties matter more than the specific size:

• The bound is enforced by the writer, so the failure mode is losing the oldest session, which is recoverable, rather than silently stopping capture, which is not.
• A session in progress is never the one rotated away.

Audio

The tablet owns the car's single aux input. It is the music player, and that role is not moving to the Orin.

  tablet: media (Spotify etc.) ──┐
                                 ├──▶ Android mixer ──▶ aux in
  Orin:   GPT-SoVITS (TTS) ──────┘
          └── utterances sent over the reliable boundary channel

  Orin:   mic ──▶ Whisper (ASR)        stays local, never crosses

The tablet is the media source, so it holds the aux cable whatever is decided about speech. Giving the Orin its own aux line would not free the display unit — it would add a second permanent cable to the car and leave the first one in place.

TTS crosses as chunked utterances

An utterance travels as a sequence of chunks with an explicit terminator, over the reliable channel. Not as one object, and not as a clock-synchronised audio stream.

This is deliberately the shape that keeps both modes open, because whole-utterance is simply a stream of one chunk:

TTS capability	What it sends
Synthesises fully, then returns	one chunk, then terminator
Streams as it synthesises	chunks as produced, then terminator

Same protocol, same receiver. The receiver starts playback on the first chunk, appends as more arrive, and finishes on the terminator — which is what Android's AudioTrack streaming mode does natively.

What must not be built: an utterance as a single response body with a "download the file, then play the file" player. That works today and forecloses streaming permanently. Whether the TTS we land on streams is still open, so the transport must not assume either way.

The transport is testable with no real TTS

Audio is the lowest-priority part of the system, but its transport does not have to wait for GPT-SoVITS. A canned WAV, or espeak-ng — instant, tiny, and awful — exercises chunking, ordering, the terminator, ducking and the no-sink path completely. Voice quality is irrelevant to every one of those.

Ducking is the tablet's job

Android's AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK ducks other applications' audio — Spotify included — for the duration of a request, then restores it. This is the mechanism every navigation app already uses, and it is less work than the PipeWire policy it replaces. Cross-application ducking is a solved problem on the tablet and would have been a bespoke one on the Orin.

The microphone stays on the Orin

Whisper runs on the Orin, so its input does too. Audio in on one box and audio out on the other is asymmetric on purpose: each side does the simple thing, and a microphone has no reason to reach the head unit.

Degrade when there is no sink

With no tablet connected the Orin can still hear — the mic is local — but cannot speak. The voice stack expires or drops pending utterances rather than queueing them unboundedly or blocking on a sink that may never arrive.

The consequence to accept: unplug the tablet and the car has no voice. That is tolerable because unplugging it also removes the music, so the tablet was already load-bearing for audio. Speech does not make it more so.