project-yukart/docs/ORIN-RUNTIME.md

---
type: reference
---
# Orin runtime

[ARCHITECTURE.md](ARCHITECTURE.md) describes services and the bus without saying
how they coexist on one 8 GB board. This document covers that: what runs, in what
order, under what budget, and what happens when it goes wrong.

Cross-cutting by nature — it spans `svc-core`, `svc-nav`, `svc-gw` and the host
repo, so it lives in the hub rather than in any one of them.

## Layers

```
  clients        app-flutter, app-esp32          off the Orin
  ─────────────────────────────────────────────────────────
  serving        svc-gw, svc-nav
  processing     decoder, recorder
  acquisition    can-bridge, obd-poller, gnss
  bus            mosquitto
  ─────────────────────────────────────────────────────────
  host           kernel modules, gpsd, ALSA, network         orin-nano-env-setup
```

A layer may depend on the one below it and on the bus. Nothing depends sideways:
no service waits on another service's process, only on the broker.

## The binding constraint: 8 GB, shared

The Orin Nano has 8 GB of LPDDR5 **shared between CPU and GPU**. There is no
separate VRAM to fall back on — a model loaded onto the GPU takes memory away
from everything else on the board. Roughly 6–7 GB is left once JetPack and the OS
have taken their share.

| Consumer | Resident | Note |
|---|---|---|
| OS + JetPack | ~1–1.5 GB | before anything of ours starts |
| mosquitto, acquisition, `svc-gw` | tens of MB | not a factor |
| Valhalla server | small; tiles are **mmap'd page cache** | evictable — see below |
| Whisper | 273 MB (tiny) → 3.9 GB (large) | `small` is 244 M params; `large` does not fit |
| GPT-SoVITS | upstream guidance assumes an 8 GB VRAM card | the largest unknown; **measure before designing around it** |
| vision (Phase 5) | capture and H.264: small; inference: — | capture + software H.264 costs ~1.3–2 of 6 CPU cores at 1080p (no NVENC on the Orin Nano); inference competes directly with voice for the same pool |

Four rules follow.

**Reasoning runs in the cloud.** The local voice stack fits only because the LLM
is not resident. An on-device LLM and the voice stack do not coexist in this
budget.

**Size Whisper by vocabulary, not by benchmark.** In-car commands are a narrow,
predictable set. `large` buys accuracy this application cannot spend, at a cost
this board cannot pay.

**Voice and vision cannot both be resident.** Phase 5 has to plan for eviction,
not addition. Worth knowing now, while vision is still deferred.

**Valhalla's memory-mapped tiles are not a leak.** They appear as page cache and
are evictable. Under pressure the kernel drops tile pages and re-faults them on
next use — that costs routing latency, not an OOM. This is the graceful failure
in the system and should be left alone.

## Startup tiers

Ordered by what is lost if a tier is late.

| Tier | Units | Why here |
|---|---|---|
| 0 | kernel modules, gpsd, network | Host layer; nothing of ours can precede it |
| 1 | `mosquitto`, `can-bridge`, `recorder` | **Data is lost if late.** Must reach ready fastest |
| 2 | `decoder`, `gnss`, `obd-poller` | Derive from tier 1; a late start loses nothing |
| 3 | `svc-gw`, `svc-nav` | Serving; a display can wait |
| 4 | voice — ASR, TTS | Interactive; seconds late is harmless |

**Capture first** is the whole principle. A CAN frame missed at tier 1 is gone
forever; a map that loads five seconds into the drive costs nothing. Ordering the
boot the intuitive way — get the screen up quickly — trades irreplaceable data for
a cosmetic gain.

## Failure isolation

- `Restart=always` on every unit. A service dying mid-drive is a normal event.
- Units order themselves `After=` the broker and nothing else. No service may wait
  on another service, or a restart cascades.
- The broker is the single point of failure, and that is accepted: retained topics
  make reconnection cheap, so a client that comes back has state on its first
  message rather than waiting for the next publish.
- No service holds another's state. Anything worth surviving a restart is either
  retained on the bus or on disk.

## Ignition off

Shutdown is the reverse of startup, so tier 1 dies last.

The only thing that *must* complete is the recorder's flush. It therefore owns its
own flush and shutdown handling rather than trusting the OS to get there in time —
a truncated session is the one failure that cannot be recovered by restarting
anything.

## Storage

Self-built binaries live under `/home/mikkeli/infra/` per the host repo's
guidelines. Data does not.

| Data | Size | Growth |
|---|---|---|
| Map artifacts | GB | Replaced wholesale, quarterly at most |
| Voice models | GB | Static |
| Recorded sessions | GB per drive | **Unbounded** |

Recordings are the problem. Without a bound the disk fills mid-drive and capture
stops — silently, because a full disk is not a crash.

**Recordings are bounded with rotation.** The recorder enforces its own ceiling and
discards oldest-first; it does not rely on an external cleanup job that might not
have run. Two properties matter more than the specific size:

- The bound is enforced by the writer, so the failure mode is *losing the oldest
  session*, which is recoverable, rather than *silently stopping capture*, which
  is not.
- A session in progress is never the one rotated away.

## Audio

The **tablet** owns the car's single aux input. It is the music player, and that
role is not moving to the Orin.

```
  tablet: media (Spotify etc.) ──┐
                                 ├──▶ Android mixer ──▶ aux in
  Orin:   GPT-SoVITS (TTS) ──────┘
          └── utterances sent over the reliable boundary channel

  Orin:   mic ──▶ Whisper (ASR)        stays local, never crosses
```

The tablet is the media source, so it holds the aux cable whatever is decided
about speech. Giving the Orin its own aux line would not free the display unit —
it would add a second permanent cable to the car and leave the first one in place.

### TTS crosses as chunked utterances

An utterance travels as a **sequence of chunks with an explicit terminator**, over
the reliable channel. Not as one object, and not as a clock-synchronised audio
stream.

This is deliberately the shape that keeps both modes open, because whole-utterance
is simply a stream of one chunk:

| TTS capability | What it sends |
|---|---|
| Synthesises fully, then returns | one chunk, then terminator |
| Streams as it synthesises | chunks as produced, then terminator |

Same protocol, same receiver. The receiver starts playback on the first chunk,
appends as more arrive, and finishes on the terminator — which is what Android's
`AudioTrack` streaming mode does natively.

⚠ **Never ship an utterance as a single response body with a "download, then
play" player.** It works, and it forecloses streaming permanently — the receiver
has no way to begin before the last byte. The chunk boundary is what keeps both
TTS modes available.

What *is* fine early on: a receiver that buffers chunks and plays on the
terminator. The wire format is already the streaming one, so moving to playback
on the first chunk is a change to the receiver alone. Streaming playback is the
end state.

### The transport is testable with no real TTS

Audio is the lowest-priority part of the system, but its transport does not have
to wait for GPT-SoVITS. A canned WAV, or `espeak-ng` — instant, tiny, and awful —
exercises chunking, ordering, the terminator, ducking and the no-sink path
completely. Voice quality is irrelevant to every one of those.

### Ducking is the tablet's job

Android's `AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK` ducks other applications' audio —
Spotify included — for the duration of a request, then restores it. Cross-
application ducking is native on the tablet.

### The microphone stays on the Orin

Whisper runs on the Orin, so its input does too. Audio *in* on one box and audio
*out* on the other is asymmetric on purpose: each side does the simple thing, and
a microphone has no reason to reach the head unit.

The USB camera has a microphone (48 kHz stereo, ALSA card `U1080P`): a candidate
input, if its placement and quality suit speech. Untested.

The Orin runs headless with no sound server: capture reads the ALSA device
directly. If a service ever needs PipeWire, run it for that service's user
(lingering), not as a desktop session.

### Degrade when there is no sink

With no tablet connected the Orin can still hear — the mic is local — but cannot
speak. The voice stack expires or drops pending utterances rather than queueing
them unboundedly or blocking on a sink that may never arrive.

**The tablet is required for any audio at all** — it is both the media source and
the speech sink.