--- type: reference --- # Orin runtime [ARCHITECTURE.md](ARCHITECTURE.md) describes services and the bus without saying how they coexist on one 8 GB board. This document covers that: what runs, in what order, under what budget, and what happens when it goes wrong. Cross-cutting by nature — it spans `svc-core`, `svc-nav`, `svc-gw` and the host repo, so it lives in the hub rather than in any one of them. ## Layers ``` clients app-flutter, app-esp32 off the Orin ───────────────────────────────────────────────────────── serving svc-gw, svc-nav processing decoder, recorder acquisition can-bridge, obd-poller, gnss bus mosquitto ───────────────────────────────────────────────────────── host kernel modules, gpsd, ALSA, network orin-nano-env-setup ``` A layer may depend on the one below it and on the bus. Nothing depends sideways: no service waits on another service's process, only on the broker. ## The binding constraint: 8 GB, shared The Orin Nano has 8 GB of LPDDR5 **shared between CPU and GPU**. There is no separate VRAM to fall back on — a model loaded onto the GPU takes memory away from everything else on the board. Roughly 6–7 GB is left once JetPack and the OS have taken their share. | Consumer | Resident | Note | |---|---|---| | OS + JetPack | ~1–1.5 GB | before anything of ours starts | | mosquitto, acquisition, `svc-gw` | tens of MB | not a factor | | Valhalla server | small; tiles are **mmap'd page cache** | evictable — see below | | Whisper | 273 MB (tiny) → 3.9 GB (large) | `small` is 244 M params; `large` does not fit | | GPT-SoVITS | upstream guidance assumes an 8 GB VRAM card | the largest unknown; **measure before designing around it** | | vision (Phase 5) | — | competes directly with voice for the same pool | Four rules follow. **Reasoning runs in the cloud.** The local voice stack fits only because the LLM is not resident. An on-device LLM and the voice stack do not coexist in this budget. **Size Whisper by vocabulary, not by benchmark.** In-car commands are a narrow, predictable set. `large` buys accuracy this application cannot spend, at a cost this board cannot pay. **Voice and vision cannot both be resident.** Phase 5 has to plan for eviction, not addition. Worth knowing now, while vision is still deferred. **Valhalla's memory-mapped tiles are not a leak.** They appear as page cache and are evictable. Under pressure the kernel drops tile pages and re-faults them on next use — that costs routing latency, not an OOM. This is the graceful failure in the system and should be left alone. ## Startup tiers Ordered by what is lost if a tier is late. | Tier | Units | Why here | |---|---|---| | 0 | kernel modules, gpsd, network | Host layer; nothing of ours can precede it | | 1 | `mosquitto`, `can-bridge`, `recorder` | **Data is lost if late.** Must reach ready fastest | | 2 | `decoder`, `gnss`, `obd-poller` | Derive from tier 1; a late start loses nothing | | 3 | `svc-gw`, `svc-nav` | Serving; a display can wait | | 4 | voice — ASR, TTS | Interactive; seconds late is harmless | **Capture first** is the whole principle. A CAN frame missed at tier 1 is gone forever; a map that loads five seconds into the drive costs nothing. Ordering the boot the intuitive way — get the screen up quickly — trades irreplaceable data for a cosmetic gain. ## Failure isolation - `Restart=always` on every unit. A service dying mid-drive is a normal event. - Units order themselves `After=` the broker and nothing else. No service may wait on another service, or a restart cascades. - The broker is the single point of failure, and that is accepted: retained topics make reconnection cheap, so a client that comes back has state on its first message rather than waiting for the next publish. - No service holds another's state. Anything worth surviving a restart is either retained on the bus or on disk. ## Ignition off Shutdown is the reverse of startup, so tier 1 dies last. The only thing that *must* complete is the recorder's flush. It therefore owns its own flush and shutdown handling rather than trusting the OS to get there in time — a truncated session is the one failure that cannot be recovered by restarting anything. ## Storage Self-built binaries live under `/home/mikkeli/infra/` per the host repo's guidelines. Data does not. | Data | Size | Growth | |---|---|---| | Map artifacts | GB | Replaced wholesale, quarterly at most | | Voice models | GB | Static | | Recorded sessions | GB per drive | **Unbounded** | Recordings are the problem. Without a bound the disk fills mid-drive and capture stops — silently, because a full disk is not a crash. **Recordings are bounded with rotation.** The recorder enforces its own ceiling and discards oldest-first; it does not rely on an external cleanup job that might not have run. Two properties matter more than the specific size: - The bound is enforced by the writer, so the failure mode is *losing the oldest session*, which is recoverable, rather than *silently stopping capture*, which is not. - A session in progress is never the one rotated away. ## Audio The **tablet** owns the car's single aux input. It is the music player, and that role is not moving to the Orin. ``` tablet: media (Spotify etc.) ──┐ ├──▶ Android mixer ──▶ aux in Orin: GPT-SoVITS (TTS) ──────┘ └── utterances sent over the reliable boundary channel Orin: mic ──▶ Whisper (ASR) stays local, never crosses ``` The tablet is the media source, so it holds the aux cable whatever is decided about speech. Giving the Orin its own aux line would not free the display unit — it would add a second permanent cable to the car and leave the first one in place. ### TTS crosses as chunked utterances An utterance travels as a **sequence of chunks with an explicit terminator**, over the reliable channel. Not as one object, and not as a clock-synchronised audio stream. This is deliberately the shape that keeps both modes open, because whole-utterance is simply a stream of one chunk: | TTS capability | What it sends | |---|---| | Synthesises fully, then returns | one chunk, then terminator | | Streams as it synthesises | chunks as produced, then terminator | Same protocol, same receiver. The receiver starts playback on the first chunk, appends as more arrive, and finishes on the terminator — which is what Android's `AudioTrack` streaming mode does natively. ⚠ **Never ship an utterance as a single response body with a "download, then play" player.** It works, and it forecloses streaming permanently — the receiver has no way to begin before the last byte. The chunk boundary is what keeps both TTS modes available. ### The transport is testable with no real TTS Audio is the lowest-priority part of the system, but its transport does not have to wait for GPT-SoVITS. A canned WAV, or `espeak-ng` — instant, tiny, and awful — exercises chunking, ordering, the terminator, ducking and the no-sink path completely. Voice quality is irrelevant to every one of those. ### Ducking is the tablet's job Android's `AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK` ducks other applications' audio — Spotify included — for the duration of a request, then restores it. Cross- application ducking is native on the tablet. ### The microphone stays on the Orin Whisper runs on the Orin, so its input does too. Audio *in* on one box and audio *out* on the other is asymmetric on purpose: each side does the simple thing, and a microphone has no reason to reach the head unit. The Orin runs headless with no sound server: capture reads the ALSA device directly. If a service ever needs PipeWire, run it for that service's user (lingering), not as a desktop session. ### Degrade when there is no sink With no tablet connected the Orin can still hear — the mic is local — but cannot speak. The voice stack expires or drops pending utterances rather than queueing them unboundedly or blocking on a sink that may never arrive. **The tablet is required for any audio at all** — it is both the media source and the speech sink.