mikkeli
b254bb2c55
Also: std::process::Command breaks the cross-compile fallback (GLIBC_2.39). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DDwfd6Ki8UfG4hK6W5YpJz
203 lines
8.8 KiB
Markdown
203 lines
8.8 KiB
Markdown
---
|
||
type: reference
|
||
---
|
||
# Orin runtime
|
||
|
||
[ARCHITECTURE.md](ARCHITECTURE.md) describes services and the bus without saying
|
||
how they coexist on one 8 GB board. This document covers that: what runs, in what
|
||
order, under what budget, and what happens when it goes wrong.
|
||
|
||
Cross-cutting by nature — it spans `svc-core`, `svc-nav`, `svc-gw` and the host
|
||
repo, so it lives in the hub rather than in any one of them.
|
||
|
||
## Layers
|
||
|
||
```
|
||
clients app-flutter, app-esp32 off the Orin
|
||
─────────────────────────────────────────────────────────
|
||
serving svc-gw, svc-nav
|
||
processing decoder, recorder
|
||
acquisition can-bridge, obd-poller, gnss
|
||
bus mosquitto
|
||
─────────────────────────────────────────────────────────
|
||
host kernel modules, gpsd, ALSA, network orin-nano-env-setup
|
||
```
|
||
|
||
A layer may depend on the one below it and on the bus. Nothing depends sideways:
|
||
no service waits on another service's process, only on the broker.
|
||
|
||
## The binding constraint: 8 GB, shared
|
||
|
||
The Orin Nano has 8 GB of LPDDR5 **shared between CPU and GPU**. There is no
|
||
separate VRAM to fall back on — a model loaded onto the GPU takes memory away
|
||
from everything else on the board. Roughly 6–7 GB is left once JetPack and the OS
|
||
have taken their share.
|
||
|
||
| Consumer | Resident | Note |
|
||
|---|---|---|
|
||
| OS + JetPack | ~1–1.5 GB | before anything of ours starts |
|
||
| mosquitto, acquisition, `svc-gw` | tens of MB | not a factor |
|
||
| Valhalla server | small; tiles are **mmap'd page cache** | evictable — see below |
|
||
| Whisper | 273 MB (tiny) → 3.9 GB (large) | `small` is 244 M params; `large` does not fit |
|
||
| GPT-SoVITS | upstream guidance assumes an 8 GB VRAM card | the largest unknown; **measure before designing around it** |
|
||
| vision (Phase 5) | capture and H.264: small; inference: — | capture + software H.264 costs ~1.3–2 of 6 CPU cores at 1080p (no NVENC on the Orin Nano); inference competes directly with voice for the same pool |
|
||
|
||
Four rules follow.
|
||
|
||
**Reasoning runs in the cloud.** The local voice stack fits only because the LLM
|
||
is not resident. An on-device LLM and the voice stack do not coexist in this
|
||
budget.
|
||
|
||
**Size Whisper by vocabulary, not by benchmark.** In-car commands are a narrow,
|
||
predictable set. `large` buys accuracy this application cannot spend, at a cost
|
||
this board cannot pay.
|
||
|
||
**Voice and vision cannot both be resident.** Phase 5 has to plan for eviction,
|
||
not addition. Worth knowing now, while vision is still deferred.
|
||
|
||
**Valhalla's memory-mapped tiles are not a leak.** They appear as page cache and
|
||
are evictable. Under pressure the kernel drops tile pages and re-faults them on
|
||
next use — that costs routing latency, not an OOM. This is the graceful failure
|
||
in the system and should be left alone.
|
||
|
||
## Startup tiers
|
||
|
||
Ordered by what is lost if a tier is late.
|
||
|
||
| Tier | Units | Why here |
|
||
|---|---|---|
|
||
| 0 | kernel modules, gpsd, network | Host layer; nothing of ours can precede it |
|
||
| 1 | `mosquitto`, `can-bridge`, `recorder` | **Data is lost if late.** Must reach ready fastest |
|
||
| 2 | `decoder`, `gnss`, `obd-poller`, `orin-status` | Derive from tier 1; a late start loses nothing |
|
||
| 3 | `svc-gw`, `svc-nav`, `svc-vision` | Serving; a display can wait |
|
||
| 4 | voice — ASR, TTS | Interactive; seconds late is harmless |
|
||
|
||
**Capture first** is the whole principle. A CAN frame missed at tier 1 is gone
|
||
forever; a map that loads five seconds into the drive costs nothing. Ordering the
|
||
boot the intuitive way — get the screen up quickly — trades irreplaceable data for
|
||
a cosmetic gain.
|
||
|
||
## Failure isolation
|
||
|
||
- `Restart=always` on every unit. A service dying mid-drive is a normal event.
|
||
- Units order themselves `After=` the broker and nothing else. No service may wait
|
||
on another service, or a restart cascades.
|
||
- The broker is the single point of failure, and that is accepted: retained topics
|
||
make reconnection cheap, so a client that comes back has state on its first
|
||
message rather than waiting for the next publish.
|
||
- No service holds another's state. Anything worth surviving a restart is either
|
||
retained on the bus or on disk.
|
||
|
||
## Ignition off
|
||
|
||
Shutdown is the reverse of startup, so tier 1 dies last.
|
||
|
||
The only thing that *must* complete is the recorder's flush. It therefore owns its
|
||
own flush and shutdown handling rather than trusting the OS to get there in time —
|
||
a truncated session is the one failure that cannot be recovered by restarting
|
||
anything.
|
||
|
||
## Storage
|
||
|
||
Self-built binaries live under `/home/mikkeli/infra/` per the host repo's
|
||
guidelines. Data does not.
|
||
|
||
| Data | Size | Growth |
|
||
|---|---|---|
|
||
| Map artifacts | GB | Replaced wholesale, quarterly at most |
|
||
| Voice models | GB | Static |
|
||
| Recorded sessions | GB per drive | **Unbounded** |
|
||
|
||
Recordings are the problem. Without a bound the disk fills mid-drive and capture
|
||
stops — silently, because a full disk is not a crash.
|
||
|
||
**Recordings are bounded with rotation.** The recorder enforces its own ceiling and
|
||
discards oldest-first; it does not rely on an external cleanup job that might not
|
||
have run. Two properties matter more than the specific size:
|
||
|
||
- The bound is enforced by the writer, so the failure mode is *losing the oldest
|
||
session*, which is recoverable, rather than *silently stopping capture*, which
|
||
is not.
|
||
- A session in progress is never the one rotated away.
|
||
|
||
## Audio
|
||
|
||
The **tablet** owns the car's single aux input. It is the music player, and that
|
||
role is not moving to the Orin.
|
||
|
||
```
|
||
tablet: media (Spotify etc.) ──┐
|
||
├──▶ Android mixer ──▶ aux in
|
||
Orin: GPT-SoVITS (TTS) ──────┘
|
||
└── utterances sent over the reliable boundary channel
|
||
|
||
Orin: mic ──▶ Whisper (ASR) stays local, never crosses
|
||
```
|
||
|
||
The tablet is the media source, so it holds the aux cable whatever is decided
|
||
about speech. Giving the Orin its own aux line would not free the display unit —
|
||
it would add a second permanent cable to the car and leave the first one in place.
|
||
|
||
### TTS crosses as chunked utterances
|
||
|
||
An utterance travels as a **sequence of chunks with an explicit terminator**, over
|
||
the reliable channel. Not as one object, and not as a clock-synchronised audio
|
||
stream.
|
||
|
||
This is deliberately the shape that keeps both modes open, because whole-utterance
|
||
is simply a stream of one chunk:
|
||
|
||
| TTS capability | What it sends |
|
||
|---|---|
|
||
| Synthesises fully, then returns | one chunk, then terminator |
|
||
| Streams as it synthesises | chunks as produced, then terminator |
|
||
|
||
Same protocol, same receiver. The receiver starts playback on the first chunk,
|
||
appends as more arrive, and finishes on the terminator — which is what Android's
|
||
`AudioTrack` streaming mode does natively.
|
||
|
||
⚠ **Never ship an utterance as a single response body with a "download, then
|
||
play" player.** It works, and it forecloses streaming permanently — the receiver
|
||
has no way to begin before the last byte. The chunk boundary is what keeps both
|
||
TTS modes available.
|
||
|
||
What *is* fine early on: a receiver that buffers chunks and plays on the
|
||
terminator. The wire format is already the streaming one, so moving to playback
|
||
on the first chunk is a change to the receiver alone. Streaming playback is the
|
||
end state.
|
||
|
||
### The transport is testable with no real TTS
|
||
|
||
Audio is the lowest-priority part of the system, but its transport does not have
|
||
to wait for GPT-SoVITS. A canned WAV, or `espeak-ng` — instant, tiny, and awful —
|
||
exercises chunking, ordering, the terminator, ducking and the no-sink path
|
||
completely. Voice quality is irrelevant to every one of those.
|
||
|
||
### Ducking is the tablet's job
|
||
|
||
Android's `AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK` ducks other applications' audio —
|
||
Spotify included — for the duration of a request, then restores it. Cross-
|
||
application ducking is native on the tablet.
|
||
|
||
### The microphone stays on the Orin
|
||
|
||
Whisper runs on the Orin, so its input does too. Audio *in* on one box and audio
|
||
*out* on the other is asymmetric on purpose: each side does the simple thing, and
|
||
a microphone has no reason to reach the head unit.
|
||
|
||
The USB camera has a microphone (48 kHz stereo, ALSA card `U1080P`): a candidate
|
||
input, if its placement and quality suit speech. Untested.
|
||
|
||
The Orin runs headless with no sound server: capture reads the ALSA device
|
||
directly. If a service ever needs PipeWire, run it for that service's user
|
||
(lingering), not as a desktop session.
|
||
|
||
### Degrade when there is no sink
|
||
|
||
With no tablet connected the Orin can still hear — the mic is local — but cannot
|
||
speak. The voice stack expires or drops pending utterances rather than queueing
|
||
them unboundedly or blocking on a sink that may never arrive.
|
||
|
||
**The tablet is required for any audio at all** — it is both the media source and
|
||
the speech sink.
|