diff --git a/OVERVIEW.md b/OVERVIEW.md index ca80cf4..a83c394 100644 --- a/OVERVIEW.md +++ b/OVERVIEW.md @@ -13,4 +13,4 @@ A Jetson Orin Nano that can: - voice recognition with local `whisper` - cloud LLM-based reasoning - navigation capabilities using GCP APIs (or equivalent) -- vision via camera module (pending, low priority) \ No newline at end of file +- vision via a USB camera (IMX662; on the car unit, roles and inference pending) \ No newline at end of file diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 31b1a4c..f45732c 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -35,7 +35,7 @@ The Orin runs headless: no local video out. The tablet app is the display. | `decoder` | Raw frames → named signals via DBC | No | | `recorder` | Persist raw + decoded sessions to disk; raw CAN and GNSS read directly, not via the bus (decision 25) | No | | `telemetry-gw` | Serve external display clients | No | -| `vision` | Camera capture and on-device inference (later) | No | +| `vision` | Camera capture (USB UVC, MJPEG), software H.264 encoding, recording and live view; on-device inference later | No | Each ships as a systemd unit. The host-level prerequisites they depend on (kernel modules, device permissions, gpsd) are the environment repo's job, not @@ -212,7 +212,9 @@ rewrite: the seam stays visible even while the process is one. ## Display contract **Every display is data-only.** The Orin sends no video to anything. Each device -subscribes to the bus and renders itself. +subscribes to the bus and renders itself. The one proposed exception is the +camera's live view (decision 27): a paced video stream beside the boundary, +never on the bus. | Path | Transport | Notes | |---|---|---| @@ -371,3 +373,5 @@ rewriting the document. | 18 | Stubs publish the real topic and schema, never a fixture-specific one | Accepted | 2026-09-03 | | 19 | A service mocks itself behind its own input interface; there is one implementation of each contract | Accepted | 2026-09-04 | | 20 | Every display is data-only. Directly-driven screens are for permanently installed, car-powered panels only | Accepted | 2026-09-04 | +| 27 | Camera live view is the one video stream to the tablet: its own connection beside the boundary, paced to the link (≤ ~4 MB/s), never on the bus. Exception to 20 | **Proposed** | 2026-10-11 | +| 28 | Every Orin → tablet sender caps its socket (`SO_MAX_PACING_RATE`, ~5 MB/s); unpaced bursts stall the USB link for seconds | **Proposed** | 2026-10-11 | diff --git a/docs/ORIN-RUNTIME.md b/docs/ORIN-RUNTIME.md index dbe8767..9e012dc 100644 --- a/docs/ORIN-RUNTIME.md +++ b/docs/ORIN-RUNTIME.md @@ -40,7 +40,7 @@ have taken their share. | Valhalla server | small; tiles are **mmap'd page cache** | evictable — see below | | Whisper | 273 MB (tiny) → 3.9 GB (large) | `small` is 244 M params; `large` does not fit | | GPT-SoVITS | upstream guidance assumes an 8 GB VRAM card | the largest unknown; **measure before designing around it** | -| vision (Phase 5) | — | competes directly with voice for the same pool | +| vision (Phase 5) | capture and H.264: small; inference: — | capture + software H.264 costs ~1.3–2 of 6 CPU cores at 1080p (no NVENC on the Orin Nano); inference competes directly with voice for the same pool | Four rules follow. @@ -185,6 +185,9 @@ Whisper runs on the Orin, so its input does too. Audio *in* on one box and audio *out* on the other is asymmetric on purpose: each side does the simple thing, and a microphone has no reason to reach the head unit. +The USB camera has a microphone (48 kHz stereo, ALSA card `U1080P`): a candidate +input, if its placement and quality suit speech. Untested. + The Orin runs headless with no sound server: capture reads the ALSA device directly. If a service ever needs PipeWire, run it for that service's user (lingering), not as a desktop session. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 450b465..33229bb 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,7 +1,7 @@ --- type: plan status: active -updated: 2026-10-09 +updated: 2026-10-11 --- # Roadmap @@ -24,6 +24,9 @@ Getting the Jetson to see the hardware at all. documented, sources kept in `~/infra`; still manual, and breaks silently if forgotten - [x] Network time pinned to NICT; timezone `Asia/Tokyo` +- [x] USB camera (IMX662, Sunplus UVC bridge) recognised on the car unit with + in-tree drivers: MJPEG up to 1080p60, microphone; measured in + orin-nano-env-setup `docs/impl/reports/USB-CAMERA.md` - [ ] Optional: RTC backup battery (CR2032) — without it the clock boots stale until NTP or GPS corrects it - [ ] Power and mounting in the car: clean shutdown on ignition off @@ -79,10 +82,29 @@ Order chosen by feedback speed, not ambition. car unit and a text-readout starter app; not yet run on the tablet - [ ] `app-esp32` — RLCD/e-ink glanceable subset. Deferred, no hardware in scope -## Phase 5 — Camera and inference · *separate task* +## Phase 5 — Camera, then inference -Deliberately deferred. It is the reason the compute is a Jetson rather than a -Pi, but it shares nothing with the CAN work beyond the bus. +The camera is on the car unit (2026-10-11). Capturing and recording need no +model and no GPU, so they can come before inference, which stays deferred: it is +the reason the compute is a Jetson rather than a Pi, but it shares nothing with +the CAN work beyond the bus, and it competes with voice for memory. + +What the hardware gives (USB camera report): MJPEG only, ~7.3 MB/s at 1080p30; +no hardware video encoder on the Orin Nano, so H.264 is software, ~1.3–2 of 6 +cores at 1080p; frame rate falls with the light. + +- [ ] Decide the camera's roles and mounting: dash recording, live view on the + tablet, inference input, or several +- [ ] Measure in the car: daylight and night frame rate and MJPEG size on a road + scene +- [ ] One capture process owns `/dev/video0` (V4L2 is single-reader) and feeds + the others; sets UVC controls on open, since they reset on re-enumeration +- [ ] Recording: H.264 segments on the recorder's clock and session marks, + bounded like CAN sessions (~2.7 GB per hour at 6 Mbit/s) +- [ ] Live view in `app-flutter` (decision 27, proposed): 720p MJPEG passthrough + (~3.4 MB/s, no CPU) or H.264, paced per decision 28 +- [ ] Inference — deferred; plan for evicting voice, not for coexisting with it + ([ORIN-RUNTIME.md](ORIN-RUNTIME.md)) ## Navigation · *parallel workstream*