Camera in the plan: roadmap Phase 5 split into camera then inference; proposed decisions 27 (live view) and 28 (paced Orin -> tablet senders)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DDwfd6Ki8UfG4hK6W5YpJz
2026-10-11 17:21:41 +09:00
parent e6ca835992
commit a290a01f03
4 changed files with 37 additions and 8 deletions
@@ -13,4 +13,4 @@ A Jetson Orin Nano that can:
 - voice recognition with local `whisper`
 - cloud LLM-based reasoning
 - navigation capabilities using GCP APIs (or equivalent)
- vision via camera module (pending, low priority)
+- vision via a USB camera (IMX662; on the car unit, roles and inference pending)
@@ -35,7 +35,7 @@ The Orin runs headless: no local video out. The tablet app is the display.
 | `decoder` | Raw frames → named signals via DBC | No |
 | `recorder` | Persist raw + decoded sessions to disk; raw CAN and GNSS read directly, not via the bus (decision 25) | No |
 | `telemetry-gw` | Serve external display clients | No |
-| `vision` | Camera capture and on-device inference (later) | No |
+| `vision` | Camera capture (USB UVC, MJPEG), software H.264 encoding, recording and live view; on-device inference later | No |

 Each ships as a systemd unit. The host-level prerequisites they depend on
 (kernel modules, device permissions, gpsd) are the environment repo's job, not
@@ -212,7 +212,9 @@ rewrite: the seam stays visible even while the process is one.
 ## Display contract

 **Every display is data-only.** The Orin sends no video to anything. Each device
-subscribes to the bus and renders itself.
+subscribes to the bus and renders itself. The one proposed exception is the
+camera's live view (decision 27): a paced video stream beside the boundary,
+never on the bus.

 | Path | Transport | Notes |
 |---|---|---|
@@ -371,3 +373,5 @@ rewriting the document.
 | 18 | Stubs publish the real topic and schema, never a fixture-specific one | Accepted | 2026-09-03 |
 | 19 | A service mocks itself behind its own input interface; there is one implementation of each contract | Accepted | 2026-09-04 |
 | 20 | Every display is data-only. Directly-driven screens are for permanently installed, car-powered panels only | Accepted | 2026-09-04 |
+| 27 | Camera live view is the one video stream to the tablet: its own connection beside the boundary, paced to the link (≤ ~4 MB/s), never on the bus. Exception to 20 | **Proposed** | 2026-10-11 |
+| 28 | Every Orin → tablet sender caps its socket (`SO_MAX_PACING_RATE`, ~5 MB/s); unpaced bursts stall the USB link for seconds | **Proposed** | 2026-10-11 |
@@ -40,7 +40,7 @@ have taken their share.
 | Valhalla server | small; tiles are **mmap'd page cache** | evictable — see below |
 | Whisper | 273 MB (tiny) → 3.9 GB (large) | `small` is 244 M params; `large` does not fit |
 | GPT-SoVITS | upstream guidance assumes an 8 GB VRAM card | the largest unknown; **measure before designing around it** |
-| vision (Phase 5) | — | competes directly with voice for the same pool |
+| vision (Phase 5) | capture and H.264: small; inference: — | capture + software H.264 costs ~1.3–2 of 6 CPU cores at 1080p (no NVENC on the Orin Nano); inference competes directly with voice for the same pool |

 Four rules follow.

@@ -185,6 +185,9 @@ Whisper runs on the Orin, so its input does too. Audio *in* on one box and audio
 *out* on the other is asymmetric on purpose: each side does the simple thing, and
 a microphone has no reason to reach the head unit.

+The USB camera has a microphone (48 kHz stereo, ALSA card `U1080P`): a candidate
+input, if its placement and quality suit speech. Untested.
+
 The Orin runs headless with no sound server: capture reads the ALSA device
 directly. If a service ever needs PipeWire, run it for that service's user
 (lingering), not as a desktop session.
@@ -1,7 +1,7 @@
 ---
 type: plan
 status: active
-updated: 2026-10-09
+updated: 2026-10-11
 ---
 # Roadmap

@@ -24,6 +24,9 @@ Getting the Jetson to see the hardware at all.
      documented, sources kept in `~/infra`; still manual, and breaks silently
      if forgotten
 - [x] Network time pinned to NICT; timezone `Asia/Tokyo`
+- [x] USB camera (IMX662, Sunplus UVC bridge) recognised on the car unit with
+      in-tree drivers: MJPEG up to 1080p60, microphone; measured in
+      orin-nano-env-setup `docs/impl/reports/USB-CAMERA.md`
 - [ ] Optional: RTC backup battery (CR2032) — without it the clock boots stale
      until NTP or GPS corrects it
 - [ ] Power and mounting in the car: clean shutdown on ignition off
@@ -79,10 +82,29 @@ Order chosen by feedback speed, not ambition.
      car unit and a text-readout starter app; not yet run on the tablet
 - [ ] `app-esp32` — RLCD/e-ink glanceable subset. Deferred, no hardware in scope

-## Phase 5 — Camera and inference · *separate task*
+## Phase 5 — Camera, then inference

-Deliberately deferred. It is the reason the compute is a Jetson rather than a
-Pi, but it shares nothing with the CAN work beyond the bus.
+The camera is on the car unit (2026-10-11). Capturing and recording need no
+model and no GPU, so they can come before inference, which stays deferred: it is
+the reason the compute is a Jetson rather than a Pi, but it shares nothing with
+the CAN work beyond the bus, and it competes with voice for memory.
+
+What the hardware gives (USB camera report): MJPEG only, ~7.3 MB/s at 1080p30;
+no hardware video encoder on the Orin Nano, so H.264 is software, ~1.3–2 of 6
+cores at 1080p; frame rate falls with the light.
+
+- [ ] Decide the camera's roles and mounting: dash recording, live view on the
+      tablet, inference input, or several
+- [ ] Measure in the car: daylight and night frame rate and MJPEG size on a road
+      scene
+- [ ] One capture process owns `/dev/video0` (V4L2 is single-reader) and feeds
+      the others; sets UVC controls on open, since they reset on re-enumeration
+- [ ] Recording: H.264 segments on the recorder's clock and session marks,
+      bounded like CAN sessions (~2.7 GB per hour at 6 Mbit/s)
+- [ ] Live view in `app-flutter` (decision 27, proposed): 720p MJPEG passthrough
+      (~3.4 MB/s, no CPU) or H.264, paced per decision 28
+- [ ] Inference — deferred; plan for evicting voice, not for coexisting with it
+      ([ORIN-RUNTIME.md](ORIN-RUNTIME.md))

 ## Navigation · *parallel workstream*