yukart/svc-core
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add recorder: raw CAN and GNSS sessions with failsafes and a storage bound

Browse Source 
Reads SocketCAN and gpsd directly. Sessions start on the first frame or on
'ctl start', end after 15 s of bus idle or on 'ctl stop'; blips under 3 s are
discarded. candump -L chunks fdatasync'd every second and zstd'd once closed;
oldest sessions deleted to stay under 50 GB / above 20 GB free. Kernel and
queue drops, link changes and clock steps are recorded as events.

Bench-tested on vcan with the first car capture (byte-identical) and installed
on the car Orin as yukart-recorder.service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DDwfd6Ki8UfG4hK6W5YpJz
This commit is contained in:
Mikkeli Matlock
2026-10-09 20:15:13 +09:00
commit 1f678fe87f
19 changed files with 1784 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/GUIDELINES.md
+33
View File
@@ -0,0 +1,33 @@
---
type: guidelines
---
# Core services guidelines
## 1. Listen-only until `obd-poller` exists
- Nothing here opens a CAN socket for writing. The recorder only reads, and the
host brings `can_pcan` up listen-only.
- `obd-poller` will be the single writer. Until it lands, any transmit is a bug.
## 2. Capture never waits
- A frame missed is gone. Producers hand frames to the writer without blocking;
what doesn't fit is **counted**, recorded as an event, and never silently lost.
- Capture reads SocketCAN and gpsd directly. It does not depend on the broker, a
display, or any other service being up.
## 3. Every failure leaves complete files
- Append, flush and `fdatasync` about once a second. Power loss costs at most the
last second.
- Replace files by write-temp, fsync, rename. Never rewrite in place.
- Anything found half-done at startup is repaired or marked, not deleted.
## 4. Raw stays raw
- Recordings are `candump -L` text plus the gpsd JSON as received, so `can-utils`,
`canplayer` and anything already written against the first captures keep working.
- Decoding happens downstream of the recording, never instead of it.
## 5. Time is evidence, not truth
- The car unit's clock can boot stale and be stepped mid-drive. Record what is
needed to correct it later (sync state, monotonic time, observed steps, GNSS
time beside local receive time) rather than trusting it.
## 6. To be added
docs/impl/plans/INITIAL.md
+108
View File
@@ -0,0 +1,108 @@
---
type: plan
status: active
updated: 2026-10-09
---
# Initial plan
## Order
1. **`recorder`** — raw CAN and GNSS to disk. Built first because every later
piece is developed against what it records.
2. `obd-poller` — standard OBD-II PIDs at a low rate, the cross-check for
reverse-engineered signals. The first transmitter: `can_pcan` leaves
listen-only for it, a deliberate step.
3. `can-bridge` and `gnss` — publish onto the broker once it exists (project-yukart
Phase 3; MQTT at ~890 frames/s is still unproven).
4. `decoder` — frames to named signals via the DBC in `contracts`.
## Recorder
Reads SocketCAN and gpsd directly. It is tier 1, and the broker is a single point
of failure that raw capture should not share. When `can-bridge` exists, the
recorder still owns raw capture; it subscribes to the bus only for decoded
signals.
### Session triggers
| Trigger | Starts on | Ends on |
|---|---|---|
| auto | the first CAN frame | `--idle` (15 s) with no frames |
| manual | `ctl start [label]` | `ctl stop`, or shutdown |
- **No traffic, no session.** The PEAK attached at a desk, attached to a parked
car with the bus asleep, or unplugged, produces nothing — the socket is retried
every second in silence.
- **Cranking stays inside one session.** A bus dropout or adapter re-enumeration
shorter than the idle timeout is recorded as `can_link` events, not as a split.
- **Blips are discarded.** An auto session with under `--min-secs` (3 s) of
traffic and no marks is deleted at close — door unlocks and brief plug-ins.
- `ctl start` during an auto session promotes it to manual. After `ctl stop`, auto
start waits for the bus to go idle, so a stop sticks.
- `ctl auto off` (or `--no-auto`) leaves only manual sessions.
The power service planned in orin-nano-env-setup (`docs/impl/plans/POWER.md`)
is the intended second client: `ctl mark crank` for a recovered input loss, and
`systemctl stop yukart-recorder` before poweroff.
### Session layout
```
/var/lib/yukart/recordings/
.last-seq high-water mark; numbers are never reused
000042-20261009T163736/ sequence number orders; the local time is for humans
session.json status open → closed | truncated; trigger, label, host,
boot_id, start/end realtime + monotonic + clock sync,
frame and drop counts, end reason
can-0001.log.zst candump -L, closed chunks (5 min or 64 MB), zstd
can-0002.log the open chunk, plain
gnss.ndjson {"rx": <local receive time>, "msg": <gpsd report>}
events.ndjson marks, can_link, gnss_link, clock_step, drops, storage_bound
```
`zstdcat can-*.log.zst can-*.log | canplayer` replays a session.
### Failsafes
| Failure | Behaviour |
|---|---|
| Power loss, crash, SIGKILL | open chunk is fdatasync'd every second; at the next start the session is marked `truncated` and its chunks compressed |
| Crash mid-compression | compression writes `.zst.tmp` then renames; the plain chunk is removed only after |
| Adapter unplugged, link down | reopen every second; recorded as `can_link` events |
| gpsd down | reconnect every 3 s; CAN capture unaffected |
| Writer slower than the bus | 100 000-frame queue (~110 s); overflow counted as `queue_drops` |
| Kernel socket overflow | `SO_RXQ_OVFL` counted as `kernel_drops` |
| Disk filling | oldest closed sessions deleted to stay under `--cap-gb` (50) and above `--min-free-gb` (20); the open session is never deleted |
| Control socket fails | logged; auto capture continues |
Measured on the first capture: ~33 KB/s of candump text, ~5.5 KB/s after zstd —
**about 20 MB per hour of driving**. The 50 GB default holds years of it; the cap
is there for the failure where something else fills the disk.
### Time
The car unit has no RTC battery yet and can start with a stale clock that chrony
steps later. CAN frames carry the kernel's receive timestamp (`SO_TIMESTAMP`,
CLOCK_REALTIME), and GNSS reports carry their local receive time — **the same
clock**, so CAN and GNSS line up within a session however wrong that clock is.
To get absolute time back afterwards:
- `session.json` records the kernel's sync state at start and end.
- `clock_step` events record every step over 50 ms with its size.
- Each TPV holds GNSS UTC beside its local receive time; their difference is the
clock error, plus the SIM7600's NMEA latency (no PPS: expect tens to hundreds
of milliseconds, constant enough to calibrate out).
At 1 Hz GNSS against ~890 frames/s, alignment to a few hundred milliseconds is
what this gives. It is enough to match CAN speed against GNSS speed; it is not
enough to time individual frames against position.
## Open
- **Session marking from the app.** `app-flutter`'s mark button reaches the
recorder through `svc-gw`, which will forward to the control socket.
- **Native builds.** `scripts/deploy-recorder.sh` cross-compiles from the
workstation while the development Orin is offline; `check-glibc.sh` gates it.
- **Copy-off.** Recordings stay on the car until the parked-at-home step in the
power plan, or a manual `rsync`.
docs/impl/reports/RECORDER-BENCH.md
+37
View File
@@ -0,0 +1,37 @@
---
type: report
date: 2026-10-09
---
# Recorder bench test
Replaying the first in-car capture (`z27ag-20261009-163736-ign.log`, 10 s,
8925 frames) onto a `vcan0` with its original timing, against a fake gpsd on the
workstation and the real gpsd on the car Orin.
## Scenarios
| # | Scenario | Result |
|---|---|---|
| 1 | Full replay, `mark` at t≈3 s | auto session; 8925 frames; ID#DATA sequence **byte-identical** to the source; 3 chunks at `--chunk-secs 4`, all compressed |
| 2 | 1 s replay | discarded at close (under `--min-secs`, no marks) — after fixing the span to first→last frame; the first version counted the idle wait too and kept it |
| 3 | `ctl start bench-test`, no CAN, `mark`, `stop` | manual session with label, mark and GNSS; no CAN chunk left behind |
| 4 | `vcan0` deleted mid-session, recreated 1 s later | one session; `Network is down` → `No such device` → `socket open` recorded as `can_link` events |
| 5 | SIGTERM mid-session | session closed, reason `shutdown`, exit 0 |
| 6 | SIGKILL mid-session (3 s replay), restart | all 2676 frames on disk; marked `truncated` and compressed at restart |
| 7 | `--cap-gb 0` | every older closed session deleted, the active one kept; sequence numbers continue from `.last-seq` |
On the car Orin (aarch64 cross-build, highest GLIBC_2.34 against its 2.35): the
same full replay recorded 8925 frames and 55 KB of zstd; real gpsd reports (no
fix indoors) recorded with the last TPV before start stamped `before_start`.
Resident memory at idle: 5 MB.
## Not tested
- Real PCAN-USB hot-plug in the car (the `vcan` deletion stands in for it).
- Sustained throughput beyond 890 frames/s, and the queue-overflow path.
- A clock step during a session (the detection is in place; nothing stepped).
## Installed
`yukart-recorder.service` on `mikkeli-orin-nano`, enabled, running as `mikkeli`,
waiting for `can_pcan`. Recordings: `/var/lib/yukart/recordings/`.