---
type: report
date: 2026-10-13
---
# Persistent journal on the car unit

## Why

On 2026-10-13 the car Orin hung (unreachable over WiFi, LTE and the tablet's USB
link) during repeated failed BLE connections to the ELM327, and needed a hard power
cycle. Nothing could be said about the cause afterwards: the journal was volatile.
JetPack ships `Storage=auto` with no `/var/log/journal`, so logs lived in `/run`
and died with the boot.

## What is set

`/etc/systemd/journald.conf.d/50-yukart-persistent.conf`:

```ini
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=10G
SystemMaxFileSize=64M
SyncIntervalSec=30s
```

- **1 GB cap:** weeks of logs at the current volume (16 MB for the first boot's
  half hour, mostly boot noise), next to a 467 GB NVMe. `SystemKeepFree` stops
  the journal growing into a nearly full disk, which the recorder needs first.
- **64 MB files:** rotation drops old logs in small steps.
- **Sync every 30 s** (default 5 min): a hang or power loss costs at most the last
  30 s of ordinary messages. `crit` and worse are synced at once regardless.

Applied by hand (install time, sudo):

```bash
sudo mkdir -p /etc/systemd/journald.conf.d /var/log/journal
# write the file above
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
```

## Use

```bash
journalctl --list-boots              # one line per boot, kept across reboots
journalctl -b -1 -e                  # the end of the previous boot: how it ended
journalctl -b -1 -k                  # its kernel messages (driver crashes, USB resets)
journalctl --disk-usage
```

`mikkeli` reads the journal through the `adm` group; no sudo needed.

A boot that ended cleanly ends in the shutdown sequence. One that ends mid-stream
was a hang or a power loss; the last kernel messages before the gap are the lead.
Timestamps of an early boot can be off until NTP or GPS sets the clock (no RTC
battery), so order by `-o short-monotonic` within a boot.

### The previous boot hides behind the stale clock

Seen on the first reboot after enabling it (2026-10-13): `journalctl --list-boots`
showed only the new boot. With no RTC battery the clock starts at 2025-06-04, and
`fake-hwclock` only puts it right about 11 s in. The kernel and early-boot entries
logged before that carry 2025 times; flushed after the previous boot's 2026
entries, they make journald set the old file aside as `system@….journal~`, which
`journalctl` skips by default. Nothing is lost (that boot's log ends cleanly in
`Shutting down` / `Journal stopped`); it has to be named:

```bash
journalctl --file '/var/log/journal/*/system@*.journal~' --list-boots
journalctl --file '/var/log/journal/*/system@*.journal~' -e      # how that boot ended
```

Every boot does this until the clock survives power-off: the RTC backup battery
(project-yukart roadmap, Phase 0) is the fix. Early-boot entries of each boot also
read 2025-06-04; within a boot, `-o short-monotonic` orders them truthfully.

Only the car unit (`mikkeli-orin-nano`) has this. The development Orin is unchanged.