orin-nano-env-setup/docs/impl/reports/PERSISTENT-JOURNAL.md

type, date

type	date
report	2026-10-13

Persistent journal on the car unit

Why

On 2026-10-13 the car Orin hung (unreachable over WiFi, LTE and the tablet's USB link) during repeated failed BLE connections to the ELM327, and needed a hard power cycle. Nothing could be said about the cause afterwards: the journal was volatile. JetPack ships Storage=auto with no /var/log/journal, so logs lived in /run and died with the boot.

What is set

/etc/systemd/journald.conf.d/50-yukart-persistent.conf:

[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=10G
SystemMaxFileSize=64M
SyncIntervalSec=30s

• 1 GB cap: weeks of logs at the current volume (16 MB for the first boot's half hour, mostly boot noise), next to a 467 GB NVMe. SystemKeepFree stops the journal growing into a nearly full disk, which the recorder needs first.
• 64 MB files: rotation drops old logs in small steps.
• Sync every 30 s (default 5 min): a hang or power loss costs at most the last 30 s of ordinary messages. crit and worse are synced at once regardless.

Applied by hand (install time, sudo):

sudo mkdir -p /etc/systemd/journald.conf.d /var/log/journal
# write the file above
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush

Use

journalctl --list-boots              # one line per boot, kept across reboots
journalctl -b -1 -e                  # the end of the previous boot: how it ended
journalctl -b -1 -k                  # its kernel messages (driver crashes, USB resets)
journalctl --disk-usage

mikkeli reads the journal through the adm group; no sudo needed.

A boot that ended cleanly ends in the shutdown sequence. One that ends mid-stream was a hang or a power loss; the last kernel messages before the gap are the lead. Timestamps of an early boot can be off until NTP or GPS sets the clock (no RTC battery), so order by -o short-monotonic within a boot.

The previous boot hides behind the stale clock

Seen on the first reboot after enabling it (2026-10-13): journalctl --list-boots showed only the new boot. With no RTC battery the clock starts at 2025-06-04, and fake-hwclock only puts it right about 11 s in. The kernel and early-boot entries logged before that carry 2025 times; flushed after the previous boot's 2026 entries, they make journald set the old file aside as system@….journal~, which journalctl skips by default. Nothing is lost (that boot's log ends cleanly in Shutting down / Journal stopped); it has to be named:

journalctl --file '/var/log/journal/*/system@*.journal~' --list-boots
journalctl --file '/var/log/journal/*/system@*.journal~' -e      # how that boot ended

Every boot does this until the clock survives power-off: the RTC backup battery (project-yukart roadmap, Phase 0) is the fix. Early-boot entries of each boot also read 2025-06-04; within a boot, -o short-monotonic orders them truthfully.

Only the car unit (mikkeli-orin-nano) has this. The development Orin is unchanged.