diff --git a/docs/impl/reports/PERSISTENT-JOURNAL.md b/docs/impl/reports/PERSISTENT-JOURNAL.md new file mode 100644 index 0000000..5f87254 --- /dev/null +++ b/docs/impl/reports/PERSISTENT-JOURNAL.md @@ -0,0 +1,61 @@ +--- +type: report +date: 2026-10-13 +--- +# Persistent journal on the car unit + +## Why + +On 2026-10-13 the car Orin hung (unreachable over WiFi, LTE and the tablet's USB +link) during repeated failed BLE connections to the ELM327, and needed a hard power +cycle. Nothing could be said about the cause afterwards: the journal was volatile. +JetPack ships `Storage=auto` with no `/var/log/journal`, so logs lived in `/run` +and died with the boot. + +## What is set + +`/etc/systemd/journald.conf.d/50-yukart-persistent.conf`: + +```ini +[Journal] +Storage=persistent +SystemMaxUse=1G +SystemKeepFree=10G +SystemMaxFileSize=64M +SyncIntervalSec=30s +``` + +- **1 GB cap:** weeks of logs at the current volume (16 MB for the first boot's + half hour, mostly boot noise), next to a 467 GB NVMe. `SystemKeepFree` stops + the journal growing into a nearly full disk, which the recorder needs first. +- **64 MB files:** rotation drops old logs in small steps. +- **Sync every 30 s** (default 5 min): a hang or power loss costs at most the last + 30 s of ordinary messages. `crit` and worse are synced at once regardless. + +Applied by hand (install time, sudo): + +```bash +sudo mkdir -p /etc/systemd/journald.conf.d /var/log/journal +# write the file above +sudo systemd-tmpfiles --create --prefix /var/log/journal +sudo systemctl restart systemd-journald +sudo journalctl --flush +``` + +## Use + +```bash +journalctl --list-boots # one line per boot, kept across reboots +journalctl -b -1 -e # the end of the previous boot: how it ended +journalctl -b -1 -k # its kernel messages (driver crashes, USB resets) +journalctl --disk-usage +``` + +`mikkeli` reads the journal through the `adm` group; no sudo needed. + +A boot that ended cleanly ends in the shutdown sequence. One that ends mid-stream +was a hang or a power loss; the last kernel messages before the gap are the lead. +Timestamps of an early boot can be off until NTP or GPS sets the clock (no RTC +battery), so order by `-o short-monotonic` within a boot. + +Only the car unit (`mikkeli-orin-nano`) has this. The development Orin is unchanged.