mikkeli
47ba62bb17
/ crosscheck (push) Successful in 1s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DDwfd6Ki8UfG4hK6W5YpJz
81 lines
3.0 KiB
Markdown
81 lines
3.0 KiB
Markdown
---
|
|
type: report
|
|
date: 2026-10-13
|
|
---
|
|
# Persistent journal on the car unit
|
|
|
|
## Why
|
|
|
|
On 2026-10-13 the car Orin hung (unreachable over WiFi, LTE and the tablet's USB
|
|
link) during repeated failed BLE connections to the ELM327, and needed a hard power
|
|
cycle. Nothing could be said about the cause afterwards: the journal was volatile.
|
|
JetPack ships `Storage=auto` with no `/var/log/journal`, so logs lived in `/run`
|
|
and died with the boot.
|
|
|
|
## What is set
|
|
|
|
`/etc/systemd/journald.conf.d/50-yukart-persistent.conf`:
|
|
|
|
```ini
|
|
[Journal]
|
|
Storage=persistent
|
|
SystemMaxUse=1G
|
|
SystemKeepFree=10G
|
|
SystemMaxFileSize=64M
|
|
SyncIntervalSec=30s
|
|
```
|
|
|
|
- **1 GB cap:** weeks of logs at the current volume (16 MB for the first boot's
|
|
half hour, mostly boot noise), next to a 467 GB NVMe. `SystemKeepFree` stops
|
|
the journal growing into a nearly full disk, which the recorder needs first.
|
|
- **64 MB files:** rotation drops old logs in small steps.
|
|
- **Sync every 30 s** (default 5 min): a hang or power loss costs at most the last
|
|
30 s of ordinary messages. `crit` and worse are synced at once regardless.
|
|
|
|
Applied by hand (install time, sudo):
|
|
|
|
```bash
|
|
sudo mkdir -p /etc/systemd/journald.conf.d /var/log/journal
|
|
# write the file above
|
|
sudo systemd-tmpfiles --create --prefix /var/log/journal
|
|
sudo systemctl restart systemd-journald
|
|
sudo journalctl --flush
|
|
```
|
|
|
|
## Use
|
|
|
|
```bash
|
|
journalctl --list-boots # one line per boot, kept across reboots
|
|
journalctl -b -1 -e # the end of the previous boot: how it ended
|
|
journalctl -b -1 -k # its kernel messages (driver crashes, USB resets)
|
|
journalctl --disk-usage
|
|
```
|
|
|
|
`mikkeli` reads the journal through the `adm` group; no sudo needed.
|
|
|
|
A boot that ended cleanly ends in the shutdown sequence. One that ends mid-stream
|
|
was a hang or a power loss; the last kernel messages before the gap are the lead.
|
|
Timestamps of an early boot can be off until NTP or GPS sets the clock (no RTC
|
|
battery), so order by `-o short-monotonic` within a boot.
|
|
|
|
### The previous boot hides behind the stale clock
|
|
|
|
Seen on the first reboot after enabling it (2026-10-13): `journalctl --list-boots`
|
|
showed only the new boot. With no RTC battery the clock starts at 2025-06-04, and
|
|
`fake-hwclock` only puts it right about 11 s in. The kernel and early-boot entries
|
|
logged before that carry 2025 times; flushed after the previous boot's 2026
|
|
entries, they make journald set the old file aside as `system@….journal~`, which
|
|
`journalctl` skips by default. Nothing is lost (that boot's log ends cleanly in
|
|
`Shutting down` / `Journal stopped`); it has to be named:
|
|
|
|
```bash
|
|
journalctl --file '/var/log/journal/*/system@*.journal~' --list-boots
|
|
journalctl --file '/var/log/journal/*/system@*.journal~' -e # how that boot ended
|
|
```
|
|
|
|
Every boot does this until the clock survives power-off: the RTC backup battery
|
|
(project-yukart roadmap, Phase 0) is the fix. Early-boot entries of each boot also
|
|
read 2025-06-04; within a boot, `-o short-monotonic` orders them truthfully.
|
|
|
|
Only the car unit (`mikkeli-orin-nano`) has this. The development Orin is unchanged.
|