irene
2c6e227c82
`feat/bom-generator` carried one commit that never landed on master: file
content hashes in the BOM ("v2 with hashes"). The Python rewrite dropped that
capability, so port it forward rather than losing it when the branch goes.
- `--hash` records a short BLAKE2b digest per file. Opt-in, because it reads
every asset in full and that is wasted work for a plain inventory.
- Hashes are streamed in chunks with `hashlib.blake2b`, not `md5` via haveged
imports, and not whole-file reads — these assets run to hundreds of megabytes.
- YAML gains `hash:` after `size:`; Markdown gains a hash column only when
hashes were requested, so the common table stays readable.
Fix a pre-existing bug found while testing: the BOM marker was written as
`; generated by ...`, but `;` is not a YAML comment, so every generated BOM
failed to parse with "mapping values are not allowed here". Now `#`. Verified by
re-parsing output from both the PyYAML path and the built-in fallback writer.
Sizes alone cannot tell you whether an asset changed; a same-length re-render
keeps the byte count identical. That is the gap the hash closes.
218 lines
7.7 KiB
Markdown
218 lines
7.7 KiB
Markdown
# NCMR songs-videos — tools
|
|
|
|
Helper scripts for building NCMR song videos: converting lyrics to subtitles,
|
|
burning them in, muxing audio, and working out what this machine can encode.
|
|
|
|
Everything here is **Python standard library only**. No virtualenv, no install
|
|
step, no dependencies to break in six months. Run them straight from the repo:
|
|
|
|
```
|
|
python3 scripts/tools.py --help
|
|
```
|
|
|
|
---
|
|
|
|
## Quick start
|
|
|
|
```sh
|
|
cd ~/dev/mikkeli/ncmr-songs-videos
|
|
|
|
# 1. What can this machine actually encode?
|
|
python3 scripts/tools.py hwprobe
|
|
|
|
# 2. Lyrics -> subtitles
|
|
python3 scripts/tools.py lrc2srt song.lrc -o song.srt
|
|
python3 scripts/tools.py srt2ass song.srt -o song.ass
|
|
|
|
# 3. Audio + still image -> a video (the "PV" case)
|
|
python3 scripts/tools.py ffmpeg still cover.png song.flac -o pv.mkv
|
|
|
|
# 4. Burn the subtitles in, copy the audio through untouched
|
|
python3 scripts/tools.py ffmpeg bake pv.mkv song.ass -o final.mp4 --format flac
|
|
|
|
# 5. Record what untracked assets went in
|
|
python3 scripts/tools.py bom --dir .
|
|
```
|
|
|
|
Every subcommand takes `--help`. Every subcommand accepts `--dry-run` (`-n`) to
|
|
print the work it would do without doing it, which is the fastest way to see
|
|
what ffmpeg command you are about to run.
|
|
|
|
---
|
|
|
|
## The tools
|
|
|
|
| Tool | Purpose |
|
|
| --- | --- |
|
|
| `tools.py` | Dispatcher. `python3 scripts/tools.py <tool> [args...]`. |
|
|
| `hwprobe.py` | Report which encoders this machine can really use, and why not. |
|
|
| `lrc2srt.py` | LRC lyrics → SRT subtitles. |
|
|
| `srt2ass.py` | SRT → ASS, for burn-in via ffmpeg's `ass` filter. |
|
|
| `ffmpeg_tools.py` | `mux`, `bake`, `still`, `strip`, `probe` — the ffmpeg wrappers. |
|
|
| `bom_gen.py` | Write a `BOM.yaml` (or `BOM.md`) of a song's untracked assets. |
|
|
| `_common.py` | Shared helpers. Not a command; imported by the rest. |
|
|
|
|
Each script also runs standalone (`python3 scripts/hwprobe.py`) and can be
|
|
imported as a module. The dispatcher is a convenience, not a requirement.
|
|
|
|
---
|
|
|
|
## Hardware encoding
|
|
|
|
This machine is an **AMD Strix Halo** (Radeon 8060S). There is no NVIDIA card,
|
|
so `nvenc` is unavailable here. What does work is **VA-API** via the Mesa
|
|
`radeonsi` driver:
|
|
|
|
| Codec | Encoder | Status |
|
|
| --- | --- | --- |
|
|
| H.264 | `h264_vaapi` | works |
|
|
| HEVC | `hevc_vaapi` | works |
|
|
| AV1 | `av1_vaapi` | works |
|
|
| H.264/HEVC | `libx264`/`libx265` | software fallback, always available |
|
|
|
|
`ffmpeg_tools.py` picks a hardware encoder automatically when one is usable and
|
|
falls back to `libx264` when it is not, so the same command works on a machine
|
|
without a usable GPU. Override it when you want to:
|
|
|
|
```sh
|
|
--video auto # default: hardware if possible, else software
|
|
--video x264 # force software H.264
|
|
--video x265 # force software HEVC
|
|
--video copy # no re-encode at all
|
|
--codec hevc # which hardware codec family to try with --video auto
|
|
--device PATH # VA-API render node (default /dev/dri/renderD128)
|
|
```
|
|
|
|
### If hardware encoding stops working
|
|
|
|
The usual cause is permissions on the DRM render node. `/dev/dri/renderD128` is
|
|
owned by `root:render`, so the user needs to be in the `render` group:
|
|
|
|
```sh
|
|
getent group render # who is in it
|
|
python3 scripts/tools.py hwprobe # says plainly whether the device opens
|
|
```
|
|
|
|
Group membership only applies to *new* logins. If `id` shows `render` but
|
|
`hwprobe` still reports the device as unreadable, the running process is holding
|
|
stale credentials — log out and back in, or start the process under the group
|
|
explicitly:
|
|
|
|
```sh
|
|
sg render -c 'python3 scripts/hwprobe.py'
|
|
```
|
|
|
|
`hwprobe` never guesses. It opens the device, runs a two-frame test encode
|
|
through each candidate encoder, and reports what actually worked.
|
|
|
|
```sh
|
|
python3 scripts/tools.py hwprobe --shell # export-friendly KEY=value output
|
|
```
|
|
|
|
---
|
|
|
|
## Subtitles
|
|
|
|
LRC is the timestamped-lyrics format lyric sites emit; SRT is what most players
|
|
read; ASS is what you need to burn styled text into a video with ffmpeg.
|
|
|
|
```sh
|
|
python3 scripts/tools.py lrc2srt song.lrc -o song.srt --shift -0.5
|
|
```
|
|
|
|
`lrc2srt` handles the messy real-world cases: `[mm:ss]`, `[mm:ss.xx]` and
|
|
`[mm:ss.xxx]` timestamps, several timestamps on one line (repeated choruses —
|
|
each becomes its own cue), `[ar:]`/`[ti:]` metadata tags, and blank lines inside
|
|
a cue. The final cue is held for a few seconds rather than dropped; adjust with
|
|
`--tail`. Use `--shift` to nudge everything earlier or later if the lyrics drift
|
|
against the audio.
|
|
|
|
```sh
|
|
python3 scripts/tools.py srt2ass song.srt -o song.ass --play-res 1920x1080
|
|
```
|
|
|
|
`srt2ass` converts to ASS and gives ffmpeg something it can burn in. `--play-res`
|
|
sets the script resolution used for font scaling; match it to your video so text
|
|
sizes come out as designed. `--dry-run` prints the ASS to stdout for inspection.
|
|
|
|
---
|
|
|
|
## ffmpeg wrappers
|
|
|
|
Long ffmpeg command lines are easy to get subtly wrong and impossible to
|
|
remember. These wrap the five the project actually needs.
|
|
|
|
```sh
|
|
# Attach audio to a video-only file; video stream copied, never re-encoded.
|
|
python3 scripts/tools.py ffmpeg mux render.mp4 song.flac -o pv.mkv --format flac
|
|
|
|
# Burn subtitles into a video.
|
|
python3 scripts/tools.py ffmpeg bake pv.mkv song.ass -o final.mp4 --format flac
|
|
|
|
# One still image + audio -> a video.
|
|
python3 scripts/tools.py ffmpeg still cover.png song.flac -o pv.mkv --fps 60
|
|
|
|
# Pull the audio back out as MP3.
|
|
python3 scripts/tools.py ffmpeg strip final.mp4 -o final.mp3
|
|
|
|
# What is actually in this file?
|
|
python3 scripts/tools.py ffmpeg probe final.mp4
|
|
```
|
|
|
|
Notes on the choices baked in:
|
|
|
|
- **`mux` uses `-c:v copy`.** The Remotion render is already encoded; re-encoding
|
|
it to add audio costs quality and time for nothing. Containers are picked from
|
|
the output extension.
|
|
- **`--format` presets** are `aac`, `mp3`, `flac`, `opus` and `copy`. `flac` is
|
|
the default because these are archival masters; use `aac`/`opus` for anything
|
|
you are actually shipping. `--bitrate` applies to the lossy ones.
|
|
- **`-shortest`** is set where a still image is looped, so the video ends with
|
|
the audio instead of running on for ever.
|
|
- **`-movflags +faststart`** is set for MP4 output, so the file starts playing
|
|
before it has fully downloaded.
|
|
|
|
For anything these do not cover, use `--dry-run` to print the generated command,
|
|
copy it, and edit it — the wrapper is a convenience, not a cage. `hwprobe.py`
|
|
reports what this host can actually encode, so you can tell at a glance whether a
|
|
recipe will need a hardware-specific encoder.
|
|
|
|
---
|
|
|
|
## Bill of materials
|
|
|
|
Binary assets are git-ignored, so a clone does not tell you what went into a
|
|
song. `bom_gen.py` walks the asset directories and records it:
|
|
|
|
```sh
|
|
python3 scripts/tools.py bom --dir . --format yaml
|
|
```
|
|
|
|
It writes `BOM.yaml` listing every untracked file under `audio/`, `images/`,
|
|
`fonts/` and `videos/` with its size, plus whatever description you give it. Use
|
|
`--format md` for Markdown, `--format json` for machine-readable output, and
|
|
`-y` to skip the interactive description prompt. If PyYAML is available it is
|
|
used for clean output; otherwise a small built-in writer produces the same
|
|
schema. Neither is required.
|
|
|
|
Add `--hash` to record a short content hash per file:
|
|
|
|
```sh
|
|
python3 scripts/tools.py bom --dir . --hash
|
|
```
|
|
|
|
That is how you answer "has this asset changed since the BOM was written?" —
|
|
sizes stay identical when you re-render a take at the same length, but the hash
|
|
will not. Hashing reads every asset in full, so it is off by default; when you
|
|
only want the inventory, leave it out.
|
|
|
|
---
|
|
|
|
## Conventions
|
|
|
|
- `master` is the shared template. It is intended to freeze once the workspace is
|
|
finalised; until then, tooling work can land on it directly. Per-song work goes
|
|
on `<song>/master`, and features on `<song>/<feature>`. See `README.md`.
|
|
- Binary assets stay out of git; the BOM is how you know what was used.
|
|
- Keep new tools in `scripts/`, stdlib only, with a `--dry-run` and `--help`.
|