diff --git a/pipelines/songs/README.md b/pipelines/songs/README.md new file mode 100644 index 0000000..5de76b3 --- /dev/null +++ b/pipelines/songs/README.md @@ -0,0 +1,66 @@ +# pipelines/songs + +Publishes lyric pages from `mikkeli/ncmr-songs` to `https://wiki.novoyuuparosk.org`. + +For the source-file contract (frontmatter and body conventions), see [SCHEMA.md](SCHEMA.md). This README covers the pipeline implementation and its runtime. + +## v1 scope + +Implemented: + +- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md) +- Pandoc-based markdown → wikitext body rendering +- First-line h1 stripping +- Auto-generated banner + album category injection +- MediaWiki bot API write with idempotency (no-op edits skipped) + +Deferred (schema reserves the fields; renderer doesn't yet act on them): + +- LRC parsing and embedding +- Inter-page sibling placeholder resolution +- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect) +- Auto-generated album index pages (intentionally NOT done — user writes album category-page descriptions freeform) + +## Invocation + +The pipeline is invoked via a reusable Gitea Actions workflow (`/.gitea/workflows/publish-songs.yml` in this repo). The `ncmr-songs` repo holds a thin stub workflow that calls into this one on push to `master`. + +Required environment at runtime: + +| Variable | Source | Purpose | +|---|---|---| +| `WIKI_API_URL` | Gitea variable | MediaWiki action API endpoint | +| `WIKI_BASE_URL` | Gitea variable | Wiki base URL (used in the banner link) | +| `WIKI_BOT_USER` | Gitea secret | Bot login (`Dubrowski@giteaAutomaton`) | +| `WIKI_BOT_PASSWORD` | Gitea secret | BotPasswords value | +| `SOURCE_DIR` | workflow input | Path to checked-out `ncmr-songs` working tree | +| `SOURCE_REF` | workflow input | The pushed-to commit SHA | +| `SOURCE_BASE_REF` | workflow input | The SHA prior to the push (for diff-based change detection) | + +## Dependencies + +- Python 3.12+ +- Pandoc 2.x or newer (apt-installable in the runner container) +- `mwclient` or `requests` for the MediaWiki API +- `PyYAML` for frontmatter parsing +- `python-frontmatter` (convenience wrapper around PyYAML for markdown frontmatter) + +`requirements.txt` will be added when the implementation lands. + +## One-off wiki setup + +Before the first run, two things must exist on the wiki: + +1. **`Template:Auto-generated`** — the banner injected at the top of every auto-published page. Wikitext for this template is included in the implementation step (not yet written). +2. **The bot user has edit rights** for the namespace(s) the pipeline writes to. The default main namespace is fine; verify by attempting a manual edit via the bot account before relying on the pipeline. + +## Modes + +The pipeline supports two invocation modes via the entry point: + +- **Incremental** (default, triggered by push): publishes only files changed between `SOURCE_BASE_REF` and `SOURCE_REF`. +- **Full** (`--all` flag, triggered by `workflow_dispatch`): re-publishes every publishable file in the source tree. Use after template or renderer changes. + +## Not yet implemented + +This README will grow as the pipeline does. Current commit ships only the schema and this overview — no executable code yet. diff --git a/pipelines/songs/SCHEMA.md b/pipelines/songs/SCHEMA.md new file mode 100644 index 0000000..dd4d84d --- /dev/null +++ b/pipelines/songs/SCHEMA.md @@ -0,0 +1,177 @@ +# SCHEMA — song source files + +Source-file contract for the songs pipeline. Files in `mikkeli/ncmr-songs` must follow this contract to be picked up by the auto-publisher. + +The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar. + +## File layout + +```markdown +--- +title: pulse under latent semantic envelopes +album: Shirakaba Express Service +wiki: + publish: true +release_date: 2026-06-09 +--- + +# pulse under latent semantic envelopes + +... body markdown ... +``` + +The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc. + +## Fields + +### Required + +#### `title` (string) + +The MediaWiki page name that the bot writes or updates. MediaWiki capitalises the first letter automatically; lowercase input is fine. + +Example: `title: pulse under latent semantic envelopes` → wiki page `Pulse under latent semantic envelopes` at URL `/wiki/Pulse_under_latent_semantic_envelopes`. + +The body should contain a matching `# Title` first-line heading for human readability when viewing the source file directly (in an editor, on Gitea's blob view, etc.). The renderer strips this first-line heading during conversion — the wiki page already supplies its own title. + +#### `album` (string) + +The full album name. Drives `[[Category:]]` injection at the bottom of the rendered wiki page. MediaWiki auto-creates the category page when at least one page references it. + +The pipeline **does not** create or update a dedicated album info/index page. Album landing pages are the wiki's category pages, which you write freeform yourself. + +Example: `album: Shirakaba Express Service` → category tag `[[Category:Shirakaba Express Service]]`. + +For files where no album applies (placeholders, songs without an assigned album), use a sensible literal like `(unassigned)` and accept that they'll get a `[[Category:(unassigned)]]` tag. The field is required to keep the schema's contract simple. + +#### `wiki.publish` (boolean) + +Publishing gate. `true` means the pipeline writes this file to the wiki. `false` means the file is ignored. + +The `wiki` key is a mapping rather than a flat field so future toggles (`wiki.protected`, `wiki.summary_template`, etc.) can be added without restructuring. None are defined yet. + +Files under `wip/` are also skipped, regardless of `wiki.publish`. + +### Optional + +#### `release_date` (ISO 8601 date) + +The song's release date in `YYYY-MM-DD` format. Skipped if absent. Currently informational; may drive sort order on future auto-generated indexes. + +#### `lrc` (string, relative path) + +Pointer to a companion LRC file, relative to the `.md` file's location. + +Example: in `ses/tunnels.md`, `lrc: tunnels.lrc` declares that `ses/tunnels.lrc` is the companion synced-lyrics file. + +**Deferred for v1**: the field is permitted by the schema but the renderer doesn't yet read or embed LRC contents. + +#### `siblings` (mapping of placeholder → relative path) + +Placeholder-to-file mapping for inter-page wikilinks. + +```yaml +siblings: + PULSE_DEV: pulse_dev.md +``` + +In the body markdown, write `[[PULSE_DEV]]` for a wikilink that displays the target's title, or `[[PULSE_DEV|custom display]]` for an aliased display. Resolution uses the target file's `title:` frontmatter as the wiki page name. + +**Deferred for v1**: the field is permitted by the schema but placeholders pass through untouched. + +## Body + +Plain markdown. The renderer applies a single transformation before handing the body to Pandoc: the first-line `# Heading`, if present, is stripped. + +Do not embed wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body. Use markdown idioms only; the renderer adds the metadata-derived bits (banner, categories, sibling resolution) around Pandoc's output. + +## Renderer behaviour + +For each `.md` with `wiki.publish: true` and not under `wip/`: + +1. Parse and validate frontmatter. +2. Strip the leading `# Heading` from the body if present. +3. Replace `[[PLACEHOLDER]]` markers with safe tokens. *(Deferred v1; no-op until siblings are implemented.)* +4. Pipe the body through `pandoc -f markdown -t mediawiki`. +5. Re-substitute sibling tokens with resolved `[[Page|Display]]` wikilinks. *(Deferred v1.)* +6. Prepend the auto-generated banner: `{{Auto-generated|source=|generated_at=}}`. +7. Append the album category: `[[Category:]]`. +8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean). +9. Otherwise, write the page with an edit summary referencing the source commit. + +## v1 scope + +**Implemented:** +- Frontmatter parsing and validation for `title`, `album`, `wiki.publish`, `release_date` +- Markdown body → wikitext via Pandoc +- First-line h1 stripping +- Banner template injection (requires `Template:Auto-generated` to exist on the wiki — see the pipeline `README.md` for the one-off setup) +- Album category injection +- Idempotent writes (no-op skip) + +**Deferred:** +- `lrc` field consumption +- `siblings` placeholder resolution +- Multi-page aggregation (no album index pages generated; each file → its own wiki page) +- Language-aware rendering (the body conveys what languages exist; the renderer doesn't introspect) + +## Examples + +### A released song + +```yaml +--- +title: pulse under latent semantic envelopes +album: Shirakaba Express Service +wiki: + publish: true +release_date: 2026-06-09 +--- +``` + +### A dev sister file (not published) + +```yaml +--- +title: pulse under latent semantic envelopes — personal dev notes +album: Shirakaba Express Service +wiki: + publish: false +--- +``` + +### A placeholder file (no real song yet) + +```yaml +--- +title: (placeholder) +album: kairo +wiki: + publish: false +--- +``` + +### A WIP file (in `wip/`, schema still applies but the file is skipped regardless) + +```yaml +--- +title: 海淀 +album: (unassigned) +wiki: + publish: false +--- +``` + +## Validation errors the renderer must produce + +The pipeline fails fast and loud on the following: + +- Missing or unparseable frontmatter block +- Missing required field (`title`, `album`, `wiki.publish`) +- `title` is empty or whitespace-only +- `wiki.publish` is not a boolean +- `release_date` (if present) does not parse as YYYY-MM-DD +- `siblings` (if present) references a file that doesn't exist +- Two source files declare the same `title` (page-name collision) + +Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.