automation/songs #1
@@ -0,0 +1,66 @@
|
||||
# pipelines/songs
|
||||
|
||||
Publishes lyric pages from `mikkeli/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
|
||||
|
||||
For the source-file contract (frontmatter and body conventions), see [SCHEMA.md](SCHEMA.md). This README covers the pipeline implementation and its runtime.
|
||||
|
||||
## v1 scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
|
||||
- Pandoc-based markdown → wikitext body rendering
|
||||
- First-line h1 stripping
|
||||
- Auto-generated banner + album category injection
|
||||
- MediaWiki bot API write with idempotency (no-op edits skipped)
|
||||
|
||||
Deferred (schema reserves the fields; renderer doesn't yet act on them):
|
||||
|
||||
- LRC parsing and embedding
|
||||
- Inter-page sibling placeholder resolution
|
||||
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
|
||||
- Auto-generated album index pages (intentionally NOT done — user writes album category-page descriptions freeform)
|
||||
|
||||
## Invocation
|
||||
|
||||
The pipeline is invoked via a reusable Gitea Actions workflow (`/.gitea/workflows/publish-songs.yml` in this repo). The `ncmr-songs` repo holds a thin stub workflow that calls into this one on push to `master`.
|
||||
|
||||
Required environment at runtime:
|
||||
|
||||
| Variable | Source | Purpose |
|
||||
|---|---|---|
|
||||
| `WIKI_API_URL` | Gitea variable | MediaWiki action API endpoint |
|
||||
| `WIKI_BASE_URL` | Gitea variable | Wiki base URL (used in the banner link) |
|
||||
| `WIKI_BOT_USER` | Gitea secret | Bot login (`Dubrowski@giteaAutomaton`) |
|
||||
| `WIKI_BOT_PASSWORD` | Gitea secret | BotPasswords value |
|
||||
| `SOURCE_DIR` | workflow input | Path to checked-out `ncmr-songs` working tree |
|
||||
| `SOURCE_REF` | workflow input | The pushed-to commit SHA |
|
||||
| `SOURCE_BASE_REF` | workflow input | The SHA prior to the push (for diff-based change detection) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Python 3.12+
|
||||
- Pandoc 2.x or newer (apt-installable in the runner container)
|
||||
- `mwclient` or `requests` for the MediaWiki API
|
||||
- `PyYAML` for frontmatter parsing
|
||||
- `python-frontmatter` (convenience wrapper around PyYAML for markdown frontmatter)
|
||||
|
||||
`requirements.txt` will be added when the implementation lands.
|
||||
|
||||
## One-off wiki setup
|
||||
|
||||
Before the first run, two things must exist on the wiki:
|
||||
|
||||
1. **`Template:Auto-generated`** — the banner injected at the top of every auto-published page. Wikitext for this template is included in the implementation step (not yet written).
|
||||
2. **The bot user has edit rights** for the namespace(s) the pipeline writes to. The default main namespace is fine; verify by attempting a manual edit via the bot account before relying on the pipeline.
|
||||
|
||||
## Modes
|
||||
|
||||
The pipeline supports two invocation modes via the entry point:
|
||||
|
||||
- **Incremental** (default, triggered by push): publishes only files changed between `SOURCE_BASE_REF` and `SOURCE_REF`.
|
||||
- **Full** (`--all` flag, triggered by `workflow_dispatch`): re-publishes every publishable file in the source tree. Use after template or renderer changes.
|
||||
|
||||
## Not yet implemented
|
||||
|
||||
This README will grow as the pipeline does. Current commit ships only the schema and this overview — no executable code yet.
|
||||
@@ -0,0 +1,177 @@
|
||||
# SCHEMA — song source files
|
||||
|
||||
Source-file contract for the songs pipeline. Files in `mikkeli/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
|
||||
|
||||
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar.
|
||||
|
||||
## File layout
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: pulse under latent semantic envelopes
|
||||
album: Shirakaba Express Service
|
||||
wiki:
|
||||
publish: true
|
||||
release_date: 2026-06-09
|
||||
---
|
||||
|
||||
# pulse under latent semantic envelopes
|
||||
|
||||
... body markdown ...
|
||||
```
|
||||
|
||||
The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc.
|
||||
|
||||
## Fields
|
||||
|
||||
### Required
|
||||
|
||||
#### `title` (string)
|
||||
|
||||
The MediaWiki page name that the bot writes or updates. MediaWiki capitalises the first letter automatically; lowercase input is fine.
|
||||
|
||||
Example: `title: pulse under latent semantic envelopes` → wiki page `Pulse under latent semantic envelopes` at URL `/wiki/Pulse_under_latent_semantic_envelopes`.
|
||||
|
||||
The body should contain a matching `# Title` first-line heading for human readability when viewing the source file directly (in an editor, on Gitea's blob view, etc.). The renderer strips this first-line heading during conversion — the wiki page already supplies its own title.
|
||||
|
||||
#### `album` (string)
|
||||
|
||||
The full album name. Drives `[[Category:<album name>]]` injection at the bottom of the rendered wiki page. MediaWiki auto-creates the category page when at least one page references it.
|
||||
|
||||
The pipeline **does not** create or update a dedicated album info/index page. Album landing pages are the wiki's category pages, which you write freeform yourself.
|
||||
|
||||
Example: `album: Shirakaba Express Service` → category tag `[[Category:Shirakaba Express Service]]`.
|
||||
|
||||
For files where no album applies (placeholders, songs without an assigned album), use a sensible literal like `(unassigned)` and accept that they'll get a `[[Category:(unassigned)]]` tag. The field is required to keep the schema's contract simple.
|
||||
|
||||
#### `wiki.publish` (boolean)
|
||||
|
||||
Publishing gate. `true` means the pipeline writes this file to the wiki. `false` means the file is ignored.
|
||||
|
||||
The `wiki` key is a mapping rather than a flat field so future toggles (`wiki.protected`, `wiki.summary_template`, etc.) can be added without restructuring. None are defined yet.
|
||||
|
||||
Files under `wip/` are also skipped, regardless of `wiki.publish`.
|
||||
|
||||
### Optional
|
||||
|
||||
#### `release_date` (ISO 8601 date)
|
||||
|
||||
The song's release date in `YYYY-MM-DD` format. Skipped if absent. Currently informational; may drive sort order on future auto-generated indexes.
|
||||
|
||||
#### `lrc` (string, relative path)
|
||||
|
||||
Pointer to a companion LRC file, relative to the `.md` file's location.
|
||||
|
||||
Example: in `ses/tunnels.md`, `lrc: tunnels.lrc` declares that `ses/tunnels.lrc` is the companion synced-lyrics file.
|
||||
|
||||
**Deferred for v1**: the field is permitted by the schema but the renderer doesn't yet read or embed LRC contents.
|
||||
|
||||
#### `siblings` (mapping of placeholder → relative path)
|
||||
|
||||
Placeholder-to-file mapping for inter-page wikilinks.
|
||||
|
||||
```yaml
|
||||
siblings:
|
||||
PULSE_DEV: pulse_dev.md
|
||||
```
|
||||
|
||||
In the body markdown, write `[[PULSE_DEV]]` for a wikilink that displays the target's title, or `[[PULSE_DEV|custom display]]` for an aliased display. Resolution uses the target file's `title:` frontmatter as the wiki page name.
|
||||
|
||||
**Deferred for v1**: the field is permitted by the schema but placeholders pass through untouched.
|
||||
|
||||
## Body
|
||||
|
||||
Plain markdown. The renderer applies a single transformation before handing the body to Pandoc: the first-line `# Heading`, if present, is stripped.
|
||||
|
||||
Do not embed wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body. Use markdown idioms only; the renderer adds the metadata-derived bits (banner, categories, sibling resolution) around Pandoc's output.
|
||||
|
||||
## Renderer behaviour
|
||||
|
||||
For each `.md` with `wiki.publish: true` and not under `wip/`:
|
||||
|
||||
1. Parse and validate frontmatter.
|
||||
2. Strip the leading `# Heading` from the body if present.
|
||||
3. Replace `[[PLACEHOLDER]]` markers with safe tokens. *(Deferred v1; no-op until siblings are implemented.)*
|
||||
4. Pipe the body through `pandoc -f markdown -t mediawiki`.
|
||||
5. Re-substitute sibling tokens with resolved `[[Page|Display]]` wikilinks. *(Deferred v1.)*
|
||||
6. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|generated_at=<ISO timestamp>}}`.
|
||||
7. Append the album category: `[[Category:<album>]]`.
|
||||
8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
|
||||
9. Otherwise, write the page with an edit summary referencing the source commit.
|
||||
|
||||
## v1 scope
|
||||
|
||||
**Implemented:**
|
||||
- Frontmatter parsing and validation for `title`, `album`, `wiki.publish`, `release_date`
|
||||
- Markdown body → wikitext via Pandoc
|
||||
- First-line h1 stripping
|
||||
- Banner template injection (requires `Template:Auto-generated` to exist on the wiki — see the pipeline `README.md` for the one-off setup)
|
||||
- Album category injection
|
||||
- Idempotent writes (no-op skip)
|
||||
|
||||
**Deferred:**
|
||||
- `lrc` field consumption
|
||||
- `siblings` placeholder resolution
|
||||
- Multi-page aggregation (no album index pages generated; each file → its own wiki page)
|
||||
- Language-aware rendering (the body conveys what languages exist; the renderer doesn't introspect)
|
||||
|
||||
## Examples
|
||||
|
||||
### A released song
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: pulse under latent semantic envelopes
|
||||
album: Shirakaba Express Service
|
||||
wiki:
|
||||
publish: true
|
||||
release_date: 2026-06-09
|
||||
---
|
||||
```
|
||||
|
||||
### A dev sister file (not published)
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: pulse under latent semantic envelopes — personal dev notes
|
||||
album: Shirakaba Express Service
|
||||
wiki:
|
||||
publish: false
|
||||
---
|
||||
```
|
||||
|
||||
### A placeholder file (no real song yet)
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: (placeholder)
|
||||
album: kairo
|
||||
wiki:
|
||||
publish: false
|
||||
---
|
||||
```
|
||||
|
||||
### A WIP file (in `wip/`, schema still applies but the file is skipped regardless)
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: 海淀
|
||||
album: (unassigned)
|
||||
wiki:
|
||||
publish: false
|
||||
---
|
||||
```
|
||||
|
||||
## Validation errors the renderer must produce
|
||||
|
||||
The pipeline fails fast and loud on the following:
|
||||
|
||||
- Missing or unparseable frontmatter block
|
||||
- Missing required field (`title`, `album`, `wiki.publish`)
|
||||
- `title` is empty or whitespace-only
|
||||
- `wiki.publish` is not a boolean
|
||||
- `release_date` (if present) does not parse as YYYY-MM-DD
|
||||
- `siblings` (if present) references a file that doesn't exist
|
||||
- Two source files declare the same `title` (page-name collision)
|
||||
|
||||
Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.
|
||||
Reference in New Issue
Block a user