wan
3ab59b493e
tests / pytest (push) Successful in 6s
Removes the decision logs, the dated rows, the strikethrough retirements and the setup progress checkboxes from all four READMEs. A decision log records how the system came to be; a reader needs to know how it is. The two had drifted far enough apart that the docs asserted things the shipped config contradicted. Corrected, all of which were stale rather than merely verbose: - the architecture section described the source-repo workflow as self-contained and called the reusable workflows here "not actively called; may be stale". They are the pipelines; the source-repo workflow is a thin caller. - `workflow_call` was recorded as abandoned, and the instance as having no orgs. Both pipelines run through cross-repo `workflow_call` under an org. - tech-blogs was described as not yet initialised. It has published. - ses and tech invocation sections claimed a self-contained workflow too. Kept, rewritten as behaviour rather than as dated decisions, because they are live operational constraints and not history: - the callee workflow cache, why a `publish-*.yml` edit appears not to take, and the flush command with its `--no-deps` and `--pull` caveats - why this repo has to stay public - the `URL_TO_GITEA` / `PUBLIC_URL_TO_GITEA` split and what validates it - host network mode, and that `kaniko-act` cannot be rebuilt by build-image.yml Also documents what the renderer, its failure modes and its publish scope actually are, which the decision logs had been standing in for, and adds the architecture note that renderer code is cloned fresh per run while workflow YAML is cached — the distinction that governs whether a change takes effect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
87 lines
6.0 KiB
Markdown
87 lines
6.0 KiB
Markdown
# pipelines/songs
|
|
|
|
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`. v1, live.
|
|
|
|
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
|
|
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
|
|
|
|
## What the renderer does
|
|
|
|
- Parses YAML frontmatter per [SCHEMA.md](SCHEMA.md); files without `wiki.publish: true` are skipped silently
|
|
- Strips the body's first-line `# Heading`, then renders markdown → wikitext with Pandoc
|
|
- Expands ` ```columns ` fenced blocks into a flex row of `<poem>` columns (shared transform in `lib/wiki.py`)
|
|
- Resolves relative Markdown links to sibling `.md` files into `[[wiki page title|label]]`, using the target's own `title` frontmatter as the page name
|
|
- Prepends the `{{Auto-generated|source=…|commit=<sha>}}` banner
|
|
- Uploads a declared `lrc` companion file when its SHA1 has changed, and appends a `[[Media:…]]` footer link
|
|
- Injects `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set, plus one `[[Category:…]]` per `categories` entry. Category pages themselves are user-written and never overwritten
|
|
- Writes via the MediaWiki bot API, skipping the write when the generated text already matches the page
|
|
|
|
Deliberately not in scope: multi-language metadata (the body conveys what languages exist; the renderer doesn't introspect), and auto-generated album index pages (album category-page descriptions are written freehand).
|
|
|
|
## Failure modes
|
|
|
|
Everything below aborts the run with a non-zero exit and the wiki untouched. Validation and rendering both complete before the first write, so a broken page never leaves the wiki half-updated.
|
|
|
|
| Condition | Result |
|
|
|---|---|
|
|
| Missing/empty `title`, non-boolean `wiki.publish`, malformed `release_date`, non-list `categories` | validation error |
|
|
| Two publishable files claiming the same `title` | validation error, checked repo-wide |
|
|
| A `.md` link whose target is missing, or exists but isn't published | render error |
|
|
| A `.md` link that survives rendering unresolved (unrecognised shape, or written inside a `columns` fence) | render error |
|
|
| `GITEA_REPO_URL` resolving to a loopback, mDNS or private-range host | run aborts before any wiki contact |
|
|
|
|
Two things degrade rather than fail: a missing or non-UTF-8 `lrc` file is skipped with a warning and the page still publishes; and a page whose existing wikitext does not start with `{{Auto-generated` is skipped with a warning, to avoid overwriting hand-written content. Delete the page or add the banner to hand it to the bot.
|
|
|
|
## Source repo and branch convention
|
|
|
|
Source repo: `novoyuuparosk-wiki/ncmr-songs`, triggering on push to `master`.
|
|
|
|
**Branch convention there**: per-song short-lived branches named `autowiki/<song-slug>` (e.g. `autowiki/pulse`). Create when staging edits, merge to `master` when ready to publish, delete after one cycle.
|
|
|
|
**Paths that don't trigger a run**: `wip/**`, `README.md`, `.gitignore`, `.claude/**`. Note that `wip/**` and `README.md` are excluded by the renderer too, so they can never publish even on a run triggered by something else.
|
|
|
|
## Invocation
|
|
|
|
`novoyuuparosk-wiki/ncmr-songs/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-songs.yml`](../../.gitea/workflows/publish-songs.yml) here, which clones both repos and runs `publish.py`.
|
|
|
|
Triggers:
|
|
|
|
- `push` to `master`, path-filtered as above
|
|
- `workflow_dispatch` — manual run from the Gitea Actions UI. Passes an empty base ref, which the pipeline reads as "publish everything", so this is the way to re-render every page after a renderer or template change
|
|
|
|
Editing `publish-songs.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md).
|
|
|
|
### Publish scope
|
|
|
|
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every publishable file, because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff plus one hop in each link direction: any page a changed page links to (so a link can't point at an unpublished page), and any page that links to a changed page (so an edited `title` doesn't leave stale inbound links).
|
|
|
|
| Invocation | Publishes |
|
|
|---|---|
|
|
| `--all` | every publishable file. Used on first push and `workflow_dispatch` |
|
|
| `--files <paths…>` | those files plus their one-hop link neighbours. Used for an ordinary push |
|
|
|
|
Because the banner carries the commit SHA, every page in a run's write set is rewritten on a new commit; a page pulled in only by the link closure is a no-op API read when the run repeats at the same SHA.
|
|
|
|
### Environment
|
|
|
|
| Name | Kind | Purpose |
|
|
|---|---|---|
|
|
| `WIKI_API_URL` | variable, org-scoped | MediaWiki action API endpoint |
|
|
| `WIKI_BASE_URL` | variable, org-scoped | Wiki base URL |
|
|
| `WIKI_BOT_USER` | secret, org-scoped | Bot login (`Mikkeli@giteaBot`) |
|
|
| `WIKI_BOT_PASSWORD` | secret, org-scoped | BotPassword value |
|
|
| `URL_TO_GITEA` | variable, org-scoped | Runner-internal Gitea base URL, for cloning only |
|
|
| `PUBLIC_URL_TO_GITEA` | variable, org-scoped | Public Gitea base URL, for the banner source link only |
|
|
| `FAPAT` | secret, org-scoped | Full-Access PAT for cloning at runtime |
|
|
| `SOURCE_REF` | env, set in workflow from `github.sha` | Commit SHA, used in the banner and the edit summary |
|
|
| `GITEA_REPO_URL` | env, set in workflow from `PUBLIC_URL_TO_GITEA` + repo path | Base of the banner source link. Validated by `public_source_base()` |
|
|
|
|
## Dependencies
|
|
|
|
Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild.
|
|
|
|
## Wiki-side prerequisites
|
|
|
|
- **`Template:Auto-generated`** must exist on the wiki. Wikitext source: [`template_auto_generated.wikitext`](template_auto_generated.wikitext) in this directory.
|
|
- The bot user must be in the `bot` group, which grants `skipcaptcha`.
|