wan
e1810facd6
tests / pytest (push) Successful in 7s
Every auto-published page, in all three pipelines, carried a source link to
`localhost:3005`. The template was never at fault — it links `{{{source}}}`
verbatim; the bad URL was baked into each page's wikitext by the publisher.
Root cause was one variable serving two incompatible purposes. `URL_TO_GITEA`
has to be an address the runner container can clone from, and since the runner
uses host networking that is `http://localhost:3005`. All three workflows then
composed `GITEA_REPO_URL` — the base of the public banner link, written into
the wiki for anyone to read — out of that same loopback value.
Adds a second org variable, `PUBLIC_URL_TO_GITEA`, for the public base
(`https://git.novoyuuparosk.org`), and points the banner at it. Cloning keeps
using `URL_TO_GITEA`. The two are now documented as deliberately separate,
because collapsing them looks like tidying up.
`lib/wiki.public_source_base()` validates the banner base rather than trusting
it: unset, relative, loopback, mDNS (`*.local`) or private-range values raise
with the variable to set. A misconfiguration that would publish an unreachable
link now stops the run instead. Note the failure mode this creates on purpose:
until `PUBLIC_URL_TO_GITEA` exists, publishing aborts rather than proceeding
with a wrong URL.
Also builds the source path with `rel.as_posix()`, so the separator in a URL
does not depend on the OS the publisher runs on — it produced backslashes when
the renderer was exercised on Windows.
Prerequisites before this can publish:
1. Add the org variable `PUBLIC_URL_TO_GITEA` = https://git.novoyuuparosk.org
2. Flush the act_runner callee cache — `docker compose up -d --force-recreate
act_runner`. Per the cross-cutting decisions log, edits to `publish-*.yml`
keep executing the stale cached copy, and this change is in those files.
Pages already published still carry the old URL in their stored wikitext. They
are only corrected by re-publishing each pipeline in full.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
109 lines
8.0 KiB
Markdown
109 lines
8.0 KiB
Markdown
# pipelines/songs
|
|
|
|
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
|
|
|
|
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
|
|
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
|
|
|
|
## v1 scope
|
|
|
|
Implemented:
|
|
|
|
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
|
|
- Pandoc-based markdown → wikitext body rendering
|
|
- First-line h1 stripping
|
|
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
|
|
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set); additional categories via `categories` list
|
|
- LRC file upload (idempotent SHA1 check) + footer link
|
|
- Inter-page link resolution: a relative Markdown link (`[label](notes.md)`) becomes `[[wiki page title|label]]`, resolved against a repo-wide index so it never depends on which files a push touched
|
|
- Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact
|
|
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
|
|
- Files without `wiki.publish: true` silently skipped
|
|
- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
|
|
|
|
Not in scope (intentional):
|
|
|
|
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
|
|
- Auto-generated album index pages — user writes album category-page descriptions freeform
|
|
|
|
## Source repo and branch convention
|
|
|
|
Source repo: `novoyuuparosk-wiki/ncmr-songs`. The pipeline triggers on push to `master`.
|
|
|
|
**Branch convention in `ncmr-songs`**: per-song short-lived branches named `autowiki/<song-slug>` (e.g., `autowiki/pulse`). Create when staging edits, merge to master when ready to publish, delete after one cycle.
|
|
|
|
**Excluded paths** (won't trigger the pipeline):
|
|
|
|
- `wip/**`
|
|
- `README.md`, `.gitignore`, `.claude/**`
|
|
|
|
## Invocation
|
|
|
|
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
|
|
|
|
Triggers:
|
|
- `push` to `master` (path-filtered as above)
|
|
- `workflow_dispatch` — manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit
|
|
|
|
### Environment (secrets and variables)
|
|
|
|
| Name | Kind | Source | Purpose |
|
|
|---|---|---|---|
|
|
| `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
|
|
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
|
|
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Mikkeli@giteaBot`) |
|
|
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
|
|
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Runner-internal Gitea base URL, for cloning only |
|
|
| `PUBLIC_URL_TO_GITEA` | variable | Gitea (cross-cutting) | Public Gitea base URL, for the banner source link only |
|
|
| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
|
|
| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary |
|
|
| `GITEA_REPO_URL` | env (set in workflow) | `PUBLIC_URL_TO_GITEA` + repo path | Source-file URL base for the banner link. Validated by `public_source_base()`: a loopback, mDNS or private-range host aborts the run rather than publishing an unreachable link |
|
|
|
|
## Dependencies
|
|
|
|
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
|
|
|
|
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
|
|
|
|
## Modes
|
|
|
|
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every file with `wiki.publish: true` (not under `wip/`), because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page.
|
|
|
|
- `--all` — publish every publishable file. Used on the first push and on `workflow_dispatch`.
|
|
- `--files <paths…>` — publish these files and their one-hop link neighbours. Used for an ordinary push.
|
|
|
|
Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read.
|
|
|
|
## One-off wiki setup
|
|
|
|
- [x] **`Template:Auto-generated`** — exists on the wiki. Wikitext source: `template_auto_generated.wikitext` in this directory.
|
|
- [x] **Bot user in `bot` group** — grants `skipcaptcha`; verified working.
|
|
|
|
## Decisions log (songs pipeline)
|
|
|
|
| Decision | Value | Date |
|
|
|---|---|---|
|
|
| Metadata source | YAML frontmatter inside each `.md` (see [SCHEMA.md](SCHEMA.md)) | 2026-06-09 |
|
|
| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
|
|
| Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 |
|
|
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
|
|
| Album categories | Auto-injected `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set; category pages are user-written and not overwritten | 2026-06-09 |
|
|
| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:<value>]]` | 2026-06-10 |
|
|
| `album` optional | Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 |
|
|
| ~~Sibling resolution~~ | ~~Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc~~ — **retired 2026-09-09**, see below | 2026-06-10 |
|
|
| Inter-page links | Plain relative Markdown link (`[label](notes.md)`); target's own `title` frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the `wiki.siblings` bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch | 2026-09-09 |
|
|
| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 |
|
|
| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 |
|
|
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
|
|
| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
|
|
| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 |
|
|
| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 |
|
|
| Tests | `tests/` run by `.gitea/workflows/test.yml` in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose | 2026-09-09 |
|
|
| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 |
|
|
| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 |
|
|
| Side-by-side columns | Authored as a ` ```columns ` fenced block (separator `===`); Pandoc emits `<pre class="columns">`, expanded post-Pandoc into a flex `<div>` of `<poem>` columns. No wiki template or extension — all Pi-side before the API write. See [SCHEMA.md](SCHEMA.md#columns-side-by-side-shorthand) | 2026-06-14 |
|
|
|
|
## Status
|
|
|
|
v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org.
|