wan
f5625cde98
The `wiki.siblings` bill of materials is gone. A related co-publication is now nominated with an ordinary relative Markdown link — `[label](notes.md)` — resolved at publish time against the target file's own `title` frontmatter. No macro namespace, no per-file declaration, and the link still works in Gitea's blob view and any editor preview. The macro could not be made safe. Substitution was `str.replace(tag, …)` over the whole rendered page: no word boundary, so a tag also occurring in lyrics or a code block got rewritten; no collision check, so overlapping tags corrupted each other in declaration order. SCHEMA.md documented the placeholder as `[[TAG]]` while the code replaced the bare tag, so following the schema produced `[[[[Title|label]]]]`. Fixes a live corruption bug in the process. `sibling_map` was built only from the files in the publish batch, and an unresolved sibling degraded to plain text — so pushing `ses/pulse.md` alone republished it with its wikilink silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the workflow "always runs --all", which it never did on an ordinary push. The repository is now always scanned in full, whatever a run publishes, so link resolution and title-collision detection can never depend on which files a push touched. Writes stay bounded to the push diff plus one hop in each link direction: the page a changed page links to (so a link cannot point at an unpublished page) and the pages linking to it (so an edited `title` does not leave stale inbound links). Publishing is not instant; reading is cheap. An unresolvable `.md` link is now a hard error, never graceful degradation, and linking to a file still does not publish it — `wiki.publish: true` remains the only gate. A leftover-link guard fails the run on any link shape the resolver did not recognise, including a Markdown link written inside a verbatim ```columns fence, so a pandoc upgrade cannot silently publish literal text. Also: - render before write: every page in a run is rendered before the first is published, so a broken page leaves the wiki untouched instead of half-updated. `publish_one` is split into `render_page` (pure) and the write. - pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside subprocess's reader thread, `stdout` came back None, and the real error was lost behind a TypeError. Latent on the Debian runner via PEP 538 only. - title collisions are detected repo-wide, not just within a push, and before any wiki contact. - tests/, run by .gitea/workflows/test.yml in the publish runner image. The pandoc output contract the resolver depends on is now pinned by a test instead of by prose — which is how SCHEMA.md came to assert something false. - drop an f-string with nothing to interpolate. - quote the SOURCE_REF interpolation in the workflow instead of pasting an input straight into shell. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7.7 KiB
7.7 KiB
pipelines/songs
Publishes lyric pages from novoyuuparosk-wiki/ncmr-songs to https://wiki.novoyuuparosk.org.
- Source-file contract (frontmatter and body conventions): SCHEMA.md
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): repo root README
v1 scope
Implemented:
- YAML frontmatter parsing per SCHEMA.md
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Auto-generated banner (
{{Auto-generated|source=...|commit=<sha>}}) - Album category injection (
[[Category:Songs]]+[[Category:Album:<name>]]whenalbumis set); additional categories viacategorieslist - LRC file upload (idempotent SHA1 check) + footer link
- Inter-page link resolution: a relative Markdown link (
[label](notes.md)) becomes[[wiki page title|label]], resolved against a repo-wide index so it never depends on which files a push touched - Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
- Files without
wiki.publish: truesilently skipped - Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
Not in scope (intentional):
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
- Auto-generated album index pages — user writes album category-page descriptions freeform
Source repo and branch convention
Source repo: novoyuuparosk-wiki/ncmr-songs. The pipeline triggers on push to master.
Branch convention in ncmr-songs: per-song short-lived branches named autowiki/<song-slug> (e.g., autowiki/pulse). Create when staging edits, merge to master when ready to publish, delete after one cycle.
Excluded paths (won't trigger the pipeline):
wip/**README.md,.gitignore,.claude/**
Invocation
The pipeline runs as a self-contained Gitea Actions workflow in novoyuuparosk-wiki/ncmr-songs (.gitea/workflows/publish.yml). It clones this repo at runtime to get the renderer.
Triggers:
pushtomaster(path-filtered as above)workflow_dispatch— manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit
Environment (secrets and variables)
| Name | Kind | Source | Purpose |
|---|---|---|---|
WIKI_API_URL |
variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
WIKI_BASE_URL |
variable | Gitea (cross-cutting) | Wiki base URL |
WIKI_BOT_USER |
secret | Gitea (cross-cutting) | Bot login (Mikkeli@giteaBot) |
WIKI_BOT_PASSWORD |
secret | Gitea (cross-cutting) | BotPasswords value |
URL_TO_GITEA |
variable | Gitea (cross-cutting) | Gitea instance base URL for cloning |
FAPAT |
secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
SOURCE_REF |
env (set in workflow) | github.sha |
Pushed commit SHA, used in banner and edit summary |
GITEA_REPO_URL |
env (set in workflow) | composed from vars | Source-file URL base for the banner link |
Dependencies
Job container: novoyuuparosk-wiki-runner:latest (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
See the Dockerfile at the repo root and requirements.txt in this directory. Rebuild the image after changes to either.
Modes
Reading is always repo-wide; writing is scoped to the push. Every run scans every file with wiki.publish: true (not under wip/), because link resolution and title-collision detection have to see the whole tree. What gets written is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page.
--all— publish every publishable file. Used on the first push and onworkflow_dispatch.--files <paths…>— publish these files and their one-hop link neighbours. Used for an ordinary push.
Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read.
One-off wiki setup
Template:Auto-generated— exists on the wiki. Wikitext source:template_auto_generated.wikitextin this directory.- Bot user in
botgroup — grantsskipcaptcha; verified working.
Decisions log (songs pipeline)
| Decision | Value | Date |
|---|---|---|
| Metadata source | YAML frontmatter inside each .md (see SCHEMA.md) |
2026-06-09 |
| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
| Excluded paths | wip/**, plus repo-meta files |
2026-06-09 |
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
| Album categories | Auto-injected [[Category:Songs]] + [[Category:Album:<name>]] when album is set; category pages are user-written and not overwritten |
2026-06-09 |
| Additional categories | Optional categories list in frontmatter; each entry → [[Category:<value>]] |
2026-06-10 |
album optional |
Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 |
path + tag + optional label); [[TAG]] substituted post-Pandoc |
2026-06-10 | |
| Inter-page links | Plain relative Markdown link ([label](notes.md)); target's own title frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the wiki.siblings bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch |
2026-09-09 |
| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 |
| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 |
| Source-repo branch convention | autowiki/<song-slug> short-lived branches in ncmr-songs |
2026-06-09 |
| Idempotency | Commit SHA in banner (commit=<sha>), not timestamp — stable across identical re-runs |
2026-06-09 |
| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 |
| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 |
| Tests | tests/ run by .gitea/workflows/test.yml in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose |
2026-09-09 |
| Manual-page protection | Bot skips pages without the {{Auto-generated banner to avoid overwriting hand-written content |
2026-06-09 |
workflow_call abandoned |
Cross-repo workflow_call fails — run token scoped to triggering repo; cannot clone private callee |
2026-06-09 |
| Side-by-side columns | Authored as a ```columns fenced block (separator ===); Pandoc emits <pre class="columns">, expanded post-Pandoc into a flex <div> of <poem> columns. No wiki template or extension — all Pi-side before the API write. See SCHEMA.md |
2026-06-14 |
Status
v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org.