diff --git a/README.md b/README.md index 3c91dad..f021727 100644 --- a/README.md +++ b/README.md @@ -10,28 +10,36 @@ CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from u | [`pipelines/ses/`](pipelines/ses/) | `novoyuuparosk-wiki/ses-light-novel` | SES light novel pages | v1 in development | | [`pipelines/tech/`](pipelines/tech/) | `novoyuuparosk-wiki/tech-blogs` | Tech blog posts | v1 live | -Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime, decisions). This root README covers only what's cross-cutting. +Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime). This root README covers only what's cross-cutting. ## Architecture -Hybrid layout. The Gitea Actions trigger must live in the source repo (Gitea only fires workflows from `.gitea/workflows/` of the pushed-to repo); the rendering logic lives here. The source-repo workflow is self-contained but clones this repo at runtime to get the renderer. +Split across two repos, because Gitea only fires workflows from `.gitea/workflows/` of the repo that was pushed to, while the rendering logic belongs here. ``` / - .gitea/workflows/.yml <- workflow: clones this repo, runs renderer (deps pre-baked in job image) + .gitea/workflows/publish.yml <- thin caller: on: push, uses: the reusable workflow below -novoyuuparosk-auto-wiki/ <- this repo - .gitea/workflows/.yml <- reusable workflow stubs (kept for reference; not actively called; may be stale) - pipelines// <- per-pipeline code, schema, templates - lib/ <- shared modules (MediaWiki client, etc.) +novoyuuparosk-auto-wiki/ <- this repo + .gitea/workflows/publish-

.yml <- on: workflow_call. Clones both repos, runs the renderer + .gitea/workflows/test.yml <- on: push. Runs tests/ in the publish runner image + .gitea/workflows/build-image.yml <- builds and pins the shared job container image + pipelines// <- per-pipeline renderer, schema, templates + lib/ <- shared render and MediaWiki helpers + tests/ <- pytest suite for lib/ and the pipelines ``` -Note: `workflow_call` across private repos was abandoned — the auto-generated run token is scoped to the triggering repo only and cannot clone a private callee. The source-repo workflow clones this repo directly using `FAPAT`. +The two halves reach the runner by different routes, which matters when a change appears not to take effect: + +- **Renderer code** (`pipelines/`, `lib/`) is a plain `git clone` of this repo's **default branch** on every run. Always current, never cached. +- **Workflow YAML** (`publish-*.yml`) is resolved through the caller's `uses:` and **is cached** — see *Callee workflow cache* below. + +**This repo must stay public.** `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied, and shared ownership does not satisfy the read requirement. A private callee 404s with `repository not found`. The alternative, untested here, is Settings → Actions → General → collaborative owners (Gitea 1.26+). ## Wiki - Base URL: https://wiki.novoyuuparosk.org -- MediaWiki API: https://wiki.novoyuuparosk.org/api.php *(confirmed)* +- MediaWiki API: https://wiki.novoyuuparosk.org/api.php (classic action API) ## Bot identity @@ -39,90 +47,72 @@ MediaWiki BotPassword issued for user `Mikkeli`, bot name `giteaBot`. Login form `Mikkeli` holds the `bot` right, so pipeline edits are flagged as bot edits and stay out of default Recent Changes — `mwclient`'s `page.save()` requests the bot flag and the wiki honours it. -This section previously named `Dubrowski@giteaAutomaton`, but every bot-flagged revision in the wiki's history is attributed to `Mikkeli`, so that had been stale for some time — the live secret never matched the doc. - -Credentials are stored in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo. +Credentials live in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo. ## Runner infrastructure -One `act_runner` instance serves all pipelines. Runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) inside the same `docker-compose` stack that hosts the Gitea instance. Job execution is via the host Docker socket — runner is a container, jobs spawn as sibling containers. +One `act_runner` instance serves all pipelines. It runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) as a service in the same `docker-compose` stack as Gitea, built `linux-arm64` from the `gitea/act_runner` image. Job execution is via the host Docker socket — the runner is a container and jobs spawn as sibling containers. -`act_runner` build: `linux-arm64`, from the `gitea/act_runner` Docker image. +Container network mode is `host`, so job containers can reach Gitea on `localhost:3005` and resolve mDNS hostnames. -Container network mode: `host` — required so job containers can reach `localhost:3005` (Gitea) and resolve mDNS hostnames. +### Callee workflow cache + +`act_runner` caches each resolved cross-repo callee at `/root/.cache/act/-@` inside the runner container, and **never refreshes it per run**. The key contains no commit SHA, so pushing to `master` does not invalidate it: edits to `publish-*.yml` — including the pin job's own commits — keep executing the stale cached copy indefinitely. + +Two consequences. When a pipeline misbehaves after a workflow edit, suspect the cache before suspecting permissions. And after any `publish-*.yml` change, flush it: + +``` +docker compose up -d --no-deps --force-recreate act_runner +``` + +`--no-deps` is required: `act_runner` declares `depends_on: gitea`, so without it the recreate cascades and bounces the Gitea instance. Pass no `--pull`, or the flush silently adopts whatever `act_runner:latest` has become. The cache sits in the container layer while `.runner` is in the `/data` volume, so registration survives the recreate. + +Changing a repo's owner also changes the cache key, which flushes it as a side effect. ### Job container image -All pipelines share a single pre-built Docker image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The `Dockerfile` is at the repo root. It bakes in system deps (git, pandoc, ca-certificates) and all pipeline Python packages so job containers start instantly with no install steps. +All pipelines share one pre-built image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The root `Dockerfile` bakes in system deps (git, pandoc, ca-certificates) and every pipeline's Python packages, so job containers start with no install step. -The image builds automatically via [`.gitea/workflows/build-image.yml`](.gitea/workflows/build-image.yml), which triggers on pushes that touch the `Dockerfile`, any pipeline `requirements.txt`, or that workflow itself. It uses kaniko (daemonless, unprivileged) to build and push two tags: an immutable `:` and a moving `:latest`. +[`build-image.yml`](.gitea/workflows/build-image.yml) rebuilds it on pushes touching the `Dockerfile`, any pipeline `requirements.txt`, or itself. It uses kaniko (daemonless, unprivileged) to push an immutable `:` and a moving `:latest`. A follow-up `pin` job rewrites the `image:` pin in each `publish-*.yml` to the new short SHA and commits it back to `master` using `FAPAT`, so the publish workflows always reference an immutable tag with no manual bump. That pin commit touches only `workflow_call` files, so it triggers no further runs — but it does need a cache flush to take effect. -A follow-up `pin` job then rewrites the `image:` pin in each `publish-*.yml` to the new `:` and commits it back to `master` (using `FAPAT` for contents write). The publish workflows therefore always reference an immutable tag, kept current automatically — no manual bump. The pin commit only touches `workflow_call` files, so it triggers no further runs. +The builder image `kaniko-act` lives under `mikkeli`, not the org, and **cannot be rebuilt by `build-image.yml`**: kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so the resulting image has no `/tmp` and `wget -O /tmp/...` fails. Bumping it needs a host `docker build` and push, or a Dockerfile that forces the dir to materialise by writing a file inside it. [`build-kaniko-act.yml`](.gitea/workflows/build-kaniko-act.yml) is dispatch-only and runs in the old `mikkeli` copy. ## Gitea Actions setup (cross-cutting) -Secrets and variables are scoped to the `novoyuuparosk-wiki` org, inherited by all repos under it. Runs belong to the *caller* repo, so a source repo calling a reusable workflow here resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo. +Secrets and variables are scoped to the `novoyuuparosk-wiki` org and inherited by every repo under it. A run belongs to the *caller* repo, so a source repo resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo. **Secrets:** -- `WIKI_BOT_USER` = `Mikkeli@giteaBot` -- `WIKI_BOT_PASSWORD` = the value from *Bot identity* above -- `FAPAT` = Full-Access PAT owned by `mikkeli`, used by source-repo workflows to clone this repo at runtime. The PAT stays personal; only its storage scope moved to the org — hence the `mikkeli:` basic-auth username in the clone URLs. -- `PKGRW_PAT` = package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry + +| Name | Value / purpose | +|---|---| +| `WIKI_BOT_USER` | `Mikkeli@giteaBot` | +| `WIKI_BOT_PASSWORD` | the BotPassword from *Bot identity* above | +| `FAPAT` | Full-Access PAT owned by `mikkeli`, used by the publish workflows to clone repos at runtime. The PAT is personal; only its storage scope is the org — hence the `mikkeli:` basic-auth username in clone URLs | +| `PKGRW_PAT` | package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry | **Variables:** -- `WIKI_BASE_URL` = `https://wiki.novoyuuparosk.org` -- `WIKI_API_URL` = `https://wiki.novoyuuparosk.org/api.php` -- `URL_TO_GITEA` = **runner-internal** Gitea base URL, used only for cloning (e.g. `http://localhost:3005` — the runner uses host networking). Named with `URL_TO_` prefix — Gitea blocks variable names starting with `GITEA_` or `GITHUB_`. -- `PUBLIC_URL_TO_GITEA` = **public** Gitea base URL (`https://git.novoyuuparosk.org`), used only for the source link in each published page's banner. -These two must stay separate. `URL_TO_GITEA` is a loopback address, so composing the banner from it published every page — across all three pipelines — citing `localhost:3005`, which resolves for nobody reading the wiki. `lib/wiki.public_source_base()` now rejects a loopback, mDNS, or private-range value, so that misconfiguration fails the run instead of reaching a page. +| Name | Value | Used for | +|---|---|---| +| `WIKI_BASE_URL` | `https://wiki.novoyuuparosk.org` | wiki base | +| `WIKI_API_URL` | `https://wiki.novoyuuparosk.org/api.php` | MediaWiki action API | +| `URL_TO_GITEA` | `http://localhost:3005` | **runner-internal** address, cloning only | +| `PUBLIC_URL_TO_GITEA` | `https://git.novoyuuparosk.org` | **public** address, page banner source links only | + +Names avoid a `GITEA_`/`GITHUB_` prefix because Gitea blocks those, hence `URL_TO_GITEA` rather than `GITEA_URL`. + +The two URL variables must stay separate. `URL_TO_GITEA` has to be an address the runner container can clone from, and host networking makes that a loopback. `PUBLIC_URL_TO_GITEA` is baked into the `{{Auto-generated}}` banner of every published page and has to resolve for a reader on the internet. `lib/wiki.public_source_base()` validates the banner value and aborts the run on an unset, relative, loopback, mDNS, or private-range address, so collapsing them back into one fails loudly instead of publishing dead links. ## Branch naming -- This repo: `automation/` for pipeline-development branches (e.g., `automation/songs`). -- Source repos: each pipeline's README defines the source-side branch convention (e.g., `autowiki/` in `ncmr-songs`). +- This repo: `automation/` for pipeline-development branches (e.g. `automation/songs`). +- Source repos: each pipeline's README defines its own convention (e.g. `autowiki/` in `ncmr-songs`). -## Decisions log (cross-cutting) +## Tests -| Decision | Value | Date | -|---|---|---| -| Architecture | Hybrid: source-repo workflow clones this repo at runtime for the renderer | 2026-06-09 | -| Workflow pattern | Self-contained (not `workflow_call`) — cross-repo `workflow_call` blocked by token scoping on private repos | 2026-06-09 | -| Runner execution | Docker, added as a service to the existing Gitea docker-compose | 2026-06-09 | -| Runner network mode | `host` — job containers need to reach Gitea on localhost | 2026-06-09 | -| Secret/runner scope | User-level on `mikkeli` (no orgs on this instance) | 2026-06-09 | -| Ownership | This repo and all three source repos moved to the `novoyuuparosk-wiki` org. Secrets/variables re-created at org scope; runner re-registered instance-level so it serves org-owned runs | 2026-08-11 | -| **This repo must stay public** | `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied. A private callee therefore 404s with `repository not found`, regardless of the caller sharing its owner. Shared ownership does **not** satisfy the read requirement. Alternative if it must be private again: Settings → Actions → General → collaborative owners (Gitea 1.26+), untested here | 2026-08-11 | -| Runner cache masks this | The resolved callee is cached at `/root/.cache/act/-@`. Changing owner changes the key, so a working pipeline can break on a clone that had been served from cache for months. Suspect the cache before suspecting permissions | 2026-08-11 | -| **Callee edits need a cache flush** | The cache is *not* refreshed per run. Edits to `publish-*.yml` on `master` — including the pin job's own commits — keep executing the stale cached copy until the key changes or the cache is cleared. Observed directly: a run used the previous namespace and tag despite `master` being current. Flush with `docker compose up -d --no-deps --force-recreate act_runner`; the cache is in the container layer, while `.runner` is in the `/data` volume, so registration survives. `--no-deps` is required — `act_runner` declares `depends_on: gitea`, so without it the recreate cascades and bounces the Gitea instance itself. Pass no `--pull`, or the flush silently adopts whatever `act_runner:latest` has become | 2026-08-11 | -| kaniko cannot rebuild kaniko-act | The org rebuild exits 0 but the resulting image has no `/tmp` — kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so `wget -O /tmp/...` fails. The bootstrap note's claim that version bumps "can be built by the build-image workflow itself" does not hold as written. `kaniko-act` therefore still lives under `mikkeli`; a real bump needs a host `docker build` + push, or a Dockerfile that forces the dir to materialise (e.g. writing a file inside it) | 2026-08-11 | -| Container registry | Gitea cannot transfer packages, so images were *rebuilt* into `pi5-16.local:3005/novoyuuparosk-wiki/*` rather than moved. `kaniko-act` bootstrapped via the new dispatch-only `build-kaniko-act.yml`, running in the old `mikkeli` copy; `novoyuuparosk-wiki-runner` came from a normal `build-image.yml` run. Registry auth is still the `mikkeli`-owned `PKGRW_PAT`, hence the `mikkeli:` username in the auth blob | 2026-08-11 | -| Old images left in place | The `mikkeli/*` package versions are orphaned but retained — nothing references them, and deleting a container version is irreversible. Safe to purge once the org images have proven themselves | 2026-08-11 | -| MediaWiki API path | `api.php` (classic action API) | 2026-06-09 | -| Branch naming (this repo) | `automation/` for pipeline-development branches | 2026-06-09 | -| Variable naming | `URL_TO_GITEA` not `GITEA_URL` — Gitea blocks `GITEA_`/`GITHUB_` prefixes | 2026-06-09 | -| **Clone URL ≠ banner URL** | Split into `URL_TO_GITEA` (runner-internal, for cloning) and `PUBLIC_URL_TO_GITEA` (public, for the page banner). One variable served both, and since host networking forces the clone address to `localhost:3005`, every auto-published page in every pipeline cited a link no reader could open. The banner value is validated by `public_source_base()` — loopback, mDNS and private-range addresses abort the run. Pages published before this carry the bad URL in their wikitext and need a full re-publish per pipeline to clear | 2026-09-09 | -| Source URL separators | `rel.as_posix()` in the banner URL, so the path separator never depends on the OS the publisher happens to run on | 2026-09-09 | -| Columns shorthand | Side-by-side columns authored as a ` ```columns ` fenced block, expanded post-Pandoc in shared `lib/wiki.py` (universal across pipelines). No wiki template or PHP extension — runs Pi-side before the API call | 2026-06-14 | +``` +pip install -r requirements-dev.txt +python -m pytest tests -q +``` -Per-pipeline decisions live in each pipeline's README. - -## Setup checklist (cross-cutting) - -Via the Gitea web UI logged in as `mikkeli` (an owner of `novoyuuparosk-wiki`): - -- [x] Org-scoped secrets and variables set per *Gitea Actions setup* above - - [x] `WIKI_BOT_USER` - - [x] `WIKI_BOT_PASSWORD` - - [x] `FAPAT` (Full-Access PAT — value not stored in this README; saved directly into the Gitea secret. Regenerate if lost.) - - [x] `WIKI_BASE_URL` - - [x] `WIKI_API_URL` - - [x] `URL_TO_GITEA` - - [ ] `PUBLIC_URL_TO_GITEA` = `https://git.novoyuuparosk.org` — **required**; the publishers abort without it - -With Pi access: - -- [x] Add `act_runner` service to the existing Gitea docker-compose -- [x] Generate a runner registration token at `/-/admin/actions/runners`, bake into the compose env, `docker compose up -d act_runner`, confirm "online" in the Gitea UI - -Per-pipeline setup lives in each pipeline's README. Start with [`pipelines/songs/`](pipelines/songs/). +Pure string-in/string-out coverage of `lib/` and the pipelines' discovery, validation and publish-scoping logic — no wiki, no network. The `TestPandocContract` cases shell out to real pandoc and are skipped if it isn't installed; they pin the pandoc output shape the link resolver depends on, so a pandoc upgrade fails a test instead of publishing broken links. [`test.yml`](.gitea/workflows/test.yml) runs the suite on push in the same image the pipelines publish from. diff --git a/pipelines/ses/README.md b/pipelines/ses/README.md index 074e7b6..6c041e2 100644 --- a/pipelines/ses/README.md +++ b/pipelines/ses/README.md @@ -5,7 +5,7 @@ Publishes SES light novel pages from `novoyuuparosk-wiki/ses-light-novel` to `ht - Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md) - Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md) -## v1 scope +## What the renderer does - YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md) - `SES:` title prefix — all pages live in the SES MediaWiki namespace @@ -26,7 +26,7 @@ Not in scope (intentional): Source repo: `novoyuuparosk-wiki/ses-light-novel`. The pipeline triggers on push to `master`. -No branch convention enforced — work directly on `master` or use whatever branch workflow suits. +No branch convention enforced — work directly on `master` or use whatever branch workflow suits. There is no `wip/` exclusion; `wiki.publish: true` is the only publishing gate. **Excluded paths** (won't trigger the pipeline): @@ -34,12 +34,18 @@ No branch convention enforced — work directly on `master` or use whatever bran ## Invocation -The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ses-light-novel` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer. +`novoyuuparosk-wiki/ses-light-novel/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-ses.yml`](../../.gitea/workflows/publish-ses.yml) here, which clones both repos and runs `publish.py`. + +Editing `publish-ses.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md). Triggers: - `push` to `master` (path-filtered as above) - `workflow_dispatch` — manual trigger from the Gitea Actions UI +### Publish scope + +Every run publishes every file with `wiki.publish: true`; there is no diff detection. Writes are idempotent — the banner carries the commit SHA rather than a timestamp, so a re-run at the same commit rewrites nothing. + ### Environment (secrets and variables) | Name | Kind | Source | Purpose | @@ -56,29 +62,8 @@ Triggers: ## Dependencies -Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime. +Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild. Shared render and MediaWiki helpers live in `lib/wiki.py` at the repo root. -See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either. +## Wiki-side prerequisites -Shared code lives in `lib/wiki.py` (repo root). - -## One-off wiki setup - -- [ ] **`Template:Auto-generated`** — must exist on the wiki (shared with songs pipeline; already created). - -## Decisions log (ses pipeline) - -| Decision | Value | Date | -|---|---|---| -| Namespace | All pages prefixed `SES:` — auto-applied by renderer, not written in source `title:` | 2026-06-10 | -| Type resolution | `type:` frontmatter field; falls back to immediate parent directory name | 2026-06-10 | -| Categories | `Category:SES` + `Category:SES:` on every page | 2026-06-10 | -| Sibling links | Not implemented — write `[[SES:Page title]]` wikilinks directly in the body | 2026-06-10 | -| WIP exclusion | None — `wiki.publish: false` is the only gate | 2026-06-10 | -| Publish mode | Always `--all` | 2026-06-10 | -| Idempotency | Commit SHA in banner, not timestamp | 2026-06-10 | -| Manual-page protection | Bot skips pages without `{{Auto-generated` banner | 2026-06-10 | - -## Status - -v1 in development. +**`Template:Auto-generated`** must exist on the wiki. It is shared across all three pipelines; the wikitext source lives in [`pipelines/songs/template_auto_generated.wikitext`](../songs/template_auto_generated.wikitext). diff --git a/pipelines/songs/README.md b/pipelines/songs/README.md index 11de929..4d6de76 100644 --- a/pipelines/songs/README.md +++ b/pipelines/songs/README.md @@ -1,108 +1,86 @@ # pipelines/songs -Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`. +Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`. v1, live. - Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md) - Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md) -## v1 scope +## What the renderer does -Implemented: +- Parses YAML frontmatter per [SCHEMA.md](SCHEMA.md); files without `wiki.publish: true` are skipped silently +- Strips the body's first-line `# Heading`, then renders markdown → wikitext with Pandoc +- Expands ` ```columns ` fenced blocks into a flex row of `` columns (shared transform in `lib/wiki.py`) +- Resolves relative Markdown links to sibling `.md` files into `[[wiki page title|label]]`, using the target's own `title` frontmatter as the page name +- Prepends the `{{Auto-generated|source=…|commit=}}` banner +- Uploads a declared `lrc` companion file when its SHA1 has changed, and appends a `[[Media:…]]` footer link +- Injects `[[Category:Songs]]` + `[[Category:Album:]]` when `album` is set, plus one `[[Category:…]]` per `categories` entry. Category pages themselves are user-written and never overwritten +- Writes via the MediaWiki bot API, skipping the write when the generated text already matches the page -- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md) -- Pandoc-based markdown → wikitext body rendering -- First-line h1 stripping -- Auto-generated banner (`{{Auto-generated|source=...|commit=}}`) -- Album category injection (`[[Category:Songs]]` + `[[Category:Album:]]` when `album` is set); additional categories via `categories` list -- LRC file upload (idempotent SHA1 check) + footer link -- Inter-page link resolution: a relative Markdown link (`[label](notes.md)`) becomes `[[wiki page title|label]]`, resolved against a repo-wide index so it never depends on which files a push touched -- Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact -- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output -- Files without `wiki.publish: true` silently skipped -- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot +Deliberately not in scope: multi-language metadata (the body conveys what languages exist; the renderer doesn't introspect), and auto-generated album index pages (album category-page descriptions are written freehand). -Not in scope (intentional): +## Failure modes -- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect) -- Auto-generated album index pages — user writes album category-page descriptions freeform +Everything below aborts the run with a non-zero exit and the wiki untouched. Validation and rendering both complete before the first write, so a broken page never leaves the wiki half-updated. + +| Condition | Result | +|---|---| +| Missing/empty `title`, non-boolean `wiki.publish`, malformed `release_date`, non-list `categories` | validation error | +| Two publishable files claiming the same `title` | validation error, checked repo-wide | +| A `.md` link whose target is missing, or exists but isn't published | render error | +| A `.md` link that survives rendering unresolved (unrecognised shape, or written inside a `columns` fence) | render error | +| `GITEA_REPO_URL` resolving to a loopback, mDNS or private-range host | run aborts before any wiki contact | + +Two things degrade rather than fail: a missing or non-UTF-8 `lrc` file is skipped with a warning and the page still publishes; and a page whose existing wikitext does not start with `{{Auto-generated` is skipped with a warning, to avoid overwriting hand-written content. Delete the page or add the banner to hand it to the bot. ## Source repo and branch convention -Source repo: `novoyuuparosk-wiki/ncmr-songs`. The pipeline triggers on push to `master`. +Source repo: `novoyuuparosk-wiki/ncmr-songs`, triggering on push to `master`. -**Branch convention in `ncmr-songs`**: per-song short-lived branches named `autowiki/` (e.g., `autowiki/pulse`). Create when staging edits, merge to master when ready to publish, delete after one cycle. +**Branch convention there**: per-song short-lived branches named `autowiki/` (e.g. `autowiki/pulse`). Create when staging edits, merge to `master` when ready to publish, delete after one cycle. -**Excluded paths** (won't trigger the pipeline): - -- `wip/**` -- `README.md`, `.gitignore`, `.claude/**` +**Paths that don't trigger a run**: `wip/**`, `README.md`, `.gitignore`, `.claude/**`. Note that `wip/**` and `README.md` are excluded by the renderer too, so they can never publish even on a run triggered by something else. ## Invocation -The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer. +`novoyuuparosk-wiki/ncmr-songs/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-songs.yml`](../../.gitea/workflows/publish-songs.yml) here, which clones both repos and runs `publish.py`. Triggers: -- `push` to `master` (path-filtered as above) -- `workflow_dispatch` — manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit -### Environment (secrets and variables) +- `push` to `master`, path-filtered as above +- `workflow_dispatch` — manual run from the Gitea Actions UI. Passes an empty base ref, which the pipeline reads as "publish everything", so this is the way to re-render every page after a renderer or template change -| Name | Kind | Source | Purpose | -|---|---|---|---| -| `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint | -| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL | -| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Mikkeli@giteaBot`) | -| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value | -| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Runner-internal Gitea base URL, for cloning only | -| `PUBLIC_URL_TO_GITEA` | variable | Gitea (cross-cutting) | Public Gitea base URL, for the banner source link only | -| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos | -| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary | -| `GITEA_REPO_URL` | env (set in workflow) | `PUBLIC_URL_TO_GITEA` + repo path | Source-file URL base for the banner link. Validated by `public_source_base()`: a loopback, mDNS or private-range host aborts the run rather than publishing an unreachable link | +Editing `publish-songs.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md). + +### Publish scope + +**Reading is always repo-wide; writing is scoped to the push.** Every run scans every publishable file, because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff plus one hop in each link direction: any page a changed page links to (so a link can't point at an unpublished page), and any page that links to a changed page (so an edited `title` doesn't leave stale inbound links). + +| Invocation | Publishes | +|---|---| +| `--all` | every publishable file. Used on first push and `workflow_dispatch` | +| `--files ` | those files plus their one-hop link neighbours. Used for an ordinary push | + +Because the banner carries the commit SHA, every page in a run's write set is rewritten on a new commit; a page pulled in only by the link closure is a no-op API read when the run repeats at the same SHA. + +### Environment + +| Name | Kind | Purpose | +|---|---|---| +| `WIKI_API_URL` | variable, org-scoped | MediaWiki action API endpoint | +| `WIKI_BASE_URL` | variable, org-scoped | Wiki base URL | +| `WIKI_BOT_USER` | secret, org-scoped | Bot login (`Mikkeli@giteaBot`) | +| `WIKI_BOT_PASSWORD` | secret, org-scoped | BotPassword value | +| `URL_TO_GITEA` | variable, org-scoped | Runner-internal Gitea base URL, for cloning only | +| `PUBLIC_URL_TO_GITEA` | variable, org-scoped | Public Gitea base URL, for the banner source link only | +| `FAPAT` | secret, org-scoped | Full-Access PAT for cloning at runtime | +| `SOURCE_REF` | env, set in workflow from `github.sha` | Commit SHA, used in the banner and the edit summary | +| `GITEA_REPO_URL` | env, set in workflow from `PUBLIC_URL_TO_GITEA` + repo path | Base of the banner source link. Validated by `public_source_base()` | ## Dependencies -Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime. +Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild. -See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either. +## Wiki-side prerequisites -## Modes - -**Reading is always repo-wide; writing is scoped to the push.** Every run scans every file with `wiki.publish: true` (not under `wip/`), because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page. - -- `--all` — publish every publishable file. Used on the first push and on `workflow_dispatch`. -- `--files ` — publish these files and their one-hop link neighbours. Used for an ordinary push. - -Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read. - -## One-off wiki setup - -- [x] **`Template:Auto-generated`** — exists on the wiki. Wikitext source: `template_auto_generated.wikitext` in this directory. -- [x] **Bot user in `bot` group** — grants `skipcaptcha`; verified working. - -## Decisions log (songs pipeline) - -| Decision | Value | Date | -|---|---|---| -| Metadata source | YAML frontmatter inside each `.md` (see [SCHEMA.md](SCHEMA.md)) | 2026-06-09 | -| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 | -| Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 | -| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 | -| Album categories | Auto-injected `[[Category:Songs]]` + `[[Category:Album:]]` when `album` is set; category pages are user-written and not overwritten | 2026-06-09 | -| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:]]` | 2026-06-10 | -| `album` optional | Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 | -| ~~Sibling resolution~~ | ~~Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc~~ — **retired 2026-09-09**, see below | 2026-06-10 | -| Inter-page links | Plain relative Markdown link (`[label](notes.md)`); target's own `title` frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the `wiki.siblings` bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch | 2026-09-09 | -| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 | -| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 | -| Source-repo branch convention | `autowiki/` short-lived branches in `ncmr-songs` | 2026-06-09 | -| Idempotency | Commit SHA in banner (`commit=`), not timestamp — stable across identical re-runs | 2026-06-09 | -| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 | -| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 | -| Tests | `tests/` run by `.gitea/workflows/test.yml` in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose | 2026-09-09 | -| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 | -| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 | -| Side-by-side columns | Authored as a ` ```columns ` fenced block (separator `===`); Pandoc emits `

`, expanded post-Pandoc into a flex `
` of `` columns. No wiki template or extension — all Pi-side before the API write. See [SCHEMA.md](SCHEMA.md#columns-side-by-side-shorthand) | 2026-06-14 | - -## Status - -v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org. +- **`Template:Auto-generated`** must exist on the wiki. Wikitext source: [`template_auto_generated.wikitext`](template_auto_generated.wikitext) in this directory. +- The bot user must be in the `bot` group, which grants `skipcaptcha`. diff --git a/pipelines/songs/SCHEMA.md b/pipelines/songs/SCHEMA.md index e3debbf..3d96539 100644 --- a/pipelines/songs/SCHEMA.md +++ b/pipelines/songs/SCHEMA.md @@ -268,4 +268,4 @@ Files without a `wiki.publish: true` frontmatter field are silently skipped — Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken. -Frontmatter keys the schema does not define are ignored, so a leftover `wiki.siblings` block from the retired macro system is inert rather than an error. Delete them; they do nothing. +Frontmatter keys this schema does not define are ignored rather than rejected. diff --git a/pipelines/tech/README.md b/pipelines/tech/README.md index 4e51bde..c0ec23a 100644 --- a/pipelines/tech/README.md +++ b/pipelines/tech/README.md @@ -5,7 +5,7 @@ Publishes tech blog posts from `novoyuuparosk-wiki/tech-blogs` to `https://wiki. - Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md) - Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md) -## v1 scope +## What the renderer does - YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md) - Pandoc-based markdown → wikitext body rendering (h1 elements preserved) @@ -18,9 +18,9 @@ Publishes tech blog posts from `novoyuuparosk-wiki/tech-blogs` to `https://wiki. ## Source repo -Source repo: `novoyuuparosk-wiki/tech-blogs` (not yet initialised — pipeline code is ready). +Source repo: `novoyuuparosk-wiki/tech-blogs`, triggering on push to `master`, with `workflow_dispatch` for manual runs. -The pipeline triggers on push to `master` and supports `workflow_dispatch` for manual runs. +No branch convention enforced. There is no `wip/` exclusion — `wiki.publish: true` is the only publishing gate. **Excluded paths** (won't trigger the pipeline): @@ -28,12 +28,18 @@ The pipeline triggers on push to `master` and supports `workflow_dispatch` for m ## Invocation -The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/tech-blogs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer. +`novoyuuparosk-wiki/tech-blogs/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-tech.yml`](../../.gitea/workflows/publish-tech.yml) here, which clones both repos and runs `publish.py`. + +Editing `publish-tech.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md). Triggers: - `push` to `master` (path-filtered as above) - `workflow_dispatch` — manual trigger from the Gitea Actions UI +### Publish scope + +Every run publishes every file with `wiki.publish: true`; there is no diff detection. Writes are idempotent — the banner carries the commit SHA rather than a timestamp, so a re-run at the same commit rewrites nothing. + ### Environment (secrets and variables) | Name | Kind | Source | Purpose | @@ -50,30 +56,8 @@ Triggers: ## Dependencies -Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime. +Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild. Shared render and MediaWiki helpers live in `lib/wiki.py` at the repo root. -See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either. +## Wiki-side prerequisites -Shared code lives in `lib/wiki.py` (repo root). - -## One-off wiki setup - -- [ ] **`Template:Auto-generated`** — must exist on the wiki (shared with all pipelines; already created). - -## Decisions log (tech pipeline) - -| Decision | Value | Date | -|---|---|---| -| No title prefix | Tech posts live at bare titles — no namespace needed | 2026-06-10 | -| Fixed category | Every post gets `Category:Tech blog` | 2026-06-10 | -| Year category | `Category:Blog:` from `date` field; falls back to the execution year if no date | 2026-06-10 | -| Tag categories | Each tag → `[[Category:]]` with no prefix | 2026-06-10 | -| WIP exclusion | None — `wiki.publish: false` is the only gate | 2026-06-10 | -| Publish mode | Always `--all` | 2026-06-10 | -| Idempotency | Commit SHA in banner | 2026-06-10 | -| Manual-page protection | Bot skips pages without `{{Auto-generated` banner | 2026-06-10 | -| Source repo | `novoyuuparosk-wiki/tech-blogs` — pipeline ships before repo is initialised | 2026-06-10 | - -## Status - -v1 pipeline ready. Source repo (`novoyuuparosk-wiki/tech-blogs`) not yet initialised. +**`Template:Auto-generated`** must exist on the wiki. It is shared across all three pipelines; the wikitext source lives in [`pipelines/songs/template_auto_generated.wikitext`](../songs/template_auto_generated.wikitext).