wan
3ab59b493e
tests / pytest (push) Successful in 6s
Removes the decision logs, the dated rows, the strikethrough retirements and the setup progress checkboxes from all four READMEs. A decision log records how the system came to be; a reader needs to know how it is. The two had drifted far enough apart that the docs asserted things the shipped config contradicted. Corrected, all of which were stale rather than merely verbose: - the architecture section described the source-repo workflow as self-contained and called the reusable workflows here "not actively called; may be stale". They are the pipelines; the source-repo workflow is a thin caller. - `workflow_call` was recorded as abandoned, and the instance as having no orgs. Both pipelines run through cross-repo `workflow_call` under an org. - tech-blogs was described as not yet initialised. It has published. - ses and tech invocation sections claimed a self-contained workflow too. Kept, rewritten as behaviour rather than as dated decisions, because they are live operational constraints and not history: - the callee workflow cache, why a `publish-*.yml` edit appears not to take, and the flush command with its `--no-deps` and `--pull` caveats - why this repo has to stay public - the `URL_TO_GITEA` / `PUBLIC_URL_TO_GITEA` split and what validates it - host network mode, and that `kaniko-act` cannot be rebuilt by build-image.yml Also documents what the renderer, its failure modes and its publish scope actually are, which the decision logs had been standing in for, and adds the architecture note that renderer code is cloned fresh per run while workflow YAML is cached — the distinction that governs whether a change takes effect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
119 lines
8.3 KiB
Markdown
119 lines
8.3 KiB
Markdown
# novoyuuparosk-auto-wiki
|
|
|
|
CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from upstream content repos.
|
|
|
|
## Pipelines
|
|
|
|
| Path | Source repo | Purpose | Status |
|
|
|---|---|---|---|
|
|
| [`pipelines/songs/`](pipelines/songs/) | `novoyuuparosk-wiki/ncmr-songs` | Song lyric pages | v1 live |
|
|
| [`pipelines/ses/`](pipelines/ses/) | `novoyuuparosk-wiki/ses-light-novel` | SES light novel pages | v1 in development |
|
|
| [`pipelines/tech/`](pipelines/tech/) | `novoyuuparosk-wiki/tech-blogs` | Tech blog posts | v1 live |
|
|
|
|
Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime). This root README covers only what's cross-cutting.
|
|
|
|
## Architecture
|
|
|
|
Split across two repos, because Gitea only fires workflows from `.gitea/workflows/` of the repo that was pushed to, while the rendering logic belongs here.
|
|
|
|
```
|
|
<source-repo>/
|
|
.gitea/workflows/publish.yml <- thin caller: on: push, uses: the reusable workflow below
|
|
|
|
novoyuuparosk-auto-wiki/ <- this repo
|
|
.gitea/workflows/publish-<p>.yml <- on: workflow_call. Clones both repos, runs the renderer
|
|
.gitea/workflows/test.yml <- on: push. Runs tests/ in the publish runner image
|
|
.gitea/workflows/build-image.yml <- builds and pins the shared job container image
|
|
pipelines/<pipeline>/ <- per-pipeline renderer, schema, templates
|
|
lib/ <- shared render and MediaWiki helpers
|
|
tests/ <- pytest suite for lib/ and the pipelines
|
|
```
|
|
|
|
The two halves reach the runner by different routes, which matters when a change appears not to take effect:
|
|
|
|
- **Renderer code** (`pipelines/`, `lib/`) is a plain `git clone` of this repo's **default branch** on every run. Always current, never cached.
|
|
- **Workflow YAML** (`publish-*.yml`) is resolved through the caller's `uses:` and **is cached** — see *Callee workflow cache* below.
|
|
|
|
**This repo must stay public.** `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied, and shared ownership does not satisfy the read requirement. A private callee 404s with `repository not found`. The alternative, untested here, is Settings → Actions → General → collaborative owners (Gitea 1.26+).
|
|
|
|
## Wiki
|
|
|
|
- Base URL: https://wiki.novoyuuparosk.org
|
|
- MediaWiki API: https://wiki.novoyuuparosk.org/api.php (classic action API)
|
|
|
|
## Bot identity
|
|
|
|
MediaWiki BotPassword issued for user `Mikkeli`, bot name `giteaBot`. Login form: `Mikkeli@giteaBot`.
|
|
|
|
`Mikkeli` holds the `bot` right, so pipeline edits are flagged as bot edits and stay out of default Recent Changes — `mwclient`'s `page.save()` requests the bot flag and the wiki honours it.
|
|
|
|
Credentials live in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo.
|
|
|
|
## Runner infrastructure
|
|
|
|
One `act_runner` instance serves all pipelines. It runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) as a service in the same `docker-compose` stack as Gitea, built `linux-arm64` from the `gitea/act_runner` image. Job execution is via the host Docker socket — the runner is a container and jobs spawn as sibling containers.
|
|
|
|
Container network mode is `host`, so job containers can reach Gitea on `localhost:3005` and resolve mDNS hostnames.
|
|
|
|
### Callee workflow cache
|
|
|
|
`act_runner` caches each resolved cross-repo callee at `/root/.cache/act/<owner>-<repo>@<ref>` inside the runner container, and **never refreshes it per run**. The key contains no commit SHA, so pushing to `master` does not invalidate it: edits to `publish-*.yml` — including the pin job's own commits — keep executing the stale cached copy indefinitely.
|
|
|
|
Two consequences. When a pipeline misbehaves after a workflow edit, suspect the cache before suspecting permissions. And after any `publish-*.yml` change, flush it:
|
|
|
|
```
|
|
docker compose up -d --no-deps --force-recreate act_runner
|
|
```
|
|
|
|
`--no-deps` is required: `act_runner` declares `depends_on: gitea`, so without it the recreate cascades and bounces the Gitea instance. Pass no `--pull`, or the flush silently adopts whatever `act_runner:latest` has become. The cache sits in the container layer while `.runner` is in the `/data` volume, so registration survives the recreate.
|
|
|
|
Changing a repo's owner also changes the cache key, which flushes it as a side effect.
|
|
|
|
### Job container image
|
|
|
|
All pipelines share one pre-built image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The root `Dockerfile` bakes in system deps (git, pandoc, ca-certificates) and every pipeline's Python packages, so job containers start with no install step.
|
|
|
|
[`build-image.yml`](.gitea/workflows/build-image.yml) rebuilds it on pushes touching the `Dockerfile`, any pipeline `requirements.txt`, or itself. It uses kaniko (daemonless, unprivileged) to push an immutable `:<short-sha>` and a moving `:latest`. A follow-up `pin` job rewrites the `image:` pin in each `publish-*.yml` to the new short SHA and commits it back to `master` using `FAPAT`, so the publish workflows always reference an immutable tag with no manual bump. That pin commit touches only `workflow_call` files, so it triggers no further runs — but it does need a cache flush to take effect.
|
|
|
|
The builder image `kaniko-act` lives under `mikkeli`, not the org, and **cannot be rebuilt by `build-image.yml`**: kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so the resulting image has no `/tmp` and `wget -O /tmp/...` fails. Bumping it needs a host `docker build` and push, or a Dockerfile that forces the dir to materialise by writing a file inside it. [`build-kaniko-act.yml`](.gitea/workflows/build-kaniko-act.yml) is dispatch-only and runs in the old `mikkeli` copy.
|
|
|
|
## Gitea Actions setup (cross-cutting)
|
|
|
|
Secrets and variables are scoped to the `novoyuuparosk-wiki` org and inherited by every repo under it. A run belongs to the *caller* repo, so a source repo resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo.
|
|
|
|
**Secrets:**
|
|
|
|
| Name | Value / purpose |
|
|
|---|---|
|
|
| `WIKI_BOT_USER` | `Mikkeli@giteaBot` |
|
|
| `WIKI_BOT_PASSWORD` | the BotPassword from *Bot identity* above |
|
|
| `FAPAT` | Full-Access PAT owned by `mikkeli`, used by the publish workflows to clone repos at runtime. The PAT is personal; only its storage scope is the org — hence the `mikkeli:` basic-auth username in clone URLs |
|
|
| `PKGRW_PAT` | package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry |
|
|
|
|
**Variables:**
|
|
|
|
| Name | Value | Used for |
|
|
|---|---|---|
|
|
| `WIKI_BASE_URL` | `https://wiki.novoyuuparosk.org` | wiki base |
|
|
| `WIKI_API_URL` | `https://wiki.novoyuuparosk.org/api.php` | MediaWiki action API |
|
|
| `URL_TO_GITEA` | `http://localhost:3005` | **runner-internal** address, cloning only |
|
|
| `PUBLIC_URL_TO_GITEA` | `https://git.novoyuuparosk.org` | **public** address, page banner source links only |
|
|
|
|
Names avoid a `GITEA_`/`GITHUB_` prefix because Gitea blocks those, hence `URL_TO_GITEA` rather than `GITEA_URL`.
|
|
|
|
The two URL variables must stay separate. `URL_TO_GITEA` has to be an address the runner container can clone from, and host networking makes that a loopback. `PUBLIC_URL_TO_GITEA` is baked into the `{{Auto-generated}}` banner of every published page and has to resolve for a reader on the internet. `lib/wiki.public_source_base()` validates the banner value and aborts the run on an unset, relative, loopback, mDNS, or private-range address, so collapsing them back into one fails loudly instead of publishing dead links.
|
|
|
|
## Branch naming
|
|
|
|
- This repo: `automation/<pipeline-name>` for pipeline-development branches (e.g. `automation/songs`).
|
|
- Source repos: each pipeline's README defines its own convention (e.g. `autowiki/<song-slug>` in `ncmr-songs`).
|
|
|
|
## Tests
|
|
|
|
```
|
|
pip install -r requirements-dev.txt
|
|
python -m pytest tests -q
|
|
```
|
|
|
|
Pure string-in/string-out coverage of `lib/` and the pipelines' discovery, validation and publish-scoping logic — no wiki, no network. The `TestPandocContract` cases shell out to real pandoc and are skipped if it isn't installed; they pin the pandoc output shape the link resolver depends on, so a pandoc upgrade fails a test instead of publishing broken links. [`test.yml`](.gitea/workflows/test.yml) runs the suite on push in the same image the pipelines publish from.
|