Compare commits
4 Commits
162c053377
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
| 3ab59b493e | |||
| f19538b607 | |||
| 9c293f75c3 | |||
| e1810facd6 |
@@ -34,7 +34,7 @@ jobs:
|
||||
WIKI_BOT_USER: ${{ secrets.WIKI_BOT_USER }}
|
||||
WIKI_BOT_PASSWORD: ${{ secrets.WIKI_BOT_PASSWORD }}
|
||||
SOURCE_REF: ${{ inputs.SOURCE_REF }}
|
||||
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/novoyuuparosk-wiki/ses-light-novel
|
||||
GITEA_REPO_URL: ${{ vars.PUBLIC_URL_TO_GITEA }}/novoyuuparosk-wiki/ses-light-novel
|
||||
run: |
|
||||
python auto-wiki/pipelines/ses/publish.py \
|
||||
--source-dir ses-light-novel \
|
||||
|
||||
@@ -62,7 +62,7 @@ jobs:
|
||||
WIKI_BOT_USER: ${{ secrets.WIKI_BOT_USER }}
|
||||
WIKI_BOT_PASSWORD: ${{ secrets.WIKI_BOT_PASSWORD }}
|
||||
SOURCE_REF: ${{ inputs.SOURCE_REF }}
|
||||
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/novoyuuparosk-wiki/ncmr-songs
|
||||
GITEA_REPO_URL: ${{ vars.PUBLIC_URL_TO_GITEA }}/novoyuuparosk-wiki/ncmr-songs
|
||||
MODE: ${{ steps.diff.outputs.mode }}
|
||||
FILES: ${{ steps.diff.outputs.files }}
|
||||
run: |
|
||||
|
||||
@@ -34,7 +34,7 @@ jobs:
|
||||
WIKI_BOT_USER: ${{ secrets.WIKI_BOT_USER }}
|
||||
WIKI_BOT_PASSWORD: ${{ secrets.WIKI_BOT_PASSWORD }}
|
||||
SOURCE_REF: ${{ inputs.SOURCE_REF }}
|
||||
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/novoyuuparosk-wiki/tech-blogs
|
||||
GITEA_REPO_URL: ${{ vars.PUBLIC_URL_TO_GITEA }}/novoyuuparosk-wiki/tech-blogs
|
||||
run: |
|
||||
python auto-wiki/pipelines/tech/publish.py \
|
||||
--source-dir tech-blogs \
|
||||
|
||||
@@ -10,28 +10,36 @@ CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from u
|
||||
| [`pipelines/ses/`](pipelines/ses/) | `novoyuuparosk-wiki/ses-light-novel` | SES light novel pages | v1 in development |
|
||||
| [`pipelines/tech/`](pipelines/tech/) | `novoyuuparosk-wiki/tech-blogs` | Tech blog posts | v1 live |
|
||||
|
||||
Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime, decisions). This root README covers only what's cross-cutting.
|
||||
Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime). This root README covers only what's cross-cutting.
|
||||
|
||||
## Architecture
|
||||
|
||||
Hybrid layout. The Gitea Actions trigger must live in the source repo (Gitea only fires workflows from `.gitea/workflows/` of the pushed-to repo); the rendering logic lives here. The source-repo workflow is self-contained but clones this repo at runtime to get the renderer.
|
||||
Split across two repos, because Gitea only fires workflows from `.gitea/workflows/` of the repo that was pushed to, while the rendering logic belongs here.
|
||||
|
||||
```
|
||||
<source-repo>/
|
||||
.gitea/workflows/<name>.yml <- workflow: clones this repo, runs renderer (deps pre-baked in job image)
|
||||
.gitea/workflows/publish.yml <- thin caller: on: push, uses: the reusable workflow below
|
||||
|
||||
novoyuuparosk-auto-wiki/ <- this repo
|
||||
.gitea/workflows/<pipeline>.yml <- reusable workflow stubs (kept for reference; not actively called; may be stale)
|
||||
pipelines/<pipeline>/ <- per-pipeline code, schema, templates
|
||||
lib/ <- shared modules (MediaWiki client, etc.)
|
||||
.gitea/workflows/publish-<p>.yml <- on: workflow_call. Clones both repos, runs the renderer
|
||||
.gitea/workflows/test.yml <- on: push. Runs tests/ in the publish runner image
|
||||
.gitea/workflows/build-image.yml <- builds and pins the shared job container image
|
||||
pipelines/<pipeline>/ <- per-pipeline renderer, schema, templates
|
||||
lib/ <- shared render and MediaWiki helpers
|
||||
tests/ <- pytest suite for lib/ and the pipelines
|
||||
```
|
||||
|
||||
Note: `workflow_call` across private repos was abandoned — the auto-generated run token is scoped to the triggering repo only and cannot clone a private callee. The source-repo workflow clones this repo directly using `FAPAT`.
|
||||
The two halves reach the runner by different routes, which matters when a change appears not to take effect:
|
||||
|
||||
- **Renderer code** (`pipelines/`, `lib/`) is a plain `git clone` of this repo's **default branch** on every run. Always current, never cached.
|
||||
- **Workflow YAML** (`publish-*.yml`) is resolved through the caller's `uses:` and **is cached** — see *Callee workflow cache* below.
|
||||
|
||||
**This repo must stay public.** `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied, and shared ownership does not satisfy the read requirement. A private callee 404s with `repository not found`. The alternative, untested here, is Settings → Actions → General → collaborative owners (Gitea 1.26+).
|
||||
|
||||
## Wiki
|
||||
|
||||
- Base URL: https://wiki.novoyuuparosk.org
|
||||
- MediaWiki API: https://wiki.novoyuuparosk.org/api.php *(confirmed)*
|
||||
- MediaWiki API: https://wiki.novoyuuparosk.org/api.php (classic action API)
|
||||
|
||||
## Bot identity
|
||||
|
||||
@@ -39,84 +47,72 @@ MediaWiki BotPassword issued for user `Mikkeli`, bot name `giteaBot`. Login form
|
||||
|
||||
`Mikkeli` holds the `bot` right, so pipeline edits are flagged as bot edits and stay out of default Recent Changes — `mwclient`'s `page.save()` requests the bot flag and the wiki honours it.
|
||||
|
||||
This section previously named `Dubrowski@giteaAutomaton`, but every bot-flagged revision in the wiki's history is attributed to `Mikkeli`, so that had been stale for some time — the live secret never matched the doc.
|
||||
|
||||
Credentials are stored in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo.
|
||||
Credentials live in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo.
|
||||
|
||||
## Runner infrastructure
|
||||
|
||||
One `act_runner` instance serves all pipelines. Runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) inside the same `docker-compose` stack that hosts the Gitea instance. Job execution is via the host Docker socket — runner is a container, jobs spawn as sibling containers.
|
||||
One `act_runner` instance serves all pipelines. It runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) as a service in the same `docker-compose` stack as Gitea, built `linux-arm64` from the `gitea/act_runner` image. Job execution is via the host Docker socket — the runner is a container and jobs spawn as sibling containers.
|
||||
|
||||
`act_runner` build: `linux-arm64`, from the `gitea/act_runner` Docker image.
|
||||
Container network mode is `host`, so job containers can reach Gitea on `localhost:3005` and resolve mDNS hostnames.
|
||||
|
||||
Container network mode: `host` — required so job containers can reach `localhost:3005` (Gitea) and resolve mDNS hostnames.
|
||||
### Callee workflow cache
|
||||
|
||||
`act_runner` caches each resolved cross-repo callee at `/root/.cache/act/<owner>-<repo>@<ref>` inside the runner container, and **never refreshes it per run**. The key contains no commit SHA, so pushing to `master` does not invalidate it: edits to `publish-*.yml` — including the pin job's own commits — keep executing the stale cached copy indefinitely.
|
||||
|
||||
Two consequences. When a pipeline misbehaves after a workflow edit, suspect the cache before suspecting permissions. And after any `publish-*.yml` change, flush it:
|
||||
|
||||
```
|
||||
docker compose up -d --no-deps --force-recreate act_runner
|
||||
```
|
||||
|
||||
`--no-deps` is required: `act_runner` declares `depends_on: gitea`, so without it the recreate cascades and bounces the Gitea instance. Pass no `--pull`, or the flush silently adopts whatever `act_runner:latest` has become. The cache sits in the container layer while `.runner` is in the `/data` volume, so registration survives the recreate.
|
||||
|
||||
Changing a repo's owner also changes the cache key, which flushes it as a side effect.
|
||||
|
||||
### Job container image
|
||||
|
||||
All pipelines share a single pre-built Docker image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The `Dockerfile` is at the repo root. It bakes in system deps (git, pandoc, ca-certificates) and all pipeline Python packages so job containers start instantly with no install steps.
|
||||
All pipelines share one pre-built image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The root `Dockerfile` bakes in system deps (git, pandoc, ca-certificates) and every pipeline's Python packages, so job containers start with no install step.
|
||||
|
||||
The image builds automatically via [`.gitea/workflows/build-image.yml`](.gitea/workflows/build-image.yml), which triggers on pushes that touch the `Dockerfile`, any pipeline `requirements.txt`, or that workflow itself. It uses kaniko (daemonless, unprivileged) to build and push two tags: an immutable `:<short-sha>` and a moving `:latest`.
|
||||
[`build-image.yml`](.gitea/workflows/build-image.yml) rebuilds it on pushes touching the `Dockerfile`, any pipeline `requirements.txt`, or itself. It uses kaniko (daemonless, unprivileged) to push an immutable `:<short-sha>` and a moving `:latest`. A follow-up `pin` job rewrites the `image:` pin in each `publish-*.yml` to the new short SHA and commits it back to `master` using `FAPAT`, so the publish workflows always reference an immutable tag with no manual bump. That pin commit touches only `workflow_call` files, so it triggers no further runs — but it does need a cache flush to take effect.
|
||||
|
||||
A follow-up `pin` job then rewrites the `image:` pin in each `publish-*.yml` to the new `:<short-sha>` and commits it back to `master` (using `FAPAT` for contents write). The publish workflows therefore always reference an immutable tag, kept current automatically — no manual bump. The pin commit only touches `workflow_call` files, so it triggers no further runs.
|
||||
The builder image `kaniko-act` lives under `mikkeli`, not the org, and **cannot be rebuilt by `build-image.yml`**: kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so the resulting image has no `/tmp` and `wget -O /tmp/...` fails. Bumping it needs a host `docker build` and push, or a Dockerfile that forces the dir to materialise by writing a file inside it. [`build-kaniko-act.yml`](.gitea/workflows/build-kaniko-act.yml) is dispatch-only and runs in the old `mikkeli` copy.
|
||||
|
||||
## Gitea Actions setup (cross-cutting)
|
||||
|
||||
Secrets and variables are scoped to the `novoyuuparosk-wiki` org, inherited by all repos under it. Runs belong to the *caller* repo, so a source repo calling a reusable workflow here resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo.
|
||||
Secrets and variables are scoped to the `novoyuuparosk-wiki` org and inherited by every repo under it. A run belongs to the *caller* repo, so a source repo resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo.
|
||||
|
||||
**Secrets:**
|
||||
- `WIKI_BOT_USER` = `Mikkeli@giteaBot`
|
||||
- `WIKI_BOT_PASSWORD` = the value from *Bot identity* above
|
||||
- `FAPAT` = Full-Access PAT owned by `mikkeli`, used by source-repo workflows to clone this repo at runtime. The PAT stays personal; only its storage scope moved to the org — hence the `mikkeli:` basic-auth username in the clone URLs.
|
||||
- `PKGRW_PAT` = package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry
|
||||
|
||||
| Name | Value / purpose |
|
||||
|---|---|
|
||||
| `WIKI_BOT_USER` | `Mikkeli@giteaBot` |
|
||||
| `WIKI_BOT_PASSWORD` | the BotPassword from *Bot identity* above |
|
||||
| `FAPAT` | Full-Access PAT owned by `mikkeli`, used by the publish workflows to clone repos at runtime. The PAT is personal; only its storage scope is the org — hence the `mikkeli:` basic-auth username in clone URLs |
|
||||
| `PKGRW_PAT` | package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry |
|
||||
|
||||
**Variables:**
|
||||
- `WIKI_BASE_URL` = `https://wiki.novoyuuparosk.org`
|
||||
- `WIKI_API_URL` = `https://wiki.novoyuuparosk.org/api.php`
|
||||
- `URL_TO_GITEA` = Gitea instance base URL (e.g. `http://localhost:3005`). Named with `URL_TO_` prefix — Gitea blocks variable names starting with `GITEA_` or `GITHUB_`.
|
||||
|
||||
| Name | Value | Used for |
|
||||
|---|---|---|
|
||||
| `WIKI_BASE_URL` | `https://wiki.novoyuuparosk.org` | wiki base |
|
||||
| `WIKI_API_URL` | `https://wiki.novoyuuparosk.org/api.php` | MediaWiki action API |
|
||||
| `URL_TO_GITEA` | `http://localhost:3005` | **runner-internal** address, cloning only |
|
||||
| `PUBLIC_URL_TO_GITEA` | `https://git.novoyuuparosk.org` | **public** address, page banner source links only |
|
||||
|
||||
Names avoid a `GITEA_`/`GITHUB_` prefix because Gitea blocks those, hence `URL_TO_GITEA` rather than `GITEA_URL`.
|
||||
|
||||
The two URL variables must stay separate. `URL_TO_GITEA` has to be an address the runner container can clone from, and host networking makes that a loopback. `PUBLIC_URL_TO_GITEA` is baked into the `{{Auto-generated}}` banner of every published page and has to resolve for a reader on the internet. `lib/wiki.public_source_base()` validates the banner value and aborts the run on an unset, relative, loopback, mDNS, or private-range address, so collapsing them back into one fails loudly instead of publishing dead links.
|
||||
|
||||
## Branch naming
|
||||
|
||||
- This repo: `automation/<pipeline-name>` for pipeline-development branches (e.g., `automation/songs`).
|
||||
- Source repos: each pipeline's README defines the source-side branch convention (e.g., `autowiki/<song-slug>` in `ncmr-songs`).
|
||||
- This repo: `automation/<pipeline-name>` for pipeline-development branches (e.g. `automation/songs`).
|
||||
- Source repos: each pipeline's README defines its own convention (e.g. `autowiki/<song-slug>` in `ncmr-songs`).
|
||||
|
||||
## Decisions log (cross-cutting)
|
||||
## Tests
|
||||
|
||||
| Decision | Value | Date |
|
||||
|---|---|---|
|
||||
| Architecture | Hybrid: source-repo workflow clones this repo at runtime for the renderer | 2026-06-09 |
|
||||
| Workflow pattern | Self-contained (not `workflow_call`) — cross-repo `workflow_call` blocked by token scoping on private repos | 2026-06-09 |
|
||||
| Runner execution | Docker, added as a service to the existing Gitea docker-compose | 2026-06-09 |
|
||||
| Runner network mode | `host` — job containers need to reach Gitea on localhost | 2026-06-09 |
|
||||
| Secret/runner scope | User-level on `mikkeli` (no orgs on this instance) | 2026-06-09 |
|
||||
| Ownership | This repo and all three source repos moved to the `novoyuuparosk-wiki` org. Secrets/variables re-created at org scope; runner re-registered instance-level so it serves org-owned runs | 2026-08-11 |
|
||||
| **This repo must stay public** | `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied. A private callee therefore 404s with `repository not found`, regardless of the caller sharing its owner. Shared ownership does **not** satisfy the read requirement. Alternative if it must be private again: Settings → Actions → General → collaborative owners (Gitea 1.26+), untested here | 2026-08-11 |
|
||||
| Runner cache masks this | The resolved callee is cached at `/root/.cache/act/<owner>-<repo>@<ref>`. Changing owner changes the key, so a working pipeline can break on a clone that had been served from cache for months. Suspect the cache before suspecting permissions | 2026-08-11 |
|
||||
| **Callee edits need a cache flush** | The cache is *not* refreshed per run. Edits to `publish-*.yml` on `master` — including the pin job's own commits — keep executing the stale cached copy until the key changes or the cache is cleared. Observed directly: a run used the previous namespace and tag despite `master` being current. Flush with `docker compose up -d --force-recreate act_runner`; the cache is in the container layer, while `.runner` is in the `/data` volume, so registration survives | 2026-08-11 |
|
||||
| kaniko cannot rebuild kaniko-act | The org rebuild exits 0 but the resulting image has no `/tmp` — kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so `wget -O /tmp/...` fails. The bootstrap note's claim that version bumps "can be built by the build-image workflow itself" does not hold as written. `kaniko-act` therefore still lives under `mikkeli`; a real bump needs a host `docker build` + push, or a Dockerfile that forces the dir to materialise (e.g. writing a file inside it) | 2026-08-11 |
|
||||
| Container registry | Gitea cannot transfer packages, so images were *rebuilt* into `pi5-16.local:3005/novoyuuparosk-wiki/*` rather than moved. `kaniko-act` bootstrapped via the new dispatch-only `build-kaniko-act.yml`, running in the old `mikkeli` copy; `novoyuuparosk-wiki-runner` came from a normal `build-image.yml` run. Registry auth is still the `mikkeli`-owned `PKGRW_PAT`, hence the `mikkeli:` username in the auth blob | 2026-08-11 |
|
||||
| Old images left in place | The `mikkeli/*` package versions are orphaned but retained — nothing references them, and deleting a container version is irreversible. Safe to purge once the org images have proven themselves | 2026-08-11 |
|
||||
| MediaWiki API path | `api.php` (classic action API) | 2026-06-09 |
|
||||
| Branch naming (this repo) | `automation/<pipeline>` for pipeline-development branches | 2026-06-09 |
|
||||
| Variable naming | `URL_TO_GITEA` not `GITEA_URL` — Gitea blocks `GITEA_`/`GITHUB_` prefixes | 2026-06-09 |
|
||||
| Columns shorthand | Side-by-side columns authored as a ` ```columns ` fenced block, expanded post-Pandoc in shared `lib/wiki.py` (universal across pipelines). No wiki template or PHP extension — runs Pi-side before the API call | 2026-06-14 |
|
||||
```
|
||||
pip install -r requirements-dev.txt
|
||||
python -m pytest tests -q
|
||||
```
|
||||
|
||||
Per-pipeline decisions live in each pipeline's README.
|
||||
|
||||
## Setup checklist (cross-cutting)
|
||||
|
||||
Via the Gitea web UI logged in as `mikkeli` (an owner of `novoyuuparosk-wiki`):
|
||||
|
||||
- [x] Org-scoped secrets and variables set per *Gitea Actions setup* above
|
||||
- [x] `WIKI_BOT_USER`
|
||||
- [x] `WIKI_BOT_PASSWORD`
|
||||
- [x] `FAPAT` (Full-Access PAT — value not stored in this README; saved directly into the Gitea secret. Regenerate if lost.)
|
||||
- [x] `WIKI_BASE_URL`
|
||||
- [x] `WIKI_API_URL`
|
||||
- [x] `URL_TO_GITEA`
|
||||
|
||||
With Pi access:
|
||||
|
||||
- [x] Add `act_runner` service to the existing Gitea docker-compose
|
||||
- [x] Generate a runner registration token at `/-/admin/actions/runners`, bake into the compose env, `docker compose up -d act_runner`, confirm "online" in the Gitea UI
|
||||
|
||||
Per-pipeline setup lives in each pipeline's README. Start with [`pipelines/songs/`](pipelines/songs/).
|
||||
Pure string-in/string-out coverage of `lib/` and the pipelines' discovery, validation and publish-scoping logic — no wiki, no network. The `TestPandocContract` cases shell out to real pandoc and are skipped if it isn't installed; they pin the pandoc output shape the link resolver depends on, so a pandoc upgrade fails a test instead of publishing broken links. [`test.yml`](.gitea/workflows/test.yml) runs the suite on push in the same image the pipelines publish from.
|
||||
|
||||
+58
@@ -1,6 +1,7 @@
|
||||
"""Shared utilities for novoyuuparosk-auto-wiki pipelines."""
|
||||
|
||||
import html
|
||||
import ipaddress
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
@@ -199,6 +200,63 @@ def markdown_to_wikitext(body: str) -> str:
|
||||
return expand_columns(result.stdout)
|
||||
|
||||
|
||||
_LOOPBACK_HOSTNAMES = frozenset({"localhost", "localhost.localdomain", "ip6-localhost"})
|
||||
|
||||
|
||||
def public_source_base(env_var: str = "GITEA_REPO_URL") -> str:
|
||||
"""The public base URL for the source links baked into published pages.
|
||||
|
||||
This URL ends up in the `{{Auto-generated}}` banner of every page, so it is
|
||||
read by anyone on the internet — it is *not* the address the runner clones
|
||||
from. Those two are different hosts here: the runner container uses host
|
||||
networking and reaches Gitea on `localhost:3005`, while a reader needs the
|
||||
public name. Composing the banner from the clone address published pages
|
||||
citing `localhost:3005`, which is why this is validated rather than trusted.
|
||||
|
||||
Rejects an address that cannot possibly work for a reader — unset, missing a
|
||||
scheme or host, loopback, mDNS (`*.local`), or a private IP range — so the
|
||||
misconfiguration fails the run instead of being written to the wiki.
|
||||
"""
|
||||
raw = os.environ.get(env_var, "").strip().rstrip("/")
|
||||
if not raw:
|
||||
raise RuntimeError(
|
||||
f"{env_var} is unset. It must be the *public* Gitea base URL plus the "
|
||||
f"repository path (e.g. https://git.example.org/org/repo); the workflow "
|
||||
f"composes it from the PUBLIC_URL_TO_GITEA Gitea variable."
|
||||
)
|
||||
|
||||
parsed = urlparse(raw)
|
||||
if not parsed.scheme or not parsed.netloc:
|
||||
raise RuntimeError(
|
||||
f"{env_var} is not an absolute URL: {raw!r}. Expected scheme and host, "
|
||||
f"e.g. https://git.example.org/org/repo."
|
||||
)
|
||||
|
||||
host = (parsed.hostname or "").lower()
|
||||
reason = None
|
||||
if host in _LOOPBACK_HOSTNAMES:
|
||||
reason = "a loopback hostname"
|
||||
elif host.endswith(".local"):
|
||||
reason = "an mDNS name, resolvable only on the local network"
|
||||
else:
|
||||
try:
|
||||
address = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
address = None
|
||||
if address is not None and (address.is_loopback or address.is_private):
|
||||
reason = "a loopback or private-range IP address"
|
||||
|
||||
if reason:
|
||||
raise RuntimeError(
|
||||
f"{env_var} points at {reason}: {raw!r}. This URL is published in the "
|
||||
f"page banner and must resolve for a reader off the network. Set the "
|
||||
f"PUBLIC_URL_TO_GITEA Gitea variable to the public Gitea base URL — "
|
||||
f"URL_TO_GITEA is the runner-internal clone address and is not usable here."
|
||||
)
|
||||
|
||||
return raw
|
||||
|
||||
|
||||
def connect_wiki() -> mwclient.Site:
|
||||
api_url = os.environ["WIKI_API_URL"]
|
||||
parsed = urlparse(api_url)
|
||||
|
||||
+15
-29
@@ -5,7 +5,7 @@ Publishes SES light novel pages from `novoyuuparosk-wiki/ses-light-novel` to `ht
|
||||
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
|
||||
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
|
||||
|
||||
## v1 scope
|
||||
## What the renderer does
|
||||
|
||||
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
|
||||
- `SES:` title prefix — all pages live in the SES MediaWiki namespace
|
||||
@@ -26,7 +26,7 @@ Not in scope (intentional):
|
||||
|
||||
Source repo: `novoyuuparosk-wiki/ses-light-novel`. The pipeline triggers on push to `master`.
|
||||
|
||||
No branch convention enforced — work directly on `master` or use whatever branch workflow suits.
|
||||
No branch convention enforced — work directly on `master` or use whatever branch workflow suits. There is no `wip/` exclusion; `wiki.publish: true` is the only publishing gate.
|
||||
|
||||
**Excluded paths** (won't trigger the pipeline):
|
||||
|
||||
@@ -34,12 +34,18 @@ No branch convention enforced — work directly on `master` or use whatever bran
|
||||
|
||||
## Invocation
|
||||
|
||||
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ses-light-novel` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
|
||||
`novoyuuparosk-wiki/ses-light-novel/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-ses.yml`](../../.gitea/workflows/publish-ses.yml) here, which clones both repos and runs `publish.py`.
|
||||
|
||||
Editing `publish-ses.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md).
|
||||
|
||||
Triggers:
|
||||
- `push` to `master` (path-filtered as above)
|
||||
- `workflow_dispatch` — manual trigger from the Gitea Actions UI
|
||||
|
||||
### Publish scope
|
||||
|
||||
Every run publishes every file with `wiki.publish: true`; there is no diff detection. Writes are idempotent — the banner carries the commit SHA rather than a timestamp, so a re-run at the same commit rewrites nothing.
|
||||
|
||||
### Environment (secrets and variables)
|
||||
|
||||
| Name | Kind | Source | Purpose |
|
||||
@@ -48,36 +54,16 @@ Triggers:
|
||||
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
|
||||
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login |
|
||||
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
|
||||
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Gitea instance base URL for cloning |
|
||||
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Runner-internal Gitea base URL, for cloning only |
|
||||
| `PUBLIC_URL_TO_GITEA` | variable | Gitea (cross-cutting) | Public Gitea base URL, for the banner source link only |
|
||||
| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
|
||||
| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary |
|
||||
| `GITEA_REPO_URL` | env (set in workflow) | composed from vars | Source-file URL base for the banner link |
|
||||
| `GITEA_REPO_URL` | env (set in workflow) | `PUBLIC_URL_TO_GITEA` + repo path | Source-file URL base for the banner link. Validated by `public_source_base()`: a loopback, mDNS or private-range host aborts the run rather than publishing an unreachable link |
|
||||
|
||||
## Dependencies
|
||||
|
||||
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
|
||||
Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild. Shared render and MediaWiki helpers live in `lib/wiki.py` at the repo root.
|
||||
|
||||
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
|
||||
## Wiki-side prerequisites
|
||||
|
||||
Shared code lives in `lib/wiki.py` (repo root).
|
||||
|
||||
## One-off wiki setup
|
||||
|
||||
- [ ] **`Template:Auto-generated`** — must exist on the wiki (shared with songs pipeline; already created).
|
||||
|
||||
## Decisions log (ses pipeline)
|
||||
|
||||
| Decision | Value | Date |
|
||||
|---|---|---|
|
||||
| Namespace | All pages prefixed `SES:` — auto-applied by renderer, not written in source `title:` | 2026-06-10 |
|
||||
| Type resolution | `type:` frontmatter field; falls back to immediate parent directory name | 2026-06-10 |
|
||||
| Categories | `Category:SES` + `Category:SES:<type>` on every page | 2026-06-10 |
|
||||
| Sibling links | Not implemented — write `[[SES:Page title]]` wikilinks directly in the body | 2026-06-10 |
|
||||
| WIP exclusion | None — `wiki.publish: false` is the only gate | 2026-06-10 |
|
||||
| Publish mode | Always `--all` | 2026-06-10 |
|
||||
| Idempotency | Commit SHA in banner, not timestamp | 2026-06-10 |
|
||||
| Manual-page protection | Bot skips pages without `{{Auto-generated` banner | 2026-06-10 |
|
||||
|
||||
## Status
|
||||
|
||||
v1 in development.
|
||||
**`Template:Auto-generated`** must exist on the wiki. It is shared across all three pipelines; the wikitext source lives in [`pipelines/songs/template_auto_generated.wikitext`](../songs/template_auto_generated.wikitext).
|
||||
|
||||
@@ -10,7 +10,12 @@ import frontmatter
|
||||
import mwclient
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
from lib.wiki import AUTO_BANNER_PREFIX, connect_wiki, markdown_to_wikitext
|
||||
from lib.wiki import (
|
||||
AUTO_BANNER_PREFIX,
|
||||
connect_wiki,
|
||||
markdown_to_wikitext,
|
||||
public_source_base,
|
||||
)
|
||||
|
||||
|
||||
def validate(fm: dict, path: Path) -> None:
|
||||
@@ -59,7 +64,7 @@ def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, sou
|
||||
page_type = get_page_type(fm, abs_path, source_dir)
|
||||
|
||||
body_wikitext = markdown_to_wikitext(post.content)
|
||||
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
|
||||
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel.as_posix()}"
|
||||
page_content = build_wikitext(fm, body_wikitext, source_url, source_ref, page_type)
|
||||
|
||||
page = site.pages[title]
|
||||
@@ -111,7 +116,7 @@ def main() -> None:
|
||||
|
||||
source_dir = args.source_dir.resolve()
|
||||
source_ref = os.environ.get("SOURCE_REF", "")
|
||||
gitea_repo_url = os.environ.get("GITEA_REPO_URL", "").rstrip("/")
|
||||
gitea_repo_url = public_source_base()
|
||||
|
||||
if args.all:
|
||||
candidates = list(source_dir.rglob("*.md"))
|
||||
|
||||
+59
-80
@@ -1,107 +1,86 @@
|
||||
# pipelines/songs
|
||||
|
||||
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
|
||||
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`. v1, live.
|
||||
|
||||
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
|
||||
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
|
||||
|
||||
## v1 scope
|
||||
## What the renderer does
|
||||
|
||||
Implemented:
|
||||
- Parses YAML frontmatter per [SCHEMA.md](SCHEMA.md); files without `wiki.publish: true` are skipped silently
|
||||
- Strips the body's first-line `# Heading`, then renders markdown → wikitext with Pandoc
|
||||
- Expands ` ```columns ` fenced blocks into a flex row of `<poem>` columns (shared transform in `lib/wiki.py`)
|
||||
- Resolves relative Markdown links to sibling `.md` files into `[[wiki page title|label]]`, using the target's own `title` frontmatter as the page name
|
||||
- Prepends the `{{Auto-generated|source=…|commit=<sha>}}` banner
|
||||
- Uploads a declared `lrc` companion file when its SHA1 has changed, and appends a `[[Media:…]]` footer link
|
||||
- Injects `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set, plus one `[[Category:…]]` per `categories` entry. Category pages themselves are user-written and never overwritten
|
||||
- Writes via the MediaWiki bot API, skipping the write when the generated text already matches the page
|
||||
|
||||
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
|
||||
- Pandoc-based markdown → wikitext body rendering
|
||||
- First-line h1 stripping
|
||||
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
|
||||
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set); additional categories via `categories` list
|
||||
- LRC file upload (idempotent SHA1 check) + footer link
|
||||
- Inter-page link resolution: a relative Markdown link (`[label](notes.md)`) becomes `[[wiki page title|label]]`, resolved against a repo-wide index so it never depends on which files a push touched
|
||||
- Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact
|
||||
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
|
||||
- Files without `wiki.publish: true` silently skipped
|
||||
- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
|
||||
Deliberately not in scope: multi-language metadata (the body conveys what languages exist; the renderer doesn't introspect), and auto-generated album index pages (album category-page descriptions are written freehand).
|
||||
|
||||
Not in scope (intentional):
|
||||
## Failure modes
|
||||
|
||||
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
|
||||
- Auto-generated album index pages — user writes album category-page descriptions freeform
|
||||
Everything below aborts the run with a non-zero exit and the wiki untouched. Validation and rendering both complete before the first write, so a broken page never leaves the wiki half-updated.
|
||||
|
||||
| Condition | Result |
|
||||
|---|---|
|
||||
| Missing/empty `title`, non-boolean `wiki.publish`, malformed `release_date`, non-list `categories` | validation error |
|
||||
| Two publishable files claiming the same `title` | validation error, checked repo-wide |
|
||||
| A `.md` link whose target is missing, or exists but isn't published | render error |
|
||||
| A `.md` link that survives rendering unresolved (unrecognised shape, or written inside a `columns` fence) | render error |
|
||||
| `GITEA_REPO_URL` resolving to a loopback, mDNS or private-range host | run aborts before any wiki contact |
|
||||
|
||||
Two things degrade rather than fail: a missing or non-UTF-8 `lrc` file is skipped with a warning and the page still publishes; and a page whose existing wikitext does not start with `{{Auto-generated` is skipped with a warning, to avoid overwriting hand-written content. Delete the page or add the banner to hand it to the bot.
|
||||
|
||||
## Source repo and branch convention
|
||||
|
||||
Source repo: `novoyuuparosk-wiki/ncmr-songs`. The pipeline triggers on push to `master`.
|
||||
Source repo: `novoyuuparosk-wiki/ncmr-songs`, triggering on push to `master`.
|
||||
|
||||
**Branch convention in `ncmr-songs`**: per-song short-lived branches named `autowiki/<song-slug>` (e.g., `autowiki/pulse`). Create when staging edits, merge to master when ready to publish, delete after one cycle.
|
||||
**Branch convention there**: per-song short-lived branches named `autowiki/<song-slug>` (e.g. `autowiki/pulse`). Create when staging edits, merge to `master` when ready to publish, delete after one cycle.
|
||||
|
||||
**Excluded paths** (won't trigger the pipeline):
|
||||
|
||||
- `wip/**`
|
||||
- `README.md`, `.gitignore`, `.claude/**`
|
||||
**Paths that don't trigger a run**: `wip/**`, `README.md`, `.gitignore`, `.claude/**`. Note that `wip/**` and `README.md` are excluded by the renderer too, so they can never publish even on a run triggered by something else.
|
||||
|
||||
## Invocation
|
||||
|
||||
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
|
||||
`novoyuuparosk-wiki/ncmr-songs/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-songs.yml`](../../.gitea/workflows/publish-songs.yml) here, which clones both repos and runs `publish.py`.
|
||||
|
||||
Triggers:
|
||||
- `push` to `master` (path-filtered as above)
|
||||
- `workflow_dispatch` — manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit
|
||||
|
||||
### Environment (secrets and variables)
|
||||
- `push` to `master`, path-filtered as above
|
||||
- `workflow_dispatch` — manual run from the Gitea Actions UI. Passes an empty base ref, which the pipeline reads as "publish everything", so this is the way to re-render every page after a renderer or template change
|
||||
|
||||
| Name | Kind | Source | Purpose |
|
||||
|---|---|---|---|
|
||||
| `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
|
||||
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
|
||||
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Mikkeli@giteaBot`) |
|
||||
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
|
||||
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Gitea instance base URL for cloning |
|
||||
| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
|
||||
| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary |
|
||||
| `GITEA_REPO_URL` | env (set in workflow) | composed from vars | Source-file URL base for the banner link |
|
||||
Editing `publish-songs.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md).
|
||||
|
||||
### Publish scope
|
||||
|
||||
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every publishable file, because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff plus one hop in each link direction: any page a changed page links to (so a link can't point at an unpublished page), and any page that links to a changed page (so an edited `title` doesn't leave stale inbound links).
|
||||
|
||||
| Invocation | Publishes |
|
||||
|---|---|
|
||||
| `--all` | every publishable file. Used on first push and `workflow_dispatch` |
|
||||
| `--files <paths…>` | those files plus their one-hop link neighbours. Used for an ordinary push |
|
||||
|
||||
Because the banner carries the commit SHA, every page in a run's write set is rewritten on a new commit; a page pulled in only by the link closure is a no-op API read when the run repeats at the same SHA.
|
||||
|
||||
### Environment
|
||||
|
||||
| Name | Kind | Purpose |
|
||||
|---|---|---|
|
||||
| `WIKI_API_URL` | variable, org-scoped | MediaWiki action API endpoint |
|
||||
| `WIKI_BASE_URL` | variable, org-scoped | Wiki base URL |
|
||||
| `WIKI_BOT_USER` | secret, org-scoped | Bot login (`Mikkeli@giteaBot`) |
|
||||
| `WIKI_BOT_PASSWORD` | secret, org-scoped | BotPassword value |
|
||||
| `URL_TO_GITEA` | variable, org-scoped | Runner-internal Gitea base URL, for cloning only |
|
||||
| `PUBLIC_URL_TO_GITEA` | variable, org-scoped | Public Gitea base URL, for the banner source link only |
|
||||
| `FAPAT` | secret, org-scoped | Full-Access PAT for cloning at runtime |
|
||||
| `SOURCE_REF` | env, set in workflow from `github.sha` | Commit SHA, used in the banner and the edit summary |
|
||||
| `GITEA_REPO_URL` | env, set in workflow from `PUBLIC_URL_TO_GITEA` + repo path | Base of the banner source link. Validated by `public_source_base()` |
|
||||
|
||||
## Dependencies
|
||||
|
||||
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
|
||||
Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild.
|
||||
|
||||
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
|
||||
## Wiki-side prerequisites
|
||||
|
||||
## Modes
|
||||
|
||||
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every file with `wiki.publish: true` (not under `wip/`), because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page.
|
||||
|
||||
- `--all` — publish every publishable file. Used on the first push and on `workflow_dispatch`.
|
||||
- `--files <paths…>` — publish these files and their one-hop link neighbours. Used for an ordinary push.
|
||||
|
||||
Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read.
|
||||
|
||||
## One-off wiki setup
|
||||
|
||||
- [x] **`Template:Auto-generated`** — exists on the wiki. Wikitext source: `template_auto_generated.wikitext` in this directory.
|
||||
- [x] **Bot user in `bot` group** — grants `skipcaptcha`; verified working.
|
||||
|
||||
## Decisions log (songs pipeline)
|
||||
|
||||
| Decision | Value | Date |
|
||||
|---|---|---|
|
||||
| Metadata source | YAML frontmatter inside each `.md` (see [SCHEMA.md](SCHEMA.md)) | 2026-06-09 |
|
||||
| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
|
||||
| Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 |
|
||||
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
|
||||
| Album categories | Auto-injected `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set; category pages are user-written and not overwritten | 2026-06-09 |
|
||||
| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:<value>]]` | 2026-06-10 |
|
||||
| `album` optional | Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 |
|
||||
| ~~Sibling resolution~~ | ~~Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc~~ — **retired 2026-09-09**, see below | 2026-06-10 |
|
||||
| Inter-page links | Plain relative Markdown link (`[label](notes.md)`); target's own `title` frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the `wiki.siblings` bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch | 2026-09-09 |
|
||||
| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 |
|
||||
| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 |
|
||||
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
|
||||
| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
|
||||
| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 |
|
||||
| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 |
|
||||
| Tests | `tests/` run by `.gitea/workflows/test.yml` in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose | 2026-09-09 |
|
||||
| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 |
|
||||
| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 |
|
||||
| Side-by-side columns | Authored as a ` ```columns ` fenced block (separator `===`); Pandoc emits `<pre class="columns">`, expanded post-Pandoc into a flex `<div>` of `<poem>` columns. No wiki template or extension — all Pi-side before the API write. See [SCHEMA.md](SCHEMA.md#columns-side-by-side-shorthand) | 2026-06-14 |
|
||||
|
||||
## Status
|
||||
|
||||
v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org.
|
||||
- **`Template:Auto-generated`** must exist on the wiki. Wikitext source: [`template_auto_generated.wikitext`](template_auto_generated.wikitext) in this directory.
|
||||
- The bot user must be in the `bot` group, which grants `skipcaptcha`.
|
||||
|
||||
@@ -268,4 +268,4 @@ Files without a `wiki.publish: true` frontmatter field are silently skipped —
|
||||
|
||||
Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken.
|
||||
|
||||
Frontmatter keys the schema does not define are ignored, so a leftover `wiki.siblings` block from the retired macro system is inert rather than an error. Delete them; they do nothing.
|
||||
Frontmatter keys this schema does not define are ignored rather than rejected.
|
||||
|
||||
@@ -19,6 +19,7 @@ from lib.wiki import (
|
||||
connect_wiki,
|
||||
extract_md_link_targets,
|
||||
markdown_to_wikitext,
|
||||
public_source_base,
|
||||
resolve_md_links,
|
||||
strip_first_h1,
|
||||
)
|
||||
@@ -153,7 +154,7 @@ def render_page(abs_path: Path, source_dir: Path, post, source_ref: str, gitea_r
|
||||
|
||||
body_wikitext = markdown_to_wikitext(strip_first_h1(post.content))
|
||||
body_wikitext = resolve_md_links(body_wikitext, abs_path, title_index)
|
||||
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
|
||||
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel.as_posix()}"
|
||||
|
||||
lrc_filename = f"{title}.lrc" if fm.get("lrc") else None
|
||||
return build_wikitext(fm, body_wikitext, source_url, source_ref, lrc_filename)
|
||||
@@ -241,7 +242,7 @@ def main() -> None:
|
||||
|
||||
source_dir = args.source_dir.resolve()
|
||||
source_ref = os.environ.get("SOURCE_REF", "")
|
||||
gitea_repo_url = os.environ.get("GITEA_REPO_URL", "").rstrip("/")
|
||||
gitea_repo_url = public_source_base()
|
||||
|
||||
# The repository is always scanned in full, whatever gets published: link
|
||||
# resolution and title-collision checks must see every publishable file, or
|
||||
|
||||
+16
-31
@@ -5,7 +5,7 @@ Publishes tech blog posts from `novoyuuparosk-wiki/tech-blogs` to `https://wiki.
|
||||
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
|
||||
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
|
||||
|
||||
## v1 scope
|
||||
## What the renderer does
|
||||
|
||||
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
|
||||
- Pandoc-based markdown → wikitext body rendering (h1 elements preserved)
|
||||
@@ -18,9 +18,9 @@ Publishes tech blog posts from `novoyuuparosk-wiki/tech-blogs` to `https://wiki.
|
||||
|
||||
## Source repo
|
||||
|
||||
Source repo: `novoyuuparosk-wiki/tech-blogs` (not yet initialised — pipeline code is ready).
|
||||
Source repo: `novoyuuparosk-wiki/tech-blogs`, triggering on push to `master`, with `workflow_dispatch` for manual runs.
|
||||
|
||||
The pipeline triggers on push to `master` and supports `workflow_dispatch` for manual runs.
|
||||
No branch convention enforced. There is no `wip/` exclusion — `wiki.publish: true` is the only publishing gate.
|
||||
|
||||
**Excluded paths** (won't trigger the pipeline):
|
||||
|
||||
@@ -28,12 +28,18 @@ The pipeline triggers on push to `master` and supports `workflow_dispatch` for m
|
||||
|
||||
## Invocation
|
||||
|
||||
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/tech-blogs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
|
||||
`novoyuuparosk-wiki/tech-blogs/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-tech.yml`](../../.gitea/workflows/publish-tech.yml) here, which clones both repos and runs `publish.py`.
|
||||
|
||||
Editing `publish-tech.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md).
|
||||
|
||||
Triggers:
|
||||
- `push` to `master` (path-filtered as above)
|
||||
- `workflow_dispatch` — manual trigger from the Gitea Actions UI
|
||||
|
||||
### Publish scope
|
||||
|
||||
Every run publishes every file with `wiki.publish: true`; there is no diff detection. Writes are idempotent — the banner carries the commit SHA rather than a timestamp, so a re-run at the same commit rewrites nothing.
|
||||
|
||||
### Environment (secrets and variables)
|
||||
|
||||
| Name | Kind | Source | Purpose |
|
||||
@@ -42,37 +48,16 @@ Triggers:
|
||||
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
|
||||
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login |
|
||||
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
|
||||
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Gitea instance base URL for cloning |
|
||||
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Runner-internal Gitea base URL, for cloning only |
|
||||
| `PUBLIC_URL_TO_GITEA` | variable | Gitea (cross-cutting) | Public Gitea base URL, for the banner source link only |
|
||||
| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
|
||||
| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary |
|
||||
| `GITEA_REPO_URL` | env (set in workflow) | composed from vars | Source-file URL base for the banner link |
|
||||
| `GITEA_REPO_URL` | env (set in workflow) | `PUBLIC_URL_TO_GITEA` + repo path | Source-file URL base for the banner link. Validated by `public_source_base()`: a loopback, mDNS or private-range host aborts the run rather than publishing an unreachable link |
|
||||
|
||||
## Dependencies
|
||||
|
||||
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
|
||||
Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild. Shared render and MediaWiki helpers live in `lib/wiki.py` at the repo root.
|
||||
|
||||
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
|
||||
## Wiki-side prerequisites
|
||||
|
||||
Shared code lives in `lib/wiki.py` (repo root).
|
||||
|
||||
## One-off wiki setup
|
||||
|
||||
- [ ] **`Template:Auto-generated`** — must exist on the wiki (shared with all pipelines; already created).
|
||||
|
||||
## Decisions log (tech pipeline)
|
||||
|
||||
| Decision | Value | Date |
|
||||
|---|---|---|
|
||||
| No title prefix | Tech posts live at bare titles — no namespace needed | 2026-06-10 |
|
||||
| Fixed category | Every post gets `Category:Tech blog` | 2026-06-10 |
|
||||
| Year category | `Category:Blog:<year>` from `date` field; falls back to the execution year if no date | 2026-06-10 |
|
||||
| Tag categories | Each tag → `[[Category:<tag>]]` with no prefix | 2026-06-10 |
|
||||
| WIP exclusion | None — `wiki.publish: false` is the only gate | 2026-06-10 |
|
||||
| Publish mode | Always `--all` | 2026-06-10 |
|
||||
| Idempotency | Commit SHA in banner | 2026-06-10 |
|
||||
| Manual-page protection | Bot skips pages without `{{Auto-generated` banner | 2026-06-10 |
|
||||
| Source repo | `novoyuuparosk-wiki/tech-blogs` — pipeline ships before repo is initialised | 2026-06-10 |
|
||||
|
||||
## Status
|
||||
|
||||
v1 pipeline ready. Source repo (`novoyuuparosk-wiki/tech-blogs`) not yet initialised.
|
||||
**`Template:Auto-generated`** must exist on the wiki. It is shared across all three pipelines; the wikitext source lives in [`pipelines/songs/template_auto_generated.wikitext`](../songs/template_auto_generated.wikitext).
|
||||
|
||||
@@ -11,7 +11,12 @@ import frontmatter
|
||||
import mwclient
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
from lib.wiki import AUTO_BANNER_PREFIX, connect_wiki, markdown_to_wikitext
|
||||
from lib.wiki import (
|
||||
AUTO_BANNER_PREFIX,
|
||||
connect_wiki,
|
||||
markdown_to_wikitext,
|
||||
public_source_base,
|
||||
)
|
||||
|
||||
|
||||
def validate(fm: dict, path: Path) -> None:
|
||||
@@ -58,7 +63,7 @@ def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, sou
|
||||
title = str(fm["title"])
|
||||
|
||||
body_wikitext = markdown_to_wikitext(post.content)
|
||||
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
|
||||
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel.as_posix()}"
|
||||
page_content = build_wikitext(fm, body_wikitext, source_url, source_ref)
|
||||
|
||||
page = site.pages[title]
|
||||
@@ -110,7 +115,7 @@ def main() -> None:
|
||||
|
||||
source_dir = args.source_dir.resolve()
|
||||
source_ref = os.environ.get("SOURCE_REF", "")
|
||||
gitea_repo_url = os.environ.get("GITEA_REPO_URL", "").rstrip("/")
|
||||
gitea_repo_url = public_source_base()
|
||||
|
||||
if args.all:
|
||||
candidates = list(source_dir.rglob("*.md"))
|
||||
|
||||
@@ -15,6 +15,7 @@ from lib.wiki import (
|
||||
expand_columns,
|
||||
extract_md_link_targets,
|
||||
markdown_to_wikitext,
|
||||
public_source_base,
|
||||
resolve_md_links,
|
||||
strip_first_h1,
|
||||
)
|
||||
@@ -214,3 +215,72 @@ class TestPandocContract:
|
||||
wikitext = markdown_to_wikitext("```columns\n**A**\nl1\n===\n**B**\nl2\n```")
|
||||
assert wikitext.count("<poem>") == 2
|
||||
assert "'''A'''" in wikitext
|
||||
|
||||
|
||||
class TestPublicSourceBase:
|
||||
"""The banner URL is read by the public, so a runner-only address must fail.
|
||||
|
||||
Every one of these rejections was a page published with a link to
|
||||
localhost:3005, which is what the whole guard exists to prevent.
|
||||
"""
|
||||
|
||||
def test_accepts_a_public_https_url(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "https://git.novoyuuparosk.org/org/repo")
|
||||
assert public_source_base() == "https://git.novoyuuparosk.org/org/repo"
|
||||
|
||||
def test_accepts_a_public_host_with_a_port(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "http://git.novoyuuparosk.org:1551/org/repo")
|
||||
assert public_source_base() == "http://git.novoyuuparosk.org:1551/org/repo"
|
||||
|
||||
def test_strips_a_trailing_slash(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "https://git.example.org/org/repo/")
|
||||
assert public_source_base() == "https://git.example.org/org/repo"
|
||||
|
||||
def test_strips_surrounding_whitespace(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", " https://git.example.org/org/repo ")
|
||||
assert public_source_base() == "https://git.example.org/org/repo"
|
||||
|
||||
def test_unset_is_rejected(self, monkeypatch):
|
||||
monkeypatch.delenv("GITEA_REPO_URL", raising=False)
|
||||
with pytest.raises(RuntimeError, match="is unset"):
|
||||
public_source_base()
|
||||
|
||||
def test_empty_is_rejected(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "")
|
||||
with pytest.raises(RuntimeError, match="is unset"):
|
||||
public_source_base()
|
||||
|
||||
def test_a_bare_path_is_rejected(self, monkeypatch):
|
||||
# What an unset PUBLIC_URL_TO_GITEA leaves the workflow composing.
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "/novoyuuparosk-wiki/ncmr-songs")
|
||||
with pytest.raises(RuntimeError, match="not an absolute URL"):
|
||||
public_source_base()
|
||||
|
||||
def test_localhost_is_rejected(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "http://localhost:3005/org/repo")
|
||||
with pytest.raises(RuntimeError, match="loopback hostname"):
|
||||
public_source_base()
|
||||
|
||||
def test_loopback_ip_is_rejected(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "http://127.0.0.1:3005/org/repo")
|
||||
with pytest.raises(RuntimeError, match="loopback or private-range"):
|
||||
public_source_base()
|
||||
|
||||
def test_private_range_ip_is_rejected(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "http://192.168.1.20:3005/org/repo")
|
||||
with pytest.raises(RuntimeError, match="loopback or private-range"):
|
||||
public_source_base()
|
||||
|
||||
def test_mdns_name_is_rejected(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "http://pi5-16.local:3005/org/repo")
|
||||
with pytest.raises(RuntimeError, match="mDNS"):
|
||||
public_source_base()
|
||||
|
||||
def test_rejection_names_the_variable_to_set(self, monkeypatch):
|
||||
monkeypatch.setenv("GITEA_REPO_URL", "http://localhost:3005/org/repo")
|
||||
with pytest.raises(RuntimeError, match="PUBLIC_URL_TO_GITEA"):
|
||||
public_source_base()
|
||||
|
||||
def test_reads_the_named_variable(self, monkeypatch):
|
||||
monkeypatch.setenv("OTHER_BASE", "https://git.example.org/org/repo")
|
||||
assert public_source_base("OTHER_BASE") == "https://git.example.org/org/repo"
|
||||
|
||||
Reference in New Issue
Block a user