25 Commits

Author SHA1 Message Date
wan 162c053377 Merge branch 'automation/md-links': Markdown links replace the siblings macro
tests / pytest (push) Successful in 9s
Retires the wiki.siblings bill of materials in favour of ordinary relative
Markdown links, fixes the batch-scoped resolution that silently stripped
wikilinks on incremental pushes, and adds the test suite that should have
caught it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 16:28:46 +09:00
wan f5625cde98 songs: replace the wiki.siblings macro with plain Markdown links
The `wiki.siblings` bill of materials is gone. A related co-publication is
now nominated with an ordinary relative Markdown link — `[label](notes.md)` —
resolved at publish time against the target file's own `title` frontmatter.
No macro namespace, no per-file declaration, and the link still works in
Gitea's blob view and any editor preview.

The macro could not be made safe. Substitution was `str.replace(tag, …)` over
the whole rendered page: no word boundary, so a tag also occurring in lyrics or
a code block got rewritten; no collision check, so overlapping tags corrupted
each other in declaration order. SCHEMA.md documented the placeholder as
`[[TAG]]` while the code replaced the bare tag, so following the schema
produced `[[[[Title|label]]]]`.

Fixes a live corruption bug in the process. `sibling_map` was built only from
the files in the publish batch, and an unresolved sibling degraded to plain
text — so pushing `ses/pulse.md` alone republished it with its wikilink
silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the
workflow "always runs --all", which it never did on an ordinary push.

The repository is now always scanned in full, whatever a run publishes, so
link resolution and title-collision detection can never depend on which files
a push touched. Writes stay bounded to the push diff plus one hop in each link
direction: the page a changed page links to (so a link cannot point at an
unpublished page) and the pages linking to it (so an edited `title` does not
leave stale inbound links). Publishing is not instant; reading is cheap.

An unresolvable `.md` link is now a hard error, never graceful degradation, and
linking to a file still does not publish it — `wiki.publish: true` remains the
only gate. A leftover-link guard fails the run on any link shape the resolver
did not recognise, including a Markdown link written inside a verbatim
```columns fence, so a pandoc upgrade cannot silently publish literal text.

Also:

- render before write: every page in a run is rendered before the first is
  published, so a broken page leaves the wiki untouched instead of
  half-updated. `publish_one` is split into `render_page` (pure) and the write.
- pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale
  encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside
  subprocess's reader thread, `stdout` came back None, and the real error was
  lost behind a TypeError. Latent on the Debian runner via PEP 538 only.
- title collisions are detected repo-wide, not just within a push, and before
  any wiki contact.
- tests/, run by .gitea/workflows/test.yml in the publish runner image. The
  pandoc output contract the resolver depends on is now pinned by a test
  instead of by prose — which is how SCHEMA.md came to assert something false.
- drop an f-string with nothing to interpolate.
- quote the SOURCE_REF interpolation in the workflow instead of pasting an
  input straight into shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 16:24:36 +09:00
mikkeli 4e7d84d665 fix(ci): point kaniko-act builder at the only working copy
The org copy this workflow produced cannot build (no /tmp), so pointing
its own container at it made the workflow unrunnable.
2026-08-11 20:41:09 +09:00
mikkeli e4ae304954 docs: record act cache staleness and the kaniko-act /tmp limitation 2026-08-11 20:38:27 +09:00
wiki-runner image bot bdca12837e build: pin runner image to 4f134af 2026-08-11 11:35:27 +00:00
mikkeli 4f134af7bc fix(ci): build in the known-good kaniko-act
/ build (push) Successful in 1m5s
/ pin (push) Successful in 10s
The org rebuild exits 0 but produces an image without /tmp, so the
source fetch fails. Runner image still publishes to the org namespace;
only the builder stays put.
2026-08-11 20:34:07 +09:00
mikkeli 905a1051a8 ci: move container images to the org registry namespace
/ build (push) Failing after 2s
/ pin (push) Has been skipped
Build destination, both builder-image references and the three publish
pins now point at novoyuuparosk-wiki/*. The publish pins name a tag that
does not exist in the new namespace until this run's build job pushes it
and the pin job rewrites them, so publishes are briefly broken mid-run.
2026-08-11 20:32:45 +09:00
mikkeli 4919ce9450 ci: add dispatch-only kaniko-act build, targeting the org registry
Bootstrap step for moving container images off the personal namespace.
Builds using the existing mikkeli/kaniko-act; the org copy is what this
run produces.
2026-08-11 20:31:04 +09:00
mikkeli 03be97799d docs: record why this repo must stay public
Correcting a wrong claim in the previous commit: sharing an owner with
the caller does not grant read access to a private callee. act_runner
clones the reusable workflow anonymously, so the repo has to be public.

Also notes the act cache key, which hid this for months, and the stale
Dubrowski bot identity.
2026-08-11 20:26:33 +09:00
wiki-runner image bot 01901b3821 build: pin runner image to 6dc9f68 2026-08-11 10:48:50 +00:00
mikkeli 6dc9f68b12 chore: repoint pipeline at the novoyuuparosk-wiki org
/ build (push) Successful in 1m3s
/ pin (push) Successful in 2s
This repo and the three source repos moved from the `mikkeli` account
into the `novoyuuparosk-wiki` org. Rewrites cover repo paths only —
the `mikkeli:` HTTP basic-auth usernames stay, since FAPAT/PKGRW_PAT
remain personally owned, as do the `3005/mikkeli/*` image paths
(Gitea cannot transfer packages).

Bot identity moves to `Mikkeli@giteaBot`; both accounts hold the bot
right, so edits stay flagged as bot edits.
2026-08-11 19:47:35 +09:00
mikkeli 83a5d86115 Merge branch 'automation/columns-md-emphasis'
Convert Markdown emphasis (*italic*, **bold**, ***bold-italic***) to wikitext inside columns blocks.
2026-06-14 21:13:31 +09:00
mikkeli 2072db4980 feat: Markdown emphasis inside columns blocks
Inside a columns fenced block (verbatim, so Pandoc doesn't process
Markdown), convert *italic* / **bold** / ***bold-italic*** to wikitext
emphasis. Asterisk style, single line; precedence bold-italic > bold >
italic. Raw inline HTML still works via the existing HTML-unescape path.

Docs + example updated to use ** ** rather than <b>.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:13:31 +09:00
mikkeli 00527e46b1 Merge branch 'automation/columns-shorthand'
Side-by-side columns shorthand: a 'columns' fenced block (columns split by a line of ===) is expanded post-Pandoc in shared lib/wiki.py into a flex div of poem columns. Universal across pipelines; no wiki template or PHP extension.
2026-06-14 21:02:35 +09:00
mikkeli 34a278870d feat: side-by-side columns shorthand (```columns fence)
Authors write a ```columns fenced block (columns separated by a line of
===); Pandoc passes the body through verbatim as <pre class="columns">,
and a new post-Pandoc transform in lib.wiki.expand_columns expands it
into a flex <div> of <poem> columns. Runs entirely Pi-side before the
MediaWiki API write — no wiki template or PHP extension required.

The transform lives in shared lib/wiki.py (called from markdown_to_wikitext),
so it is universal across all pipelines. Stdlib only; no new deps.

Docs: SCHEMA.md author contract + songs/root decision logs.
Also ignore __pycache__/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:02:24 +09:00
mikkeli 736f40f953 Update README.md 2026-06-11 04:08:05 +00:00
wiki-runner image bot 73a8b5834a build: pin runner image to b2714ab 2026-06-11 04:07:09 +00:00
mikkeli b2714ab5af feat(ci): auto-pin publish workflows to freshly built runner image
/ build (push) Successful in 59s
/ pin (push) Successful in 10s
build-image.yml now pushes a moving :latest tag alongside :<short-sha>,
and a follow-up pin job (running in :latest) rewrites the image pin in all
three publish-*.yml to the new sha and commits it back to master with
FAPAT. Publish workflows keep immutable sha pins, kept current with no
manual bump. No rebuild loop: the pin commit only touches workflow_call
files. Root README image-flow section updated to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 13:05:43 +09:00
mikkeli bf96cd10e7 fix(tech): default Blog year category to execution year
Dateless posts silently lost their Blog:<year> category. build_wikitext
now falls back to datetime.now().year when no date is declared, so every
published post lands in a year category. Docs corrected accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 13:05:29 +09:00
mikkeli bb57f319c5 fix(ci): authenticate source fetch; repo is private again
/ build (push) Successful in 1m20s
Fetch uses the job's per-run auto-token (repo-scoped, nothing stored);
push moves to the PKGRW_PAT user secret, separating CI credentials
from the host's docker login token.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 12:17:43 +09:00
mikkeli e838d3b105 build: bump runner image to CI-built 217d633
First image produced by the kaniko build workflow; replaces the
manually pushed v0 bootstrap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 11:32:27 +09:00
mikkeli 217d633853 fix(ci): add /tmp to kaniko-act (r2)
/ build (push) Failing after 1s
kaniko's scratch image has no /tmp; the build script and act's
RUNNER_TEMP both assume it exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 11:20:59 +09:00
mikkeli 3d7af25aed fix(ci): wrap kaniko in act-compatible image
/ build (push) Failing after 1s
act starts job containers with entrypoint /bin/sleep, which kaniko's
scratch-based image lacks (busybox lives under /busybox). kaniko-act
adds the single missing symlink; bootstrap build is manual, documented
in the Dockerfile.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 11:18:17 +09:00
mikkeli 47e5127299 feat(ci): build and push runner image with kaniko
/ build (push) Failing after 12s
Daemonless image build inside the job container itself
(kaniko :debug). Source fetched via Gitea archive API, pushed to the
Gitea registry tagged with the short commit SHA. Triggered by changes
to Dockerfile, any pipeline requirements.txt, or this workflow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 11:13:11 +09:00
mikkeli 2103a423fe build: pin job container image to Gitea registry (v0)
Workflows now pull pi5-16.local:3005/mikkeli/novoyuuparosk-wiki-runner:v0
instead of relying on the host dockerd image cache. Registry is the
authority; act_runner pull_image flipped to true accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 10:27:41 +09:00
22 changed files with 1159 additions and 237 deletions
+102
View File
@@ -0,0 +1,102 @@
on:
push:
branches: [master]
paths:
- Dockerfile
- "pipelines/**/requirements.txt"
- .gitea/workflows/build-image.yml
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
container:
# Daemonless image builder: kaniko :debug plus the /bin/sleep symlink
# act needs for job-container PID 1 (see ci/Dockerfile.kaniko-act).
# No docker socket, no privileges.
# Deliberately the `mikkeli` copy: the org rebuild of kaniko-act loses
# /tmp (kaniko does not persist the empty dir), which breaks the wget
# below. See build-kaniko-act.yml. Do not repoint without testing.
image: pi5-16.local:3005/mikkeli/kaniko-act:v1.23.2-r2
steps:
- name: Fetch source, build, push
# kaniko image has no bash — busybox sh only
shell: sh
env:
# package r/w PAT (user-level secret) — registry push only
PKG_TOKEN: ${{ secrets.PKGRW_PAT }}
# per-run auto-token, read access to this repo only — source fetch
JOB_TOKEN: ${{ github.token }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
run: |
set -eu
HOST="${URL_TO_GITEA#http://}"
SHORT_SHA=$(echo "${GITHUB_SHA}" | cut -c1-7)
IMAGE="${HOST}/novoyuuparosk-wiki/novoyuuparosk-wiki-runner"
# Source via Gitea archive API (no git/node in this image; repo is
# private — authenticate with the job's own per-run token)
wget -qO /tmp/src.tar.gz \
"http://mikkeli:${JOB_TOKEN}@${HOST}/api/v1/repos/novoyuuparosk-wiki/novoyuuparosk-auto-wiki/archive/${GITHUB_SHA}.tar.gz"
mkdir -p /tmp/src
tar -xzf /tmp/src.tar.gz -C /tmp/src --strip-components=1
# Registry auth for the push
AUTH=$(printf '%s' "mikkeli:${PKG_TOKEN}" | base64 | tr -d '\n')
printf '{"auths":{"%s":{"auth":"%s"}}}' "${HOST}" "${AUTH}" \
> /kaniko/.docker/config.json
# --insecure: push target is the plain-HTTP Gitea registry.
# Base image pull (docker.io) stays HTTPS — no --insecure-pull.
# :latest is a convenience tag for internal CI (the pin job below
# runs in it). Publish workflows stay pinned to the immutable sha.
/kaniko/executor \
--context dir:///tmp/src \
--dockerfile /tmp/src/Dockerfile \
--destination "${IMAGE}:${SHORT_SHA}" \
--destination "${IMAGE}:latest" \
--insecure
echo "Pushed ${IMAGE}:${SHORT_SHA} and ${IMAGE}:latest"
pin:
needs: build
runs-on: ubuntu-latest
container:
# Freshly built runner image: git + GNU sed baked in, and it has no
# non-shell ENTRYPOINT, so it works as a job container as-is (unlike the
# kaniko image above). Pulls :latest that the build job just pushed.
image: pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner:latest
steps:
- name: Repoint publish workflows at the new image tag
env:
# Full-access PAT — needs contents:write to push the pin commit.
# github.token is read-only here; PKGRW_PAT is registry-only.
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
run: |
set -eu
HOST="${URL_TO_GITEA#http://}"
SHORT_SHA=$(echo "${GITHUB_SHA}" | cut -c1-7)
git clone "http://mikkeli:${FAPAT}@${HOST}/novoyuuparosk-wiki/novoyuuparosk-auto-wiki" repo
cd repo
for f in .gitea/workflows/publish-songs.yml \
.gitea/workflows/publish-ses.yml \
.gitea/workflows/publish-tech.yml; do
sed -i -E "s#(novoyuuparosk-wiki-runner:)[A-Za-z0-9._-]+#\1${SHORT_SHA}#" "$f"
done
if git diff --quiet; then
echo "Pins already at ${SHORT_SHA}; nothing to commit."
exit 0
fi
git config user.email "actions@novoyuuparosk.org"
git config user.name "wiki-runner image bot"
git commit -am "build: pin runner image to ${SHORT_SHA}"
# Pin commit only touches publish-*.yml (on: workflow_call) — triggers
# nothing, so no rebuild loop. Fails loudly if master moved meanwhile.
git push origin HEAD:master
+65
View File
@@ -0,0 +1,65 @@
# Rebuilds the kaniko-act builder image.
#
# Dispatch-only on purpose. This is the image build-image.yml runs inside, so
# an automatic trigger could push a broken builder and take the whole build
# path down with it. Bump deliberately, verify, then repoint build-image.yml.
#
# Self-hosting: builds the new kaniko-act using the *previous* kaniko-act, per
# the bootstrap note in ci/Dockerfile.kaniko-act.
on:
workflow_dispatch:
inputs:
VERSION:
description: "Tag to publish, e.g. v1.23.2-r2"
required: true
type: string
default: v1.23.2-r2
jobs:
build:
runs-on: ubuntu-latest
container:
# The `mikkeli` copy is the only known-good kaniko-act: it was built on
# the host with docker. The org copy this workflow produced exits 0 but
# lacks /tmp, so it cannot build anything — do not point here at it.
#
# This workflow is consequently NOT usable as-is for a version bump. Fix
# ci/Dockerfile.kaniko-act to force /tmp to materialise first, or build
# on the host. Kept because the destination and auth wiring are correct.
image: pi5-16.local:3005/mikkeli/kaniko-act:v1.23.2-r2
steps:
- name: Fetch source, build, push
# kaniko image has no bash — busybox sh only
shell: sh
env:
# package r/w PAT (org-level secret) — registry push only
PKG_TOKEN: ${{ secrets.PKGRW_PAT }}
# per-run auto-token, read access to this repo only — source fetch
JOB_TOKEN: ${{ github.token }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
VERSION: ${{ inputs.VERSION }}
run: |
set -eu
HOST="${URL_TO_GITEA#http://}"
IMAGE="${HOST}/novoyuuparosk-wiki/kaniko-act"
wget -qO /tmp/src.tar.gz \
"http://mikkeli:${JOB_TOKEN}@${HOST}/api/v1/repos/novoyuuparosk-wiki/novoyuuparosk-auto-wiki/archive/${GITHUB_SHA}.tar.gz"
mkdir -p /tmp/src
tar -xzf /tmp/src.tar.gz -C /tmp/src --strip-components=1
# Registry auth for the push
AUTH=$(printf '%s' "mikkeli:${PKG_TOKEN}" | base64 | tr -d '\n')
printf '{"auths":{"%s":{"auth":"%s"}}}' "${HOST}" "${AUTH}" \
> /kaniko/.docker/config.json
# --insecure: push target is the plain-HTTP Gitea registry.
# Base image pull (gcr.io) stays HTTPS — no --insecure-pull.
/kaniko/executor \
--context dir:///tmp/src \
--dockerfile /tmp/src/ci/Dockerfile.kaniko-act \
--destination "${IMAGE}:${VERSION}" \
--insecure
echo "Pushed ${IMAGE}:${VERSION}"
+4 -4
View File
@@ -10,7 +10,7 @@ jobs:
publish:
runs-on: ubuntu-latest
container:
image: novoyuuparosk-wiki-runner:latest
image: pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner:4f134af
steps:
- name: Checkout ses-light-novel
@@ -18,14 +18,14 @@ jobs:
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
run: |
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/ses-light-novel" ses-light-novel
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/novoyuuparosk-wiki/ses-light-novel" ses-light-novel
git -C ses-light-novel checkout ${{ inputs.SOURCE_REF }}
- name: Checkout auto-wiki
env:
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
run: git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/novoyuuparosk-auto-wiki" auto-wiki
run: git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/novoyuuparosk-wiki/novoyuuparosk-auto-wiki" auto-wiki
- name: Publish
env:
@@ -34,7 +34,7 @@ jobs:
WIKI_BOT_USER: ${{ secrets.WIKI_BOT_USER }}
WIKI_BOT_PASSWORD: ${{ secrets.WIKI_BOT_PASSWORD }}
SOURCE_REF: ${{ inputs.SOURCE_REF }}
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/mikkeli/ses-light-novel
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/novoyuuparosk-wiki/ses-light-novel
run: |
python auto-wiki/pipelines/ses/publish.py \
--source-dir ses-light-novel \
+31 -37
View File
@@ -6,54 +6,54 @@ on:
required: true
type: string
SOURCE_BASE_REF:
description: "Commit SHA before the push (all-zeros = first push)"
description: "Commit SHA before the push (empty or all-zeros = first push)"
required: true
type: string
FULL_PUBLISH:
description: "Re-publish every file regardless of diff"
required: false
type: boolean
default: false
jobs:
publish:
runs-on: ubuntu-latest
container:
image: novoyuuparosk-wiki-runner:latest
image: pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner:4f134af
steps:
- name: Checkout ncmr-songs
env:
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
SOURCE_REF: ${{ inputs.SOURCE_REF }}
run: |
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/ncmr-songs" ncmr-songs
git -C ncmr-songs checkout ${{ inputs.SOURCE_REF }}
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/novoyuuparosk-wiki/ncmr-songs" ncmr-songs
git -C ncmr-songs checkout "$SOURCE_REF"
- name: Checkout auto-wiki
env:
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
run: git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/novoyuuparosk-auto-wiki" auto-wiki
run: git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/novoyuuparosk-wiki/novoyuuparosk-auto-wiki" auto-wiki
# The diff bounds what gets *written*, not what gets *read*. The publisher
# always scans the whole repository to resolve links and page titles, then
# writes only these files (plus, one hop out, any page they link to or that
# links to them). Passing a subset here is safe for that reason and only
# that reason — resolution no longer depends on the batch.
- name: Detect changed files
id: diff
env:
BASE: ${{ inputs.SOURCE_BASE_REF }}
HEAD: ${{ inputs.SOURCE_REF }}
run: |
BASE="${{ inputs.SOURCE_BASE_REF }}"
HEAD="${{ inputs.SOURCE_REF }}"
# All-zeros base means first push — publish everything
# Empty or all-zeros base means first push or manual dispatch — publish everything
if [ -z "$BASE" ] || echo "$BASE" | grep -qE '^0+$'; then
echo "First push or manual dispatch — publishing everything."
echo "mode=all" >> "$GITHUB_OUTPUT"
else
git -C ncmr-songs diff --name-only "$BASE" "$HEAD" -- '*.md' \
| grep -v '^wip/' \
| grep -v '^README\.md' \
> /tmp/changed.txt 2>/dev/null || true
echo "Changed files:"
cat /tmp/changed.txt
echo "mode=incremental" >> "$GITHUB_OUTPUT"
echo "files=$(tr '\n' ' ' < /tmp/changed.txt)" >> "$GITHUB_OUTPUT"
exit 0
fi
CHANGED=$(git -C ncmr-songs diff --name-only "$BASE" "$HEAD" -- '*.md' | grep -v '^wip/' || true)
echo "Changed files:"
echo "$CHANGED"
echo "mode=incremental" >> "$GITHUB_OUTPUT"
echo "files=$(echo "$CHANGED" | tr '\n' ' ')" >> "$GITHUB_OUTPUT"
- name: Publish
env:
@@ -62,21 +62,15 @@ jobs:
WIKI_BOT_USER: ${{ secrets.WIKI_BOT_USER }}
WIKI_BOT_PASSWORD: ${{ secrets.WIKI_BOT_PASSWORD }}
SOURCE_REF: ${{ inputs.SOURCE_REF }}
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/mikkeli/ncmr-songs
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/novoyuuparosk-wiki/ncmr-songs
MODE: ${{ steps.diff.outputs.mode }}
FILES: ${{ steps.diff.outputs.files }}
run: |
MODE="${{ steps.diff.outputs.mode }}"
if [ "$MODE" = "all" ] || [ "${{ inputs.FULL_PUBLISH }}" = "true" ]; then
python auto-wiki/pipelines/songs/publish.py \
--source-dir ncmr-songs \
--all
else
FILES="${{ steps.diff.outputs.files }}"
if [ -z "$FILES" ]; then
echo "No changed .md files detected. Nothing to publish."
exit 0
fi
if [ "$MODE" = "all" ]; then
python auto-wiki/pipelines/songs/publish.py --source-dir ncmr-songs --all
elif [ -n "$(echo "$FILES" | tr -d '[:space:]')" ]; then
# shellcheck disable=SC2086
python auto-wiki/pipelines/songs/publish.py \
--source-dir ncmr-songs \
--files $FILES
python auto-wiki/pipelines/songs/publish.py --source-dir ncmr-songs --files $FILES
else
echo "No changed .md files outside wip/. Nothing to publish."
fi
+4 -4
View File
@@ -10,7 +10,7 @@ jobs:
publish:
runs-on: ubuntu-latest
container:
image: novoyuuparosk-wiki-runner:latest
image: pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner:4f134af
steps:
- name: Checkout tech-blogs
@@ -18,14 +18,14 @@ jobs:
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
run: |
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/tech-blogs" tech-blogs
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/novoyuuparosk-wiki/tech-blogs" tech-blogs
git -C tech-blogs checkout ${{ inputs.SOURCE_REF }}
- name: Checkout auto-wiki
env:
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
run: git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/novoyuuparosk-auto-wiki" auto-wiki
run: git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/novoyuuparosk-wiki/novoyuuparosk-auto-wiki" auto-wiki
- name: Publish
env:
@@ -34,7 +34,7 @@ jobs:
WIKI_BOT_USER: ${{ secrets.WIKI_BOT_USER }}
WIKI_BOT_PASSWORD: ${{ secrets.WIKI_BOT_PASSWORD }}
SOURCE_REF: ${{ inputs.SOURCE_REF }}
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/mikkeli/tech-blogs
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/novoyuuparosk-wiki/tech-blogs
run: |
python auto-wiki/pipelines/tech/publish.py \
--source-dir tech-blogs \
+30
View File
@@ -0,0 +1,30 @@
name: tests
on:
push:
pull_request:
workflow_dispatch:
jobs:
pytest:
runs-on: ubuntu-latest
# Same image the pipelines publish from, so the pandoc contract tests check
# the pandoc that actually renders the wiki, not whatever a fresh runner has.
container:
image: pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner:4f134af
steps:
- name: Checkout auto-wiki
env:
FAPAT: ${{ secrets.FAPAT }}
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
REF: ${{ github.sha }}
run: |
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/novoyuuparosk-wiki/novoyuuparosk-auto-wiki" auto-wiki
git -C auto-wiki checkout "$REF"
- name: Install test dependencies
run: pip install --no-cache-dir -r auto-wiki/requirements-dev.txt
- name: Run tests
run: python -m pytest auto-wiki/tests -q
+3
View File
@@ -1,2 +1,5 @@
# claude local settings
.claude/
# python
__pycache__/
+26 -16
View File
@@ -6,9 +6,9 @@ CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from u
| Path | Source repo | Purpose | Status |
|---|---|---|---|
| [`pipelines/songs/`](pipelines/songs/) | `mikkeli/ncmr-songs` | Song lyric pages | v1 live |
| [`pipelines/ses/`](pipelines/ses/) | `mikkeli/ses-light-novel` | SES light novel pages | v1 in development |
| [`pipelines/tech/`](pipelines/tech/) | `mikkeli/tech-blogs` | Tech blog posts | v1 ready (source repo not yet initialised) |
| [`pipelines/songs/`](pipelines/songs/) | `novoyuuparosk-wiki/ncmr-songs` | Song lyric pages | v1 live |
| [`pipelines/ses/`](pipelines/ses/) | `novoyuuparosk-wiki/ses-light-novel` | SES light novel pages | v1 in development |
| [`pipelines/tech/`](pipelines/tech/) | `novoyuuparosk-wiki/tech-blogs` | Tech blog posts | v1 live |
Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime, decisions). This root README covers only what's cross-cutting.
@@ -35,9 +35,13 @@ Note: `workflow_call` across private repos was abandoned — the auto-generated
## Bot identity
MediaWiki BotPassword issued for user `Dubrowski`, bot name `giteaAutomaton`. Login form: `Dubrowski@giteaAutomaton`.
MediaWiki BotPassword issued for user `Mikkeli`, bot name `giteaBot`. Login form: `Mikkeli@giteaBot`.
Credentials are stored in the Gitea user-scope secret vault under `mikkeli` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo.
`Mikkeli` holds the `bot` right, so pipeline edits are flagged as bot edits and stay out of default Recent Changes — `mwclient`'s `page.save()` requests the bot flag and the wiki honours it.
This section previously named `Dubrowski@giteaAutomaton`, but every bot-flagged revision in the wiki's history is attributed to `Mikkeli`, so that had been stale for some time — the live secret never matched the doc.
Credentials are stored in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo.
## Runner infrastructure
@@ -49,23 +53,21 @@ Container network mode: `host` — required so job containers can reach `localho
### Job container image
All pipelines share a single pre-built Docker image: `novoyuuparosk-wiki-runner:latest`. The `Dockerfile` is at the repo root. It bakes in system deps (git, pandoc, ca-certificates) and all pipeline Python packages so job containers start instantly with no install steps.
All pipelines share a single pre-built Docker image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The `Dockerfile` is at the repo root. It bakes in system deps (git, pandoc, ca-certificates) and all pipeline Python packages so job containers start instantly with no install steps.
The image is built manually on the Pi and stored in the local Docker daemon (`pull_image: false` in act_runner config). Rebuild after any change to the `Dockerfile` or a pipeline `requirements.txt`:
The image builds automatically via [`.gitea/workflows/build-image.yml`](.gitea/workflows/build-image.yml), which triggers on pushes that touch the `Dockerfile`, any pipeline `requirements.txt`, or that workflow itself. It uses kaniko (daemonless, unprivileged) to build and push two tags: an immutable `:<short-sha>` and a moving `:latest`.
```bash
docker build -t novoyuuparosk-wiki-runner:latest \
/home/mikkeli/dev/novoyuuparosk-auto-wiki
```
A follow-up `pin` job then rewrites the `image:` pin in each `publish-*.yml` to the new `:<short-sha>` and commits it back to `master` (using `FAPAT` for contents write). The publish workflows therefore always reference an immutable tag, kept current automatically — no manual bump. The pin commit only touches `workflow_call` files, so it triggers no further runs.
## Gitea Actions setup (cross-cutting)
Secrets and variables are scoped to user `mikkeli` (no orgs on this instance), inherited by all repos under that account.
Secrets and variables are scoped to the `novoyuuparosk-wiki` org, inherited by all repos under it. Runs belong to the *caller* repo, so a source repo calling a reusable workflow here resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo.
**Secrets:**
- `WIKI_BOT_USER` = `Dubrowski@giteaAutomaton`
- `WIKI_BOT_USER` = `Mikkeli@giteaBot`
- `WIKI_BOT_PASSWORD` = the value from *Bot identity* above
- `FAPAT` = Full-Access PAT under `mikkeli`, used by source-repo workflows to clone this repo at runtime
- `FAPAT` = Full-Access PAT owned by `mikkeli`, used by source-repo workflows to clone this repo at runtime. The PAT stays personal; only its storage scope moved to the org — hence the `mikkeli:` basic-auth username in the clone URLs.
- `PKGRW_PAT` = package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry
**Variables:**
- `WIKI_BASE_URL` = `https://wiki.novoyuuparosk.org`
@@ -86,17 +88,25 @@ Secrets and variables are scoped to user `mikkeli` (no orgs on this instance), i
| Runner execution | Docker, added as a service to the existing Gitea docker-compose | 2026-06-09 |
| Runner network mode | `host` — job containers need to reach Gitea on localhost | 2026-06-09 |
| Secret/runner scope | User-level on `mikkeli` (no orgs on this instance) | 2026-06-09 |
| Ownership | This repo and all three source repos moved to the `novoyuuparosk-wiki` org. Secrets/variables re-created at org scope; runner re-registered instance-level so it serves org-owned runs | 2026-08-11 |
| **This repo must stay public** | `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied. A private callee therefore 404s with `repository not found`, regardless of the caller sharing its owner. Shared ownership does **not** satisfy the read requirement. Alternative if it must be private again: Settings → Actions → General → collaborative owners (Gitea 1.26+), untested here | 2026-08-11 |
| Runner cache masks this | The resolved callee is cached at `/root/.cache/act/<owner>-<repo>@<ref>`. Changing owner changes the key, so a working pipeline can break on a clone that had been served from cache for months. Suspect the cache before suspecting permissions | 2026-08-11 |
| **Callee edits need a cache flush** | The cache is *not* refreshed per run. Edits to `publish-*.yml` on `master` — including the pin job's own commits — keep executing the stale cached copy until the key changes or the cache is cleared. Observed directly: a run used the previous namespace and tag despite `master` being current. Flush with `docker compose up -d --force-recreate act_runner`; the cache is in the container layer, while `.runner` is in the `/data` volume, so registration survives | 2026-08-11 |
| kaniko cannot rebuild kaniko-act | The org rebuild exits 0 but the resulting image has no `/tmp` — kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so `wget -O /tmp/...` fails. The bootstrap note's claim that version bumps "can be built by the build-image workflow itself" does not hold as written. `kaniko-act` therefore still lives under `mikkeli`; a real bump needs a host `docker build` + push, or a Dockerfile that forces the dir to materialise (e.g. writing a file inside it) | 2026-08-11 |
| Container registry | Gitea cannot transfer packages, so images were *rebuilt* into `pi5-16.local:3005/novoyuuparosk-wiki/*` rather than moved. `kaniko-act` bootstrapped via the new dispatch-only `build-kaniko-act.yml`, running in the old `mikkeli` copy; `novoyuuparosk-wiki-runner` came from a normal `build-image.yml` run. Registry auth is still the `mikkeli`-owned `PKGRW_PAT`, hence the `mikkeli:` username in the auth blob | 2026-08-11 |
| Old images left in place | The `mikkeli/*` package versions are orphaned but retained — nothing references them, and deleting a container version is irreversible. Safe to purge once the org images have proven themselves | 2026-08-11 |
| MediaWiki API path | `api.php` (classic action API) | 2026-06-09 |
| Branch naming (this repo) | `automation/<pipeline>` for pipeline-development branches | 2026-06-09 |
| Variable naming | `URL_TO_GITEA` not `GITEA_URL` — Gitea blocks `GITEA_`/`GITHUB_` prefixes | 2026-06-09 |
| Columns shorthand | Side-by-side columns authored as a ` ```columns ` fenced block, expanded post-Pandoc in shared `lib/wiki.py` (universal across pipelines). No wiki template or PHP extension — runs Pi-side before the API call | 2026-06-14 |
Per-pipeline decisions live in each pipeline's README.
## Setup checklist (cross-cutting)
Via the Gitea web UI logged in as `mikkeli`:
Via the Gitea web UI logged in as `mikkeli` (an owner of `novoyuuparosk-wiki`):
- [x] User-scoped secrets and variables set per *Gitea Actions setup* above
- [x] Org-scoped secrets and variables set per *Gitea Actions setup* above
- [x] `WIKI_BOT_USER`
- [x] `WIKI_BOT_PASSWORD`
- [x] `FAPAT` (Full-Access PAT — value not stored in this README; saved directly into the Gitea secret. Regenerate if lost.)
+15
View File
@@ -0,0 +1,15 @@
# Kaniko executor adapted for Gitea act_runner job containers.
#
# act starts job containers with entrypoint /bin/sleep and sets
# RUNNER_TEMP=/tmp; kaniko's scratch-based image has neither /bin/sleep
# (busybox lives under /busybox; /bin/sh is already symlinked) nor /tmp.
# This wrapper adds exactly those two. Nothing else changes.
#
# Bootstrap: the very first build of this image was done manually on the host
# (docker build -f ci/Dockerfile.kaniko-act -t <registry>/<owner>/kaniko-act:<ver> .)
# because no builder image existed yet. Version bumps are now built by
# .gitea/workflows/build-kaniko-act.yml (dispatch-only), which runs in the
# previous kaniko-act — confirmed working when the image moved to the org.
FROM gcr.io/kaniko-project/executor:v1.23.2-debug
SHELL ["/busybox/sh", "-c"]
RUN ln -sf /busybox/sleep /bin/sleep && mkdir -p -m 1777 /tmp
+168 -2
View File
@@ -1,13 +1,68 @@
"""Shared utilities for novoyuuparosk-auto-wiki pipelines."""
import html
import os
import re
import subprocess
import urllib.parse
from pathlib import Path
from urllib.parse import urlparse
import mwclient
AUTO_BANNER_PREFIX = "{{Auto-generated"
# A ```columns fenced block is passed through Pandoc verbatim as
# <pre class="columns">…</pre>; columns within it are separated by a line of ===.
_COLUMNS_BLOCK_RE = re.compile(r'<pre class="columns">(.*?)</pre>', re.DOTALL)
_COLUMN_SEP_RE = re.compile(r"^\s*===\s*$", re.MULTILINE)
# Minimal Markdown emphasis → wikitext, for use inside a verbatim columns block
# (Pandoc doesn't process Markdown there). Asterisk style only, single line.
# Order matters: bold-italic (***) before bold (**) before italic (*).
_BOLD_ITALIC_RE = re.compile(r"\*\*\*(.+?)\*\*\*")
_BOLD_RE = re.compile(r"\*\*(.+?)\*\*")
_ITALIC_RE = re.compile(r"\*(.+?)\*")
# Pandoc's mediawiki writer renders a *relative* Markdown link target as a
# wikilink carrying the raw path — [label](notes.md) becomes [[notes.md|label]] —
# and an absolute URL as [url label]. Both shapes are matched here so that a
# pandoc version which classifies relative targets the other way still resolves.
# The alternation is scanned left to right, so a [[...]] is consumed whole and
# the single-bracket branch never re-matches its interior.
_MD_LINK_RE = re.compile(
r"\[\[(?P<wiki_target>[^\[\]|]+?)(?:\|(?P<wiki_label>[^\[\]]*))?\]\]"
r"|\[(?P<ext_target>[^\[\]\s|]+)(?:[ \t]+(?P<ext_label>[^\[\]]*))?\]"
)
# Inline Markdown link target, read from the *source* body before Pandoc runs:
# the "…](target)" tail of [label](target). Used for the link graph only.
_MD_SOURCE_LINK_RE = re.compile(r"\]\(\s*<?([^()<>\s]+)>?")
# Anything still naming a .md file in a link position after resolution means the
# renderer failed to understand it. That aborts the run; it never reaches the wiki.
_UNRESOLVED_LINK_RES = (
(
re.compile(r"\[\[[^\[\]]*\.md(?:#[^\[\]]*)?(?:\|[^\[\]]*)?\]\]", re.IGNORECASE),
"unresolved wikilink",
),
(
re.compile(r"\[[^\[\]\s|]*\.md[^\[\]]*\]", re.IGNORECASE),
"unresolved external-style link",
),
(
re.compile(r"\]\([^()\s]*\.md[^()]*\)", re.IGNORECASE),
"raw Markdown link (Markdown is not processed inside a ```columns fence)",
),
)
def _md_emphasis_to_wikitext(text: str) -> str:
text = _BOLD_ITALIC_RE.sub(r"'''''\1'''''", text)
text = _BOLD_RE.sub(r"'''\1'''", text)
text = _ITALIC_RE.sub(r"''\1''", text)
return text
def strip_first_h1(text: str) -> str:
"""Remove the first '# Heading' line and any immediately following blank line."""
@@ -21,16 +76,127 @@ def strip_first_h1(text: str) -> str:
return "\n".join(lines)
def expand_columns(wikitext: str) -> str:
"""Expand ```columns fenced blocks into a flex row of <poem> columns.
Authors write a fenced code block tagged ``columns``; Pandoc passes its body
through verbatim as ``<pre class="columns">…</pre>`` (line breaks and blank
lines preserved, inline markup entity-escaped). Columns within the block are
separated by a line containing only ``===``. Each column is wrapped in
<poem> so its line breaks survive MediaWiki parsing. Content is
HTML-unescaped (so inline markup such as <b>…</b> renders rather than
appearing as literal text) and Markdown emphasis (``*italic*``, ``**bold**``,
``***bold-italic***``) is converted to wikitext, since Pandoc does not
process Markdown inside the verbatim block.
"""
def render(match: re.Match) -> str:
body = _md_emphasis_to_wikitext(html.unescape(match.group(1)))
columns = _COLUMN_SEP_RE.split(body)
poems = "".join("<poem>\n" + col.strip("\n") + "\n</poem>" for col in columns)
return '<div style="display:flex; gap:3em; align-items:flex-start">' + poems + "</div>"
return _COLUMNS_BLOCK_RE.sub(render, wikitext)
def extract_md_link_targets(markdown_body: str) -> list[str]:
"""Raw targets of inline Markdown links in an unrendered body.
Used to build the link graph *before* Pandoc runs, so the publisher can work
out which pages a changed page points at (and which point back) without
rendering the whole repository. Targets are returned verbatim, including any
``#fragment`` and percent-encoding; the caller resolves them.
"""
return _MD_SOURCE_LINK_RE.findall(markdown_body)
def assert_no_unresolved_md_links(wikitext: str, source_path: Path) -> None:
"""Abort if any link still names a .md file after resolution.
Guards against a link shape the resolver does not recognise: a pandoc version
that writes relative targets differently, or a Markdown link buried in a
verbatim ```columns fence. Without this, such a link would reach the wiki as
literal text, silently corrupting a published page.
"""
for pattern, kind in _UNRESOLVED_LINK_RES:
found = pattern.search(wikitext)
if found:
raise ValueError(
f"{source_path}: {kind} still references a .md file after "
f"resolution: {found.group(0)!r}"
)
def resolve_md_links(wikitext: str, source_path: Path, title_index: dict[str, str]) -> str:
"""Rewrite links to sibling ``.md`` files into wikilinks to their wiki pages.
A related co-publication is nominated with an ordinary relative Markdown
link — ``see [dev notes](pulse_dev.md)`` — which stays a working link in
Gitea's blob view and in any Markdown preview. Here the ``.md`` target is
resolved against ``title_index`` (absolute path string → wiki page title) and
replaced with ``[[Wiki page title|label]]``.
``title_index`` must cover every publishable file in the repository, not only
the batch being published, or the same link resolves differently depending on
which files a push happened to touch.
A ``.md`` target that cannot be resolved raises ``ValueError``: either the
file does not exist, or it exists but is not published. Both are authoring
mistakes, and both fail the run rather than silently degrade the page. Link
targets that do not name a ``.md`` file are left exactly as Pandoc wrote them.
"""
def resolve(match: re.Match) -> str:
groups = match.groupdict()
if groups["wiki_target"] is not None:
target, label = groups["wiki_target"], groups["wiki_label"]
else:
target, label = groups["ext_target"], groups["ext_label"]
raw_path, _, fragment = target.partition("#")
rel_path = urllib.parse.unquote(raw_path)
if not rel_path.lower().endswith(".md"):
return match.group(0)
resolved = (source_path.parent / rel_path).resolve()
title = title_index.get(str(resolved))
if title is None:
reason = (
"file exists but is not published (no 'wiki.publish: true', "
"or it lives under wip/)"
if resolved.exists()
else "file does not exist"
)
raise ValueError(
f"{source_path}: link to '{target}' cannot be resolved - {reason}"
)
anchor = f"#{fragment}" if fragment else ""
# A label that repeats the path (pandoc's form for an unlabelled link) or
# repeats the page title needs no pipe: [[Title]] already displays it.
if label is None or label.strip() in (target, title):
return f"[[{title}{anchor}]]"
return f"[[{title}{anchor}|{label}]]"
resolved = _MD_LINK_RE.sub(resolve, wikitext)
assert_no_unresolved_md_links(resolved, source_path)
return resolved
def markdown_to_wikitext(body: str) -> str:
# encoding is pinned rather than left to text=True, which would decode
# pandoc's output with the ambient locale encoding. The lyrics are Japanese
# and Chinese; on any non-UTF-8 locale that decode dies inside subprocess's
# reader thread, stdout comes back None, and the real error is lost.
result = subprocess.run(
["pandoc", "-f", "markdown", "-t", "mediawiki"],
input=body,
capture_output=True,
text=True,
encoding="utf-8",
)
if result.returncode != 0:
raise RuntimeError(f"pandoc failed: {result.stderr.strip()}")
return result.stdout
return expand_columns(result.stdout)
def connect_wiki() -> mwclient.Site:
+3 -3
View File
@@ -1,6 +1,6 @@
# pipelines/ses
Publishes SES light novel pages from `mikkeli/ses-light-novel` to `https://wiki.novoyuuparosk.org` under the `SES:` namespace.
Publishes SES light novel pages from `novoyuuparosk-wiki/ses-light-novel` to `https://wiki.novoyuuparosk.org` under the `SES:` namespace.
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
@@ -24,7 +24,7 @@ Not in scope (intentional):
## Source repo and branch convention
Source repo: `mikkeli/ses-light-novel`. The pipeline triggers on push to `master`.
Source repo: `novoyuuparosk-wiki/ses-light-novel`. The pipeline triggers on push to `master`.
No branch convention enforced — work directly on `master` or use whatever branch workflow suits.
@@ -34,7 +34,7 @@ No branch convention enforced — work directly on `master` or use whatever bran
## Invocation
The pipeline runs as a self-contained Gitea Actions workflow in `mikkeli/ses-light-novel` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ses-light-novel` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
Triggers:
- `push` to `master` (path-filtered as above)
+1 -1
View File
@@ -1,6 +1,6 @@
# SCHEMA — SES light novel source files
Source-file contract for the SES pipeline. Files in `mikkeli/ses-light-novel` must follow this contract to be picked up by the auto-publisher.
Source-file contract for the SES pipeline. Files in `novoyuuparosk-wiki/ses-light-novel` must follow this contract to be picked up by the auto-publisher.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc.
+21 -9
View File
@@ -1,6 +1,6 @@
# pipelines/songs
Publishes lyric pages from `mikkeli/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
@@ -15,7 +15,8 @@ Implemented:
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set); additional categories via `categories` list
- LRC file upload (idempotent SHA1 check) + footer link
- Inter-page sibling placeholder resolution (`[[TAG]]` → `[[wiki page title|label]]`)
- Inter-page link resolution: a relative Markdown link (`[label](notes.md)`) becomes `[[wiki page title|label]]`, resolved against a repo-wide index so it never depends on which files a push touched
- Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
- Files without `wiki.publish: true` silently skipped
- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
@@ -27,7 +28,7 @@ Not in scope (intentional):
## Source repo and branch convention
Source repo: `mikkeli/ncmr-songs`. The pipeline triggers on push to `master`.
Source repo: `novoyuuparosk-wiki/ncmr-songs`. The pipeline triggers on push to `master`.
**Branch convention in `ncmr-songs`**: per-song short-lived branches named `autowiki/<song-slug>` (e.g., `autowiki/pulse`). Create when staging edits, merge to master when ready to publish, delete after one cycle.
@@ -38,7 +39,7 @@ Source repo: `mikkeli/ncmr-songs`. The pipeline triggers on push to `master`.
## Invocation
The pipeline runs as a self-contained Gitea Actions workflow in `mikkeli/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
Triggers:
- `push` to `master` (path-filtered as above)
@@ -50,7 +51,7 @@ Triggers:
|---|---|---|---|
| `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Dubrowski@giteaAutomaton`) |
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Mikkeli@giteaBot`) |
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Gitea instance base URL for cloning |
| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
@@ -65,7 +66,12 @@ See the `Dockerfile` at the repo root and `requirements.txt` in this directory.
## Modes
The pipeline always runs `--all`: every file with `wiki.publish: true` (not under `wip/`) is checked against the current wiki content. Writes only happen when content has changed (commit SHA in the banner drives idempotency). This is appropriate for the current repo size.
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every file with `wiki.publish: true` (not under `wip/`), because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page.
- `--all` — publish every publishable file. Used on the first push and on `workflow_dispatch`.
- `--files <paths…>` — publish these files and their one-hop link neighbours. Used for an ordinary push.
Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read.
## One-off wiki setup
@@ -82,13 +88,19 @@ The pipeline always runs `--all`: every file with `wiki.publish: true` (not unde
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
| Album categories | Auto-injected `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set; category pages are user-written and not overwritten | 2026-06-09 |
| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:<value>]]` | 2026-06-10 |
| `album` optional | Companion/sibling pages don't need an album; omitting the field emits no album category | 2026-06-10 |
| Sibling resolution | Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc | 2026-06-10 |
| `album` optional | Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 |
| ~~Sibling resolution~~ | ~~Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc~~ — **retired 2026-09-09**, see below | 2026-06-10 |
| Inter-page links | Plain relative Markdown link (`[label](notes.md)`); target's own `title` frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the `wiki.siblings` bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch | 2026-09-09 |
| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 |
| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 |
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
| Publish mode | Always `--all`; no diff detection — small repo, simpler than fragile git-diff gating | 2026-06-09 |
| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 |
| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 |
| Tests | `tests/` run by `.gitea/workflows/test.yml` in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose | 2026-09-09 |
| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 |
| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 |
| Side-by-side columns | Authored as a ` ```columns ` fenced block (separator `===`); Pandoc emits `<pre class="columns">`, expanded post-Pandoc into a flex `<div>` of `<poem>` columns. No wiki template or extension — all Pi-side before the API write. See [SCHEMA.md](SCHEMA.md#columns-side-by-side-shorthand) | 2026-06-14 |
## Status
+112 -71
View File
@@ -1,8 +1,8 @@
# SCHEMA — song source files
Source-file contract for the songs pipeline. Files in `mikkeli/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
Source-file contract for the songs pipeline. Files in `novoyuuparosk-wiki/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar, except that `[[TAG]]` placeholders declared via `wiki.siblings` are substituted with resolved wikilinks after rendering.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar. There is no macro syntax and nothing to declare: a link to another song page is an ordinary relative Markdown link, resolved to a wikilink at publish time.
## File layout
@@ -12,20 +12,16 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
# pulse under latent semantic envelopes
... body markdown, PULSE_DEV used inline ...
... body markdown; a related page is nominated as [development notes](pulse_dev.md) ...
```
The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc, then applies sibling substitutions on the wikitext output.
The renderer parses the frontmatter, validates required fields, hands the body (with the first `# Heading` line removed) to Pandoc, then resolves `.md` links in the wikitext output to the wiki page titles of their targets.
## Fields
@@ -81,79 +77,119 @@ Example: in `ses/pulse.md`, `lrc: pulse.lrc` declares that `ses/pulse.lrc` is th
If the LRC file is missing or not valid UTF-8, the upload is skipped with a warning — the page still publishes.
#### `wiki.siblings` (list of sibling declarations)
Bill of materials for inter-page wikilinks. Each entry declares a sibling `.md` file and a tag that can be used as a `[[TAG]]` placeholder anywhere in the body.
```yaml
wiki:
siblings:
- path: pulse_dev.md # relative to this file's directory
tag: PULSE_DEV # placeholder used in body as [[PULSE_DEV]]
label: development notes # optional display text
```
**Fields per entry:**
| Field | Required for resolution | Purpose |
|---|---|---|
| `path` | yes | Relative path to the sibling `.md` file |
| `tag` | yes | Placeholder identifier; pipeline does a case-sensitive string replace of `TAG` in the rendered wikitext |
| `label` | no | Display text for the resolved link |
An entry missing either `path` or `tag` is silently skipped — no substitution is attempted for that entry.
**Resolution:**
The pipeline resolves each declared sibling's `path` to its wiki page title by looking it up in the set of all publishable files processed in the same run. Substitution is a case-sensitive string replace of the bare tag. Substitution rules:
| Condition | `TAG` becomes |
|---|---|
| `path` resolves to a published page, `label` present | `[[wiki page title\|label]]` |
| `path` resolves to a published page, no `label` | `[[wiki page title]]` |
| `path` not in published set (file exists but not published), `label` present | `label` (plain text) |
| `path` not in published set, no `label` | `TAG` (plain text) |
Declared siblings whose file doesn't exist on disk are a **validation error**. Siblings that exist but aren't published (no `wiki.publish: true`) degrade gracefully per the table above.
The siblings list is a bill of materials — declaring a sibling has no effect unless its `[[TAG]]` appears in the body. Unused entries are inert.
**`--files` mode**: sibling resolution is limited to the files in that batch. Siblings pointing to files outside the batch degrade to plain text. This is acceptable since the pipeline always runs `--all` in the workflow.
## Body
Plain markdown. The renderer applies two transformations before and after Pandoc:
Plain markdown. The renderer applies these transformations before and after Pandoc:
1. **Pre-Pandoc**: strip the first-line `# Heading` if present.
2. **Post-Pandoc**: substitute `TAG` placeholders declared in `wiki.siblings`.
2. **Post-Pandoc**: expand `columns` fenced blocks into side-by-side wikitext (see below).
3. **Post-Pandoc**: resolve links to sibling `.md` files into wikilinks (see below).
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, sibling resolution) around Pandoc's output.
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, link resolution) around Pandoc's output.
### Nominating a related co-publication
Link to it the ordinary way, with a relative Markdown link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
There is nothing to declare in the frontmatter. The link stays a working link in Gitea's blob view, in an editor preview, and anywhere else the raw file is read — and on the wiki it becomes a wikilink to the target's page:
```wikitext
For more behind-the-scene shenanigans, see [[pulse under latent semantic envelopes - development notes|dev gibberish]].
```
**How the target becomes a page name.** The path is resolved relative to the linking file's directory, and the target file's own `title` frontmatter field is used verbatim as the wikilink target — the same string the bot writes that page to. There is no second name format to keep in sync: rename a page by editing its `title`, and every link aimed at it follows on the next publish. MediaWiki handles the first-letter capitalisation and the space-to-underscore in URLs itself.
| Written in the body | Rendered wikitext |
|---|---|
| `[dev gibberish](pulse_dev.md)` | `[[<target's title>\|dev gibberish]]` |
| `[pulse_dev.md](pulse_dev.md)` (label same as path) | `[[<target's title>]]` |
| `[notes](../ses/pulse_dev.md)` | `[[<target's title>\|notes]]` |
| `[notes](pulse_dev.md#Section)` | `[[<target's title>#Section\|notes]]` |
| `[site](https://example.com)` | `[https://example.com site]` (untouched) |
**A `.md` link that cannot be resolved fails the run.** Both cases abort with a non-zero exit before anything is published:
| Condition | Result |
|---|---|
| Target file does not exist | **error** — `file does not exist` |
| Target exists but is not published (no `wiki.publish: true`, or under `wip/`) | **error** — `file exists but is not published` |
Linking to a file does **not** publish it. Publication is always the target's own explicit `wiki.publish: true`; nothing is published as a side effect of being linked. If you want the link, opt the target in.
**Scope of a run.** A push publishes the files it changed, plus — one hop in each direction — any page a changed page links to, and any page that links to a changed page. The second is what keeps inbound links correct when a page's `title` moves. Regardless of how few pages a run publishes, the whole repository is always scanned first, so link resolution and title-collision checks see every publishable file.
**Anchors** (`pulse_dev.md#Section`) are passed through verbatim. MediaWiki derives its own section anchors from heading text, so an anchor that doesn't match a heading on the target page lands the reader at the top of it rather than at the section. Nothing validates this.
**Links inside a `columns` fence do not work.** The fence is verbatim by design (see below); Pandoc does not process Markdown in it, so a link written there would reach the wiki as literal text. The renderer detects this and fails the run rather than publishing it. Put the link in ordinary body prose.
### Columns (side-by-side) shorthand
For parallel content — e.g. an original and its translation — write a fenced code block tagged `columns` and separate the columns with a line containing only `===`:
````markdown
```columns
**原題**
一行目
二行目
===
**Title**
first line
second line
```
````
The renderer turns this into a flexbox row of `<poem>` columns (one per `===`-delimited section). Notes:
- **Line breaks and blank lines are preserved verbatim** — that's the point of using a fenced block; Pandoc passes the body through untouched, and each column is wrapped in `<poem>` so MediaWiki keeps the line breaks.
- **N columns**: use N−1 `===` separators. Two is the common case (original / translation).
- **Emphasis in Markdown**: write `*italic*`, `**bold**`, or `***bold-italic***` (asterisk style, single line) — these are converted to wikitext. Raw inline HTML such as `<b>…</b>` or `<br>` also still works (it's HTML-unescaped on the way out), but you shouldn't need it.
- This is a shared transform (`lib/wiki.py`), so it works for any pipeline, not just songs. Column width/gap styling currently lives in that transform.
## Renderer behaviour
For each `.md` with `wiki.publish: true` and not under `wip/`:
A run happens in three phases, in this order, and the wiki is not contacted until the third. Anything wrong in phase 1 or 2 aborts with a non-zero exit and the wiki untouched — never half-updated.
1. Parse and validate frontmatter.
2. Strip the leading `# Heading` from the body if present.
3. Pipe the body through `pandoc -f markdown -t mediawiki`.
4. Substitute `TAG` placeholders with resolved `[[Page|Display]]` wikilinks (or fallback text).
5. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
6. If `lrc` is declared: upload the LRC file if its SHA1 has changed; append a `[[Media:…]]` link.
7. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
9. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
10. Otherwise, write the page with an edit summary referencing the source commit.
**Phase 1 — scan and validate the whole repository.** For every `.md` outside `wip/` (and excluding `README.md`):
1. Parse the frontmatter; skip the file silently unless `wiki.publish: true`.
2. Validate the fields (see *Validation errors* below).
3. Register `title` and fail on a collision with any other publishable file, anywhere in the tree.
4. Build the path → page-title index every `.md` link resolves through.
**Phase 2 — render every page that this run will publish.** For each selected file:
1. Strip the leading `# Heading` from the body if present.
2. Pipe the body through `pandoc -f markdown -t mediawiki`, then expand any `columns` fenced blocks into side-by-side `<poem>` columns.
3. Resolve `.md` links to `[[Page title|label]]` wikilinks; fail on any that cannot be resolved, and on any link shape that survived rendering unresolved.
4. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
5. If `lrc` is declared, append a `[[Media:…]]` link.
6. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
**Phase 3 — write.** For each rendered page:
1. If `lrc` is declared: upload the LRC file if its SHA1 has changed.
2. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
3. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
4. Otherwise, write the page with an edit summary referencing the source commit.
## Implemented scope
- Frontmatter parsing and validation
- Repo-wide page-title collision detection, before any wiki write
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Banner template injection
- LRC file upload and footer link
- Sibling placeholder resolution (`[[TAG]]` → `[[wiki page title|label]]`)
- Relative `.md` link resolution (`[label](notes.md)` → `[[wiki page title|label]]`), repo-wide and batch-independent
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]`) when `album` is set
- Additional categories via `categories` list
- Render-before-write: a broken page fails the run instead of half-updating the wiki
- Idempotent writes (no-op skip when content matches)
- Manual-page protection (skip pages without auto-gen banner)
- Files without `wiki.publish: true` silently skipped
@@ -173,16 +209,18 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
```
### A companion/sibling document (no album)
Its body nominates the companion page with a plain relative link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
### A companion document (no album)
```yaml
---
@@ -223,8 +261,11 @@ Files without a `wiki.publish: true` frontmatter field are silently skipped —
- `wiki.publish` present but not a boolean
- `release_date` (if present) does not parse as YYYY-MM-DD
- `categories` (if present) is not a list
- `wiki.siblings` entry that has both `path` and `tag` declared, but the `path` file does not exist on disk
- `wiki.siblings` entry that is not a mapping
- Two publishable source files declare the same `title` (page-name collision)
- Two publishable source files declare the same `title` (page-name collision) — checked across the whole repository, not just the files a push touched
- A `.md` link whose target file does not exist
- A `.md` link whose target exists but is not published
- A `.md` link that survives rendering unresolved — a link shape the renderer does not recognise, or a Markdown link written inside a `columns` fence
Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.
Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken.
Frontmatter keys the schema does not define are ignored, so a leftover `wiki.siblings` block from the retired macro system is inert rather than an error. Delete them; they do nothing.
+130 -75
View File
@@ -6,6 +6,7 @@ import hashlib
import io
import os
import sys
import urllib.parse
from datetime import datetime
from pathlib import Path
@@ -13,7 +14,14 @@ import frontmatter
import mwclient
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
from lib.wiki import AUTO_BANNER_PREFIX, connect_wiki, markdown_to_wikitext, strip_first_h1
from lib.wiki import (
AUTO_BANNER_PREFIX,
connect_wiki,
extract_md_link_targets,
markdown_to_wikitext,
resolve_md_links,
strip_first_h1,
)
def validate(fm: dict, path: Path) -> None:
@@ -35,47 +43,58 @@ def validate(fm: dict, path: Path) -> None:
if "categories" in fm and not isinstance(fm["categories"], list):
errors.append("'categories' must be a list")
for i, sib in enumerate(fm.get("wiki", {}).get("siblings") or []):
if not isinstance(sib, dict):
errors.append(f"wiki.siblings[{i}] must be a mapping")
continue
sib_path = sib.get("path")
sib_tag = sib.get("tag")
if sib_path and sib_tag:
if not (path.parent / str(sib_path)).resolve().exists():
errors.append(f"wiki.siblings[{i}]: path '{sib_path}' not found")
if errors:
raise ValueError(f"{path}: " + "; ".join(errors))
def build_sibling_map(posts: dict) -> dict[str, str]:
def build_title_index(posts: dict) -> dict[str, str]:
"""Map abs_path_str → wiki_title for all published posts."""
return {str(path): str(post.metadata["title"]) for path, post in posts.items()}
def apply_sibling_substitutions(wikitext: str, this_path: Path, fm_wiki: dict, sibling_map: dict[str, str]) -> str:
"""Replace [[TAG]] placeholders using the declared sibling bill of materials."""
for sib in (fm_wiki.get("siblings") or []):
if not isinstance(sib, dict):
continue
tag = sib.get("tag")
path = sib.get("path")
if not tag or not path:
continue
label = (sib.get("label") or "").strip()
abs_sib = str((this_path.parent / str(path)).resolve())
title = sibling_map.get(abs_sib)
if title and label:
replacement = f"[[{title}|{label}]]"
elif title:
replacement = f"[[{title}]]"
elif label:
replacement = label
else:
replacement = tag
wikitext = wikitext.replace(tag, replacement)
return wikitext
def build_link_graph(posts: dict) -> dict[Path, set[Path]]:
"""Map each publishable file to the publishable .md files it links to.
Read from the raw Markdown, so it costs no Pandoc runs. Targets outside the
publishable set are dropped here; `resolve_md_links` is what refuses them
loudly at render time.
"""
graph: dict[Path, set[Path]] = {}
for path, post in posts.items():
targets = set()
for raw in extract_md_link_targets(post.content):
rel_path = urllib.parse.unquote(raw.partition("#")[0])
if not rel_path.lower().endswith(".md"):
continue
resolved = (path.parent / rel_path).resolve()
if resolved in posts:
targets.add(resolved)
graph[path] = targets
return graph
def select_publish_set(posts: dict, seeds: set[Path]) -> dict[Path, object]:
"""The changed files, plus the pages their links touch in either direction.
Publishing is not instant, so only what a push actually affects is written.
Two extra hops are unavoidable, both one level deep:
- **outbound**: a page the changed page links to, so the wikilink cannot
point at a page that was never published (a red link);
- **inbound**: a page linking *to* a changed page, because if the changed
page's `title` moved, every link aimed at it is now stale.
Ordering follows `posts` so a run's output is stable regardless of which
files the diff reported.
"""
graph = build_link_graph(posts)
selected = set(seeds)
for path, targets in graph.items():
if path in seeds:
selected |= targets
elif targets & seeds:
selected.add(path)
return {path: post for path, post in posts.items() if path in selected}
def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: str, lrc_filename: str | None = None) -> str:
@@ -84,7 +103,7 @@ def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: st
cat_parts = []
if fm.get("album"):
cat_parts.append(f"[[Category:Songs]]")
cat_parts.append("[[Category:Songs]]")
cat_parts.append(f"[[Category:Album:{fm['album']}]]")
for cat in (fm.get("categories") or []):
cat_parts.append(f"[[Category:{cat}]]")
@@ -121,25 +140,33 @@ def upload_lrc(lrc_path: Path, title: str, site: mwclient.Site) -> bool:
return True
def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, source_ref: str, gitea_repo_url: str, sibling_map: dict[str, str]) -> str:
"""Returns 'published', 'noop', or 'skipped-manual'."""
def render_page(abs_path: Path, source_dir: Path, post, source_ref: str, gitea_repo_url: str, title_index: dict[str, str]) -> str:
"""Render one file to finished wikitext. Pure: touches no wiki, writes nothing.
Kept separate from the write so that every page in a run is rendered before
the first one is published. A bad link or a broken column fence then fails
the run with the wiki untouched, instead of half-updating it.
"""
rel = abs_path.relative_to(source_dir)
fm = post.metadata
title = str(fm["title"])
body_wikitext = markdown_to_wikitext(strip_first_h1(post.content))
body_wikitext = apply_sibling_substitutions(body_wikitext, abs_path, fm.get("wiki", {}), sibling_map)
body_wikitext = resolve_md_links(body_wikitext, abs_path, title_index)
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
lrc_filename = None
lrc_rel = fm.get("lrc")
if lrc_rel:
lrc_uploaded = upload_lrc(abs_path.parent / str(lrc_rel), title, site)
lrc_filename = f"{title}.lrc"
if lrc_uploaded:
print(f" [lrc-uploaded] {lrc_filename}")
lrc_filename = f"{title}.lrc" if fm.get("lrc") else None
return build_wikitext(fm, body_wikitext, source_url, source_ref, lrc_filename)
page_content = build_wikitext(fm, body_wikitext, source_url, source_ref, lrc_filename)
def publish_one(abs_path: Path, post, page_content: str, site: mwclient.Site, source_ref: str) -> str:
"""Write one pre-rendered page. Returns 'published', 'noop', or 'skipped-manual'."""
fm = post.metadata
title = str(fm["title"])
lrc_rel = fm.get("lrc")
if lrc_rel and upload_lrc(abs_path.parent / str(lrc_rel), title, site):
print(f" [lrc-uploaded] {title}.lrc")
page = site.pages[title]
existing = page.text()
@@ -155,13 +182,29 @@ def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, sou
return "published"
def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]:
posts = {}
def discover_publishable(source_dir: Path) -> dict[Path, object]:
"""Every publishable .md in the repository, validated, keyed by absolute path.
The whole tree is scanned on every run, never a subset of it, however few
pages the run then publishes. Two things depend on seeing all of it:
- **link resolution**: a link to a sibling .md resolves through this set, so
a partial scan would resolve the same link differently depending on which
files a push happened to touch, and silently drop it when the target fell
outside the subset;
- **title collisions**: two files claiming one wiki page name must fail the
run wherever they sit in the tree, not only when both land in one push.
Files under wip/ and README.md are never published. Everything else needs an
explicit `wiki.publish: true`; a file is never published as a side effect of
something else linking to it.
"""
posts: dict[Path, object] = {}
titles: dict[str, Path] = {}
for path in files:
if not path.exists():
print(f" warn: {path} not found, skipping", file=sys.stderr)
for path in sorted(source_dir.rglob("*.md")):
rel = path.relative_to(source_dir)
if "wip" in rel.parts or path.name == "README.md":
continue
post = frontmatter.load(str(path))
@@ -175,10 +218,11 @@ def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]
title = str(post.metadata["title"])
if title in titles:
raise ValueError(
f"Duplicate wiki title '{title}': {path} and {titles[title]}"
f"Duplicate wiki title '{title}': {rel} and "
f"{titles[title].relative_to(source_dir)}"
)
titles[title] = path
posts[path] = post
posts[path.resolve()] = post
return posts
@@ -186,45 +230,56 @@ def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]
def main() -> None:
parser = argparse.ArgumentParser(description="Publish song .md files to MediaWiki.")
parser.add_argument("--source-dir", required=True, type=Path)
parser.add_argument("--files", nargs="*", default=[], help="Relative paths within source-dir")
parser.add_argument("--all", action="store_true", help="Publish all publishable files")
parser.add_argument(
"--files",
nargs="*",
default=[],
help="Changed files (relative to source-dir) to publish; the push diff",
)
parser.add_argument("--all", action="store_true", help="Publish every publishable file")
args = parser.parse_args()
source_dir = args.source_dir.resolve()
source_ref = os.environ.get("SOURCE_REF", "")
gitea_repo_url = os.environ.get("GITEA_REPO_URL", "").rstrip("/")
# The repository is always scanned in full, whatever gets published: link
# resolution and title-collision checks must see every publishable file, or
# the same link resolves differently depending on which files a push touched.
posts = discover_publishable(source_dir)
if not posts:
print("No publishable files found.")
return
title_index = build_title_index(posts)
print(f"Scanned repository: {len(posts)} publishable file(s).")
if args.all:
candidates = list(source_dir.rglob("*.md"))
selected = posts
elif args.files:
candidates = [source_dir / f for f in args.files]
seeds = {(source_dir / f).resolve() for f in args.files} & posts.keys()
if not seeds:
print("No changed file is publishable. Nothing to publish.")
return
selected = select_publish_set(posts, seeds)
else:
print("Nothing to do: pass --files or --all.")
return
files = [
p for p in candidates
if p.exists()
and p.suffix == ".md"
and "wip" not in p.relative_to(source_dir).parts
and p.name != "README.md"
]
if not files:
print("No publishable candidates after filtering.")
return
print(f"Validating {len(files)} file(s)...")
posts = load_and_validate(files, source_dir)
sibling_map = build_sibling_map(posts)
print("Validation passed.")
# Render every page before writing any of them, so a bad link fails the run
# with the wiki untouched rather than half-updated.
rendered = {
abs_path: render_page(abs_path, source_dir, post, source_ref, gitea_repo_url, title_index)
for abs_path, post in selected.items()
}
print(f"Rendered {len(rendered)} page(s). Publishing.")
site = connect_wiki()
counts = {"published": 0, "noop": 0, "skipped-manual": 0}
for abs_path, post in posts.items():
for abs_path, post in selected.items():
rel = abs_path.relative_to(source_dir)
outcome = publish_one(abs_path, source_dir, post, site, source_ref, gitea_repo_url, sibling_map)
outcome = publish_one(abs_path, post, rendered[abs_path], site, source_ref)
counts[outcome] += 1
if outcome == "skipped-manual":
print(f" [skipped-manual] {rel} ← page exists without auto-gen banner; delete or add banner to hand over to bot", file=sys.stderr)
+7 -7
View File
@@ -1,6 +1,6 @@
# pipelines/tech
Publishes tech blog posts from `mikkeli/tech-blogs` to `https://wiki.novoyuuparosk.org`.
Publishes tech blog posts from `novoyuuparosk-wiki/tech-blogs` to `https://wiki.novoyuuparosk.org`.
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
@@ -10,7 +10,7 @@ Publishes tech blog posts from `mikkeli/tech-blogs` to `https://wiki.novoyuuparo
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
- Pandoc-based markdown → wikitext body rendering (h1 elements preserved)
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
- Category injection: `[[Category:Tech blog]]` on every page; `[[Category:Blog:<year>]]` from `date` field; `[[Category:<tag>]]` per tag entry
- Category injection: `[[Category:Tech blog]]` on every page; `[[Category:Blog:<year>]]` from `date` field (falling back to the action's execution year); `[[Category:<tag>]]` per tag entry
- No title prefix — pages live at their bare title
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
- Files without `wiki.publish: true` silently skipped
@@ -18,7 +18,7 @@ Publishes tech blog posts from `mikkeli/tech-blogs` to `https://wiki.novoyuuparo
## Source repo
Source repo: `mikkeli/tech-blogs` (not yet initialised — pipeline code is ready).
Source repo: `novoyuuparosk-wiki/tech-blogs` (not yet initialised — pipeline code is ready).
The pipeline triggers on push to `master` and supports `workflow_dispatch` for manual runs.
@@ -28,7 +28,7 @@ The pipeline triggers on push to `master` and supports `workflow_dispatch` for m
## Invocation
The pipeline runs as a self-contained Gitea Actions workflow in `mikkeli/tech-blogs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/tech-blogs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
Triggers:
- `push` to `master` (path-filtered as above)
@@ -65,14 +65,14 @@ Shared code lives in `lib/wiki.py` (repo root).
|---|---|---|
| No title prefix | Tech posts live at bare titles — no namespace needed | 2026-06-10 |
| Fixed category | Every post gets `Category:Tech blog` | 2026-06-10 |
| Year category | `Category:Blog:<year>` from `date` field; omitted if no date | 2026-06-10 |
| Year category | `Category:Blog:<year>` from `date` field; falls back to the execution year if no date | 2026-06-10 |
| Tag categories | Each tag → `[[Category:<tag>]]` with no prefix | 2026-06-10 |
| WIP exclusion | None — `wiki.publish: false` is the only gate | 2026-06-10 |
| Publish mode | Always `--all` | 2026-06-10 |
| Idempotency | Commit SHA in banner | 2026-06-10 |
| Manual-page protection | Bot skips pages without `{{Auto-generated` banner | 2026-06-10 |
| Source repo | `mikkeli/tech-blogs` — pipeline ships before repo is initialised | 2026-06-10 |
| Source repo | `novoyuuparosk-wiki/tech-blogs` — pipeline ships before repo is initialised | 2026-06-10 |
## Status
v1 pipeline ready. Source repo (`mikkeli/tech-blogs`) not yet initialised.
v1 pipeline ready. Source repo (`novoyuuparosk-wiki/tech-blogs`) not yet initialised.
+4 -4
View File
@@ -1,6 +1,6 @@
# SCHEMA — tech blog source files
Source-file contract for the tech blog pipeline. Files in `mikkeli/tech-blogs` must follow this contract to be picked up by the auto-publisher.
Source-file contract for the tech blog pipeline. Files in `novoyuuparosk-wiki/tech-blogs` must follow this contract to be picked up by the auto-publisher.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body is plain markdown, rendered to MediaWiki wikitext by Pandoc.
@@ -45,7 +45,7 @@ wiki:
#### `date` (ISO 8601 date)
Publication date in `YYYY-MM-DD` format. Drives `[[Category:Blog:<year>]]` injection. If omitted, no year category is added.
Publication date in `YYYY-MM-DD` format. Drives `[[Category:Blog:<year>]]` injection. If omitted, the year falls back to the time the publish action runs.
#### `tags` (list of strings)
@@ -69,7 +69,7 @@ For each `.md` with `wiki.publish: true`:
1. Parse and validate frontmatter.
2. Pipe the body through `pandoc -f markdown -t mediawiki`.
4. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
5. Append category tags: `[[Category:Tech blog]]`; `[[Category:Blog:<year>]]` if `date` is set; `[[Category:<tag>]]` for each tag.
5. Append category tags: `[[Category:Tech blog]]`; `[[Category:Blog:<year>]]` (from `date`, or the action's execution year if `date` is absent); `[[Category:<tag>]]` for each tag.
6. Read the current wiki page content; if identical, skip the write (idempotency).
7. If the page exists without `{{Auto-generated`, skip with a warning (manual page protection).
8. Otherwise, write with an edit summary referencing the source commit.
@@ -80,7 +80,7 @@ For each `.md` with `wiki.publish: true`:
- Pandoc-based markdown → wikitext rendering (h1 elements preserved)
- Banner template injection
- `Category:Tech blog` on every published page
- `Category:Blog:<year>` from `date` field
- `Category:Blog:<year>` from `date` field, falling back to the execution year
- `Category:<tag>` per entry in `tags`
- Idempotent writes
- Manual-page protection
+4 -3
View File
@@ -41,9 +41,10 @@ def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: st
banner = f"{{{{Auto-generated|source={source_url}|commit={source_ref}}}}}"
cat_parts = ["[[Category:Tech blog]]"]
date_val = fm.get("date")
if date_val:
year = str(date_val)[:4]
cat_parts.append(f"[[Category:Blog:{year}]]")
# Fall back to the action's execution year when no date is declared, so
# every post still lands in a Blog:<year> category.
year = str(date_val)[:4] if date_val else str(datetime.now().year)
cat_parts.append(f"[[Category:Blog:{year}]]")
for tag in (fm.get("tags") or []):
cat_parts.append(f"[[Category:{tag}]]")
categories = "\n".join(cat_parts) + "\n"
+1
View File
@@ -0,0 +1 @@
pytest>=8.0.0
+42
View File
@@ -0,0 +1,42 @@
import importlib.util
import sys
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT))
def _load_pipeline(name: str):
"""Import a pipeline's publish.py under a unique module name.
The three pipelines each define a module called `publish`, so they cannot be
imported by plain name without colliding in sys.modules.
"""
spec = importlib.util.spec_from_file_location(
f"{name}_publish", REPO_ROOT / "pipelines" / name / "publish.py"
)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
@pytest.fixture(scope="session")
def songs():
return _load_pipeline("songs")
@pytest.fixture
def repo(tmp_path):
"""Build a source tree of .md files: write(relative_path, frontmatter, body)."""
def write(rel: str, front: str = "", body: str = "") -> Path:
path = tmp_path / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(f"---\n{front}\n---\n\n{body}\n", encoding="utf-8")
return path
write.root = tmp_path
return write
+169
View File
@@ -0,0 +1,169 @@
"""Tests for the songs publisher: discovery, validation, and publish scoping.
None of these touch a wiki. `discover_publishable` and `select_publish_set` are
pure filesystem reads, which is the point — every failure they catch must fail
the run before `connect_wiki()` is ever called.
"""
import pytest
PUBLISHED = "title: {title}\nwiki:\n publish: true"
UNPUBLISHED = "title: {title}\nwiki:\n publish: false"
class TestDiscoverPublishable:
def test_collects_only_opted_in_files(self, songs, repo):
repo("a.md", PUBLISHED.format(title="A"))
repo("b.md", UNPUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
assert [p.metadata["title"] for p in posts.values()] == ["A"]
def test_skips_wip_directory_even_when_opted_in(self, songs, repo):
repo("a.md", PUBLISHED.format(title="A"))
repo("wip/draft.md", PUBLISHED.format(title="Draft"))
posts = songs.discover_publishable(repo.root)
assert [p.metadata["title"] for p in posts.values()] == ["A"]
def test_skips_readme(self, songs, repo):
repo("README.md", PUBLISHED.format(title="Readme"))
assert songs.discover_publishable(repo.root) == {}
def test_scans_nested_directories(self, songs, repo):
repo("ses/a.md", PUBLISHED.format(title="A"))
repo("kairo/deep/b.md", PUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
assert sorted(p.metadata["title"] for p in posts.values()) == ["A", "B"]
def test_duplicate_titles_in_different_directories_fail(self, songs, repo):
repo("ses/a.md", PUBLISHED.format(title="Same page"))
repo("kairo/b.md", PUBLISHED.format(title="Same page"))
with pytest.raises(ValueError, match="Duplicate wiki title 'Same page'"):
songs.discover_publishable(repo.root)
def test_duplicate_title_error_names_both_files(self, songs, repo):
repo("ses/a.md", PUBLISHED.format(title="Same page"))
repo("kairo/b.md", PUBLISHED.format(title="Same page"))
with pytest.raises(ValueError) as err:
songs.discover_publishable(repo.root)
assert "a.md" in str(err.value) and "b.md" in str(err.value)
def test_a_title_colliding_with_an_unpublished_file_is_allowed(self, songs, repo):
repo("a.md", PUBLISHED.format(title="Same page"))
repo("b.md", UNPUBLISHED.format(title="Same page"))
assert len(songs.discover_publishable(repo.root)) == 1
def test_publish_as_a_string_is_rejected(self, songs, repo):
repo("a.md", 'title: A\nwiki:\n publish: "true"')
with pytest.raises(ValueError, match="must be a boolean"):
songs.discover_publishable(repo.root)
def test_missing_title_is_rejected(self, songs, repo):
repo("a.md", "wiki:\n publish: true")
with pytest.raises(ValueError, match="missing or empty 'title'"):
songs.discover_publishable(repo.root)
def test_bad_release_date_is_rejected(self, songs, repo):
repo("a.md", PUBLISHED.format(title="A") + "\nrelease_date: 14-06-2026")
with pytest.raises(ValueError, match="YYYY-MM-DD"):
songs.discover_publishable(repo.root)
def test_categories_must_be_a_list(self, songs, repo):
repo("a.md", PUBLISHED.format(title="A") + "\ncategories: kairo")
with pytest.raises(ValueError, match="'categories' must be a list"):
songs.discover_publishable(repo.root)
def test_a_leftover_siblings_key_is_ignored_not_an_error(self, songs, repo):
# The old macro declaration is inert; a stale one must not fail a run.
repo("a.md", PUBLISHED.format(title="A") + "\n siblings:\n - tag: X")
assert len(songs.discover_publishable(repo.root)) == 1
class TestSelectPublishSet:
def test_publishes_only_the_changed_file(self, songs, repo):
a = repo("a.md", PUBLISHED.format(title="A"))
repo("b.md", PUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
selected = songs.select_publish_set(posts, {a.resolve()})
assert [p.metadata["title"] for p in selected.values()] == ["A"]
def test_arms_the_page_a_changed_page_links_to(self, songs, repo):
a = repo("a.md", PUBLISHED.format(title="A"), "see [notes](b.md)")
repo("b.md", PUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
selected = songs.select_publish_set(posts, {a.resolve()})
assert sorted(p.metadata["title"] for p in selected.values()) == ["A", "B"]
def test_arms_the_page_that_links_to_a_changed_page(self, songs, repo):
repo("a.md", PUBLISHED.format(title="A"), "see [notes](b.md)")
b = repo("b.md", PUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
selected = songs.select_publish_set(posts, {b.resolve()})
assert sorted(p.metadata["title"] for p in selected.values()) == ["A", "B"]
def test_unrelated_pages_stay_out(self, songs, repo):
a = repo("a.md", PUBLISHED.format(title="A"), "see [notes](b.md)")
repo("b.md", PUBLISHED.format(title="B"))
repo("c.md", PUBLISHED.format(title="C"))
posts = songs.discover_publishable(repo.root)
selected = songs.select_publish_set(posts, {a.resolve()})
assert "C" not in [p.metadata["title"] for p in selected.values()]
def test_links_across_directories_are_followed(self, songs, repo):
a = repo("ses/a.md", PUBLISHED.format(title="A"), "see [notes](../kairo/b.md)")
repo("kairo/b.md", PUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
selected = songs.select_publish_set(posts, {a.resolve()})
assert sorted(p.metadata["title"] for p in selected.values()) == ["A", "B"]
def test_a_link_to_an_unpublished_file_adds_nothing(self, songs, repo):
# resolve_md_links is what rejects this loudly; the graph just drops it.
a = repo("a.md", PUBLISHED.format(title="A"), "see [draft](b.md)")
repo("b.md", UNPUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
selected = songs.select_publish_set(posts, {a.resolve()})
assert [p.metadata["title"] for p in selected.values()] == ["A"]
def test_a_self_link_does_not_duplicate(self, songs, repo):
a = repo("a.md", PUBLISHED.format(title="A"), "see [self](a.md)")
posts = songs.discover_publishable(repo.root)
selected = songs.select_publish_set(posts, {a.resolve()})
assert len(selected) == 1
class TestBuildTitleIndex:
def test_maps_every_publishable_path_to_its_title(self, songs, repo):
a = repo("a.md", PUBLISHED.format(title="A"))
posts = songs.discover_publishable(repo.root)
assert songs.build_title_index(posts)[str(a.resolve())] == "A"
def test_covers_the_whole_repo_not_just_a_changed_file(self, songs, repo):
repo("a.md", PUBLISHED.format(title="A"))
b = repo("nested/b.md", PUBLISHED.format(title="B"))
posts = songs.discover_publishable(repo.root)
assert str(b.resolve()) in songs.build_title_index(posts)
class TestBuildWikitext:
def test_album_injects_both_song_and_album_categories(self, songs):
out = songs.build_wikitext({"album": "kairo II"}, "body\n", "url", "sha")
assert "[[Category:Songs]]" in out
assert "[[Category:Album:kairo II]]" in out
def test_no_album_injects_no_category(self, songs):
out = songs.build_wikitext({}, "body\n", "url", "sha")
assert "[[Category:" not in out
def test_extra_categories_are_additive(self, songs):
out = songs.build_wikitext(
{"album": "kairo II", "categories": ["kairo"]}, "body\n", "url", "sha"
)
assert "[[Category:kairo]]" in out
assert "[[Category:Album:kairo II]]" in out
def test_banner_carries_source_and_commit(self, songs):
out = songs.build_wikitext({}, "body\n", "http://g/x", "abc123")
assert out.startswith("{{Auto-generated|source=http://g/x|commit=abc123}}")
def test_lrc_footer_link_is_appended_when_named(self, songs):
out = songs.build_wikitext({}, "body\n", "url", "sha", "A.lrc")
assert "[[Media:A.lrc|Synced lyrics (.lrc)]]" in out
+216
View File
@@ -0,0 +1,216 @@
"""Tests for the shared render helpers in lib/wiki.py.
All pure string in, string out, except the pandoc contract test at the bottom,
which is the canary for a pandoc upgrade changing how relative links are written.
"""
import shutil
import subprocess
from pathlib import Path
import pytest
from lib.wiki import (
assert_no_unresolved_md_links,
expand_columns,
extract_md_link_targets,
markdown_to_wikitext,
resolve_md_links,
strip_first_h1,
)
# Paths need not exist on disk: resolution is non-strict, and only the missing
# vs. unpublished distinction consults the filesystem (covered with tmp_path).
SONGS = Path(__file__).resolve().parent / "_paths" / "songs"
HERE = SONGS / "pulse.md"
INDEX = {
str((SONGS / "pulse.md").resolve()): "pulse under latent semantic envelopes",
str((SONGS / "pulse_dev.md").resolve()): (
"pulse under latent semantic envelopes - development notes"
),
str((SONGS / "two words.md").resolve()): "Two words",
}
def resolve(wikitext, source=HERE):
return resolve_md_links(wikitext, source, INDEX)
class TestStripFirstH1:
def test_removes_heading_and_following_blank_line(self):
assert strip_first_h1("# Title\n\nbody") == "body"
def test_removes_only_the_first(self):
assert strip_first_h1("# One\n\n# Two\n") == "# Two\n"
def test_leaves_body_without_h1_alone(self):
assert strip_first_h1("## Sub\n\nbody") == "## Sub\n\nbody"
def test_finds_the_h1_even_when_a_deeper_heading_precedes_it(self):
assert strip_first_h1("### Deep\n\n# Title\n") == "### Deep\n"
class TestExpandColumns:
def test_splits_on_separator_and_wraps_each_in_poem(self):
out = expand_columns('<pre class="columns">a\nb\n===\nc</pre>')
assert out.count("<poem>") == 2
assert "<poem>\na\nb\n</poem>" in out
assert "<poem>\nc\n</poem>" in out
assert out.startswith("<div style=")
def test_three_columns_need_two_separators(self):
out = expand_columns('<pre class="columns">a\n===\nb\n===\nc</pre>')
assert out.count("<poem>") == 3
def test_converts_markdown_emphasis(self):
out = expand_columns('<pre class="columns">***a*** **b** *c*</pre>')
assert "'''''a'''''" in out
assert "'''b'''" in out
assert "''c''" in out
def test_unescapes_inline_html(self):
out = expand_columns('<pre class="columns">&lt;b&gt;x&lt;/b&gt;</pre>')
assert "<b>x</b>" in out
def test_leaves_wikitext_without_a_columns_block_untouched(self):
assert expand_columns("plain '''wikitext'''") == "plain '''wikitext'''"
class TestResolveMdLinks:
def test_labelled_link_becomes_piped_wikilink(self):
assert resolve("see [[pulse_dev.md|dev notes]] ok") == (
"see [[pulse under latent semantic envelopes - development notes|dev notes]] ok"
)
def test_unlabelled_link_uses_the_page_title(self):
assert resolve("[[pulse_dev.md]]") == (
"[[pulse under latent semantic envelopes - development notes]]"
)
def test_label_identical_to_target_drops_the_pipe(self):
assert resolve("[[pulse_dev.md|pulse_dev.md]]") == (
"[[pulse under latent semantic envelopes - development notes]]"
)
def test_label_identical_to_the_page_title_drops_the_pipe(self):
assert resolve("[[pulse_dev.md|pulse under latent semantic envelopes - development notes]]") == (
"[[pulse under latent semantic envelopes - development notes]]"
)
def test_fragment_is_carried_onto_the_title(self):
assert resolve("[[pulse_dev.md#Notes|x]]") == (
"[[pulse under latent semantic envelopes - development notes#Notes|x]]"
)
def test_percent_encoding_in_target_is_decoded(self):
assert resolve("[[two%20words.md|x]]") == "[[Two words|x]]"
def test_parent_directory_target_resolves(self):
assert resolve_md_links(
"[[../pulse_dev.md|x]]", SONGS / "sub" / "a.md", INDEX
) == "[[pulse under latent semantic envelopes - development notes|x]]"
def test_external_style_link_shape_also_resolves(self):
# Insurance against a pandoc version writing relative targets this way.
assert resolve("[pulse_dev.md dev notes]") == (
"[[pulse under latent semantic envelopes - development notes|dev notes]]"
)
def test_absolute_url_is_left_alone(self):
assert resolve("[https://example.com/x label]") == "[https://example.com/x label]"
def test_non_md_wikilink_is_left_alone(self):
assert resolve("[[Category:Songs]] [[Media:x.lrc|y]]") == (
"[[Category:Songs]] [[Media:x.lrc|y]]"
)
def test_label_containing_a_pipe_is_preserved(self):
assert resolve("[[pulse_dev.md|a|b]]").endswith("|a|b]]")
def test_several_links_in_one_body(self):
out = resolve("[[pulse_dev.md|a]] then [[two%20words.md|b]]")
assert "|a]]" in out and "[[Two words|b]]" in out
def test_missing_file_is_an_error(self):
with pytest.raises(ValueError, match="does not exist"):
resolve("[[nope.md|x]]")
def test_existing_but_unpublished_file_is_an_error(self, tmp_path):
(tmp_path / "draft.md").write_text("x", encoding="utf-8")
with pytest.raises(ValueError, match="not published"):
resolve_md_links("[[draft.md|x]]", tmp_path / "song.md", {})
def test_error_names_the_source_file(self):
with pytest.raises(ValueError, match=r"pulse\.md"):
resolve("[[nope.md|x]]")
class TestUnresolvedLinkGuard:
def test_accepts_output_with_no_md_reference(self):
assert_no_unresolved_md_links("[[A page|x]] and [http://e.com y]", HERE)
def test_rejects_leftover_wikilink(self):
with pytest.raises(ValueError, match="unresolved wikilink"):
assert_no_unresolved_md_links("[[notes.md|x]]", HERE)
def test_rejects_leftover_external_style_link(self):
with pytest.raises(ValueError, match="external-style"):
assert_no_unresolved_md_links("[notes.md x]", HERE)
def test_rejects_raw_markdown_link_left_in_a_columns_fence(self):
with pytest.raises(ValueError, match="columns"):
assert_no_unresolved_md_links("<poem>\nsee [x](notes.md)\n</poem>", HERE)
def test_prose_mentioning_a_filename_is_not_a_link(self):
assert_no_unresolved_md_links("edit <code>pulse.md</code> by hand", HERE)
class TestExtractMdLinkTargets:
def test_finds_inline_targets(self):
assert extract_md_link_targets("a [x](one.md) b [y](../two.md#f)") == [
"one.md",
"../two.md#f",
]
def test_ignores_bodies_without_links(self):
assert extract_md_link_targets("no links here") == []
def test_finds_targets_inside_a_columns_fence(self):
# The fence cannot render them, but the graph must still see the edge.
assert extract_md_link_targets("```columns\n[x](one.md)\n```") == ["one.md"]
@pytest.mark.skipif(shutil.which("pandoc") is None, reason="pandoc not installed")
class TestPandocContract:
"""Pins the pandoc output shape that resolve_md_links depends on.
If a pandoc upgrade starts writing relative link targets differently, this
fails here instead of silently publishing a broken link.
"""
def test_version(self):
out = subprocess.run(
["pandoc", "--version"], capture_output=True, text=True, check=True
).stdout
print(out.splitlines()[0])
def test_relative_link_survives_the_full_render(self):
wikitext = markdown_to_wikitext("see [dev notes](pulse_dev.md) here")
assert resolve(wikitext).strip() == (
"see [[pulse under latent semantic envelopes - development notes|dev notes]] here"
)
def test_unlabelled_relative_link_survives_the_full_render(self):
wikitext = markdown_to_wikitext("[pulse_dev.md](pulse_dev.md)")
assert resolve(wikitext).strip() == (
"[[pulse under latent semantic envelopes - development notes]]"
)
def test_absolute_url_survives_untouched(self):
wikitext = markdown_to_wikitext("[site](https://example.com/x)")
assert resolve(wikitext).strip() == "[https://example.com/x site]"
def test_columns_fence_renders_to_side_by_side_poems(self):
wikitext = markdown_to_wikitext("```columns\n**A**\nl1\n===\n**B**\nl2\n```")
assert wikitext.count("<poem>") == 2
assert "'''A'''" in wikitext