novoyuuparosk-wiki/novoyuuparosk-auto-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

songs: replace the wiki.siblings macro with plain Markdown links

Browse Source 
The `wiki.siblings` bill of materials is gone. A related co-publication is
now nominated with an ordinary relative Markdown link — `[label](notes.md)` —
resolved at publish time against the target file's own `title` frontmatter.
No macro namespace, no per-file declaration, and the link still works in
Gitea's blob view and any editor preview.

The macro could not be made safe. Substitution was `str.replace(tag, …)` over
the whole rendered page: no word boundary, so a tag also occurring in lyrics or
a code block got rewritten; no collision check, so overlapping tags corrupted
each other in declaration order. SCHEMA.md documented the placeholder as
`[[TAG]]` while the code replaced the bare tag, so following the schema
produced `[[[[Title|label]]]]`.

Fixes a live corruption bug in the process. `sibling_map` was built only from
the files in the publish batch, and an unresolved sibling degraded to plain
text — so pushing `ses/pulse.md` alone republished it with its wikilink
silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the
workflow "always runs --all", which it never did on an ordinary push.

The repository is now always scanned in full, whatever a run publishes, so
link resolution and title-collision detection can never depend on which files
a push touched. Writes stay bounded to the push diff plus one hop in each link
direction: the page a changed page links to (so a link cannot point at an
unpublished page) and the pages linking to it (so an edited `title` does not
leave stale inbound links). Publishing is not instant; reading is cheap.

An unresolvable `.md` link is now a hard error, never graceful degradation, and
linking to a file still does not publish it — `wiki.publish: true` remains the
only gate. A leftover-link guard fails the run on any link shape the resolver
did not recognise, including a Markdown link written inside a verbatim
```columns fence, so a pandoc upgrade cannot silently publish literal text.

Also:

- render before write: every page in a run is rendered before the first is
  published, so a broken page leaves the wiki untouched instead of
  half-updated. `publish_one` is split into `render_page` (pure) and the write.
- pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale
  encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside
  subprocess's reader thread, `stdout` came back None, and the real error was
  lost behind a TypeError. Latent on the Debian runner via PEP 538 only.
- title collisions are detected repo-wide, not just within a push, and before
  any wiki contact.
- tests/, run by .gitea/workflows/test.yml in the publish runner image. The
  pandoc output contract the resolver depends on is now pinned by a test
  instead of by prose — which is how SCHEMA.md came to assert something false.
- drop an f-string with nothing to interpolate.
- quote the SOURCE_REF interpolation in the workflow instead of pasting an
  input straight into shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Wan Ziyu
2026-09-09 16:16:33 +09:00
parent 4e7d84d665
commit f5625cde98
10 changed files with 838 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipelines/songs/README.md
+16 -5
View File
@@ -15,7 +15,8 @@ Implemented:
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set); additional categories via `categories` list
- LRC file upload (idempotent SHA1 check) + footer link
- Inter-page sibling placeholder resolution (`[[TAG]]` → `[[wiki page title|label]]`)
- Inter-page link resolution: a relative Markdown link (`[label](notes.md)`) becomes `[[wiki page title|label]]`, resolved against a repo-wide index so it never depends on which files a push touched
- Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
- Files without `wiki.publish: true` silently skipped
- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
@@ -65,7 +66,12 @@ See the `Dockerfile` at the repo root and `requirements.txt` in this directory.
## Modes
The pipeline always runs `--all`: every file with `wiki.publish: true` (not under `wip/`) is checked against the current wiki content. Writes only happen when content has changed (commit SHA in the banner drives idempotency). This is appropriate for the current repo size.
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every file with `wiki.publish: true` (not under `wip/`), because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page.
- `--all` — publish every publishable file. Used on the first push and on `workflow_dispatch`.
- `--files <paths…>` — publish these files and their one-hop link neighbours. Used for an ordinary push.
Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read.
## One-off wiki setup
@@ -82,11 +88,16 @@ The pipeline always runs `--all`: every file with `wiki.publish: true` (not unde
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
| Album categories | Auto-injected `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set; category pages are user-written and not overwritten | 2026-06-09 |
| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:<value>]]` | 2026-06-10 |
| `album` optional | Companion/sibling pages don't need an album; omitting the field emits no album category | 2026-06-10 |
| Sibling resolution | Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc | 2026-06-10 |
| `album` optional | Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 |
| ~~Sibling resolution~~ | ~~Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc~~ — **retired 2026-09-09**, see below | 2026-06-10 |
| Inter-page links | Plain relative Markdown link (`[label](notes.md)`); target's own `title` frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the `wiki.siblings` bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch | 2026-09-09 |
| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 |
| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 |
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
| Publish mode | Always `--all`; no diff detection — small repo, simpler than fragile git-diff gating | 2026-06-09 |
| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 |
| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 |
| Tests | `tests/` run by `.gitea/workflows/test.yml` in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose | 2026-09-09 |
| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 |
| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 |
| Side-by-side columns | Authored as a ` ```columns ` fenced block (separator `===`); Pandoc emits `<pre class="columns">`, expanded post-Pandoc into a flex `<div>` of `<poem>` columns. No wiki template or extension — all Pi-side before the API write. See [SCHEMA.md](SCHEMA.md#columns-side-by-side-shorthand) | 2026-06-14 |
pipelines/songs/SCHEMA.md
+84 -69
View File
@@ -2,7 +2,7 @@
Source-file contract for the songs pipeline. Files in `novoyuuparosk-wiki/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar, except that `[[TAG]]` placeholders declared via `wiki.siblings` are substituted with resolved wikilinks after rendering.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar. There is no macro syntax and nothing to declare: a link to another song page is an ordinary relative Markdown link, resolved to a wikilink at publish time.
## File layout
@@ -12,20 +12,16 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
# pulse under latent semantic envelopes
... body markdown, PULSE_DEV used inline ...
... body markdown; a related page is nominated as [development notes](pulse_dev.md) ...
```
The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc, then applies sibling substitutions on the wikitext output.
The renderer parses the frontmatter, validates required fields, hands the body (with the first `# Heading` line removed) to Pandoc, then resolves `.md` links in the wikitext output to the wiki page titles of their targets.
## Fields
@@ -81,54 +77,54 @@ Example: in `ses/pulse.md`, `lrc: pulse.lrc` declares that `ses/pulse.lrc` is th
If the LRC file is missing or not valid UTF-8, the upload is skipped with a warning — the page still publishes.
#### `wiki.siblings` (list of sibling declarations)
Bill of materials for inter-page wikilinks. Each entry declares a sibling `.md` file and a tag that can be used as a `[[TAG]]` placeholder anywhere in the body.
```yaml
wiki:
siblings:
- path: pulse_dev.md # relative to this file's directory
tag: PULSE_DEV # placeholder used in body as [[PULSE_DEV]]
label: development notes # optional display text
```
**Fields per entry:**
| Field | Required for resolution | Purpose |
|---|---|---|
| `path` | yes | Relative path to the sibling `.md` file |
| `tag` | yes | Placeholder identifier; pipeline does a case-sensitive string replace of `TAG` in the rendered wikitext |
| `label` | no | Display text for the resolved link |
An entry missing either `path` or `tag` is silently skipped — no substitution is attempted for that entry.
**Resolution:**
The pipeline resolves each declared sibling's `path` to its wiki page title by looking it up in the set of all publishable files processed in the same run. Substitution is a case-sensitive string replace of the bare tag. Substitution rules:
| Condition | `TAG` becomes |
|---|---|
| `path` resolves to a published page, `label` present | `[[wiki page title\|label]]` |
| `path` resolves to a published page, no `label` | `[[wiki page title]]` |
| `path` not in published set (file exists but not published), `label` present | `label` (plain text) |
| `path` not in published set, no `label` | `TAG` (plain text) |
Declared siblings whose file doesn't exist on disk are a **validation error**. Siblings that exist but aren't published (no `wiki.publish: true`) degrade gracefully per the table above.
The siblings list is a bill of materials — declaring a sibling has no effect unless its `[[TAG]]` appears in the body. Unused entries are inert.
**`--files` mode**: sibling resolution is limited to the files in that batch. Siblings pointing to files outside the batch degrade to plain text. This is acceptable since the pipeline always runs `--all` in the workflow.
## Body
Plain markdown. The renderer applies these transformations before and after Pandoc:
1. **Pre-Pandoc**: strip the first-line `# Heading` if present.
2. **Post-Pandoc**: expand `columns` fenced blocks into side-by-side wikitext (see below).
3. **Post-Pandoc**: substitute `TAG` placeholders declared in `wiki.siblings`.
3. **Post-Pandoc**: resolve links to sibling `.md` files into wikilinks (see below).
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, sibling resolution) around Pandoc's output.
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, link resolution) around Pandoc's output.
### Nominating a related co-publication
Link to it the ordinary way, with a relative Markdown link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
There is nothing to declare in the frontmatter. The link stays a working link in Gitea's blob view, in an editor preview, and anywhere else the raw file is read — and on the wiki it becomes a wikilink to the target's page:
```wikitext
For more behind-the-scene shenanigans, see [[pulse under latent semantic envelopes - development notes|dev gibberish]].
```
**How the target becomes a page name.** The path is resolved relative to the linking file's directory, and the target file's own `title` frontmatter field is used verbatim as the wikilink target — the same string the bot writes that page to. There is no second name format to keep in sync: rename a page by editing its `title`, and every link aimed at it follows on the next publish. MediaWiki handles the first-letter capitalisation and the space-to-underscore in URLs itself.
| Written in the body | Rendered wikitext |
|---|---|
| `[dev gibberish](pulse_dev.md)` | `[[<target's title>\|dev gibberish]]` |
| `[pulse_dev.md](pulse_dev.md)` (label same as path) | `[[<target's title>]]` |
| `[notes](../ses/pulse_dev.md)` | `[[<target's title>\|notes]]` |
| `[notes](pulse_dev.md#Section)` | `[[<target's title>#Section\|notes]]` |
| `[site](https://example.com)` | `[https://example.com site]` (untouched) |
**A `.md` link that cannot be resolved fails the run.** Both cases abort with a non-zero exit before anything is published:
| Condition | Result |
|---|---|
| Target file does not exist | **error** — `file does not exist` |
| Target exists but is not published (no `wiki.publish: true`, or under `wip/`) | **error** — `file exists but is not published` |
Linking to a file does **not** publish it. Publication is always the target's own explicit `wiki.publish: true`; nothing is published as a side effect of being linked. If you want the link, opt the target in.
**Scope of a run.** A push publishes the files it changed, plus — one hop in each direction — any page a changed page links to, and any page that links to a changed page. The second is what keeps inbound links correct when a page's `title` moves. Regardless of how few pages a run publishes, the whole repository is always scanned first, so link resolution and title-collision checks see every publishable file.
**Anchors** (`pulse_dev.md#Section`) are passed through verbatim. MediaWiki derives its own section anchors from heading text, so an anchor that doesn't match a heading on the target page lands the reader at the top of it rather than at the section. Nothing validates this.
**Links inside a `columns` fence do not work.** The fence is verbatim by design (see below); Pandoc does not process Markdown in it, so a link written there would reach the wiki as literal text. The renderer detects this and fails the run rather than publishing it. Put the link in ordinary body prose.
### Columns (side-by-side) shorthand
@@ -157,29 +153,43 @@ The renderer turns this into a flexbox row of `<poem>` columns (one per `===`-de
## Renderer behaviour
For each `.md` with `wiki.publish: true` and not under `wip/`:
A run happens in three phases, in this order, and the wiki is not contacted until the third. Anything wrong in phase 1 or 2 aborts with a non-zero exit and the wiki untouched — never half-updated.
1. Parse and validate frontmatter.
2. Strip the leading `# Heading` from the body if present.
3. Pipe the body through `pandoc -f markdown -t mediawiki`, then expand any `columns` fenced blocks into side-by-side `<poem>` columns.
4. Substitute `TAG` placeholders with resolved `[[Page|Display]]` wikilinks (or fallback text).
5. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
6. If `lrc` is declared: upload the LRC file if its SHA1 has changed; append a `[[Media:…]]` link.
7. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
9. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
10. Otherwise, write the page with an edit summary referencing the source commit.
**Phase 1 — scan and validate the whole repository.** For every `.md` outside `wip/` (and excluding `README.md`):
1. Parse the frontmatter; skip the file silently unless `wiki.publish: true`.
2. Validate the fields (see *Validation errors* below).
3. Register `title` and fail on a collision with any other publishable file, anywhere in the tree.
4. Build the path → page-title index every `.md` link resolves through.
**Phase 2 — render every page that this run will publish.** For each selected file:
1. Strip the leading `# Heading` from the body if present.
2. Pipe the body through `pandoc -f markdown -t mediawiki`, then expand any `columns` fenced blocks into side-by-side `<poem>` columns.
3. Resolve `.md` links to `[[Page title|label]]` wikilinks; fail on any that cannot be resolved, and on any link shape that survived rendering unresolved.
4. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
5. If `lrc` is declared, append a `[[Media:…]]` link.
6. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
**Phase 3 — write.** For each rendered page:
1. If `lrc` is declared: upload the LRC file if its SHA1 has changed.
2. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
3. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
4. Otherwise, write the page with an edit summary referencing the source commit.
## Implemented scope
- Frontmatter parsing and validation
- Repo-wide page-title collision detection, before any wiki write
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Banner template injection
- LRC file upload and footer link
- Sibling placeholder resolution (`[[TAG]]` → `[[wiki page title|label]]`)
- Relative `.md` link resolution (`[label](notes.md)` → `[[wiki page title|label]]`), repo-wide and batch-independent
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]`) when `album` is set
- Additional categories via `categories` list
- Render-before-write: a broken page fails the run instead of half-updating the wiki
- Idempotent writes (no-op skip when content matches)
- Manual-page protection (skip pages without auto-gen banner)
- Files without `wiki.publish: true` silently skipped
@@ -199,16 +209,18 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
```
### A companion/sibling document (no album)
Its body nominates the companion page with a plain relative link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
### A companion document (no album)
```yaml
---
@@ -249,8 +261,11 @@ Files without a `wiki.publish: true` frontmatter field are silently skipped —
- `wiki.publish` present but not a boolean
- `release_date` (if present) does not parse as YYYY-MM-DD
- `categories` (if present) is not a list
- `wiki.siblings` entry that has both `path` and `tag` declared, but the `path` file does not exist on disk
- `wiki.siblings` entry that is not a mapping
- Two publishable source files declare the same `title` (page-name collision)
- Two publishable source files declare the same `title` (page-name collision) — checked across the whole repository, not just the files a push touched
- A `.md` link whose target file does not exist
- A `.md` link whose target exists but is not published
- A `.md` link that survives rendering unresolved — a link shape the renderer does not recognise, or a Markdown link written inside a `columns` fence
Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.
Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken.
Frontmatter keys the schema does not define are ignored, so a leftover `wiki.siblings` block from the retired macro system is inert rather than an error. Delete them; they do nothing.
pipelines/songs/publish.py
+130 -75
View File
@@ -6,6 +6,7 @@ import hashlib
import io
import os
import sys
import urllib.parse
from datetime import datetime
from pathlib import Path
@@ -13,7 +14,14 @@ import frontmatter
import mwclient
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
from lib.wiki import AUTO_BANNER_PREFIX, connect_wiki, markdown_to_wikitext, strip_first_h1
from lib.wiki import (
AUTO_BANNER_PREFIX,
connect_wiki,
extract_md_link_targets,
markdown_to_wikitext,
resolve_md_links,
strip_first_h1,
)
def validate(fm: dict, path: Path) -> None:
@@ -35,47 +43,58 @@ def validate(fm: dict, path: Path) -> None:
if "categories" in fm and not isinstance(fm["categories"], list):
errors.append("'categories' must be a list")
for i, sib in enumerate(fm.get("wiki", {}).get("siblings") or []):
if not isinstance(sib, dict):
errors.append(f"wiki.siblings[{i}] must be a mapping")
continue
sib_path = sib.get("path")
sib_tag = sib.get("tag")
if sib_path and sib_tag:
if not (path.parent / str(sib_path)).resolve().exists():
errors.append(f"wiki.siblings[{i}]: path '{sib_path}' not found")
if errors:
raise ValueError(f"{path}: " + "; ".join(errors))
def build_sibling_map(posts: dict) -> dict[str, str]:
def build_title_index(posts: dict) -> dict[str, str]:
"""Map abs_path_str → wiki_title for all published posts."""
return {str(path): str(post.metadata["title"]) for path, post in posts.items()}
def apply_sibling_substitutions(wikitext: str, this_path: Path, fm_wiki: dict, sibling_map: dict[str, str]) -> str:
"""Replace [[TAG]] placeholders using the declared sibling bill of materials."""
for sib in (fm_wiki.get("siblings") or []):
if not isinstance(sib, dict):
continue
tag = sib.get("tag")
path = sib.get("path")
if not tag or not path:
continue
label = (sib.get("label") or "").strip()
abs_sib = str((this_path.parent / str(path)).resolve())
title = sibling_map.get(abs_sib)
if title and label:
replacement = f"[[{title}|{label}]]"
elif title:
replacement = f"[[{title}]]"
elif label:
replacement = label
else:
replacement = tag
wikitext = wikitext.replace(tag, replacement)
return wikitext
def build_link_graph(posts: dict) -> dict[Path, set[Path]]:
"""Map each publishable file to the publishable .md files it links to.
Read from the raw Markdown, so it costs no Pandoc runs. Targets outside the
publishable set are dropped here; `resolve_md_links` is what refuses them
loudly at render time.
"""
graph: dict[Path, set[Path]] = {}
for path, post in posts.items():
targets = set()
for raw in extract_md_link_targets(post.content):
rel_path = urllib.parse.unquote(raw.partition("#")[0])
if not rel_path.lower().endswith(".md"):
continue
resolved = (path.parent / rel_path).resolve()
if resolved in posts:
targets.add(resolved)
graph[path] = targets
return graph
def select_publish_set(posts: dict, seeds: set[Path]) -> dict[Path, object]:
"""The changed files, plus the pages their links touch in either direction.
Publishing is not instant, so only what a push actually affects is written.
Two extra hops are unavoidable, both one level deep:
- **outbound**: a page the changed page links to, so the wikilink cannot
point at a page that was never published (a red link);
- **inbound**: a page linking *to* a changed page, because if the changed
page's `title` moved, every link aimed at it is now stale.
Ordering follows `posts` so a run's output is stable regardless of which
files the diff reported.
"""
graph = build_link_graph(posts)
selected = set(seeds)
for path, targets in graph.items():
if path in seeds:
selected |= targets
elif targets & seeds:
selected.add(path)
return {path: post for path, post in posts.items() if path in selected}
def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: str, lrc_filename: str | None = None) -> str:
@@ -84,7 +103,7 @@ def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: st
cat_parts = []
if fm.get("album"):
cat_parts.append(f"[[Category:Songs]]")
cat_parts.append("[[Category:Songs]]")
cat_parts.append(f"[[Category:Album:{fm['album']}]]")
for cat in (fm.get("categories") or []):
cat_parts.append(f"[[Category:{cat}]]")
@@ -121,25 +140,33 @@ def upload_lrc(lrc_path: Path, title: str, site: mwclient.Site) -> bool:
return True
def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, source_ref: str, gitea_repo_url: str, sibling_map: dict[str, str]) -> str:
"""Returns 'published', 'noop', or 'skipped-manual'."""
def render_page(abs_path: Path, source_dir: Path, post, source_ref: str, gitea_repo_url: str, title_index: dict[str, str]) -> str:
"""Render one file to finished wikitext. Pure: touches no wiki, writes nothing.
Kept separate from the write so that every page in a run is rendered before
the first one is published. A bad link or a broken column fence then fails
the run with the wiki untouched, instead of half-updating it.
"""
rel = abs_path.relative_to(source_dir)
fm = post.metadata
title = str(fm["title"])
body_wikitext = markdown_to_wikitext(strip_first_h1(post.content))
body_wikitext = apply_sibling_substitutions(body_wikitext, abs_path, fm.get("wiki", {}), sibling_map)
body_wikitext = resolve_md_links(body_wikitext, abs_path, title_index)
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
lrc_filename = None
lrc_rel = fm.get("lrc")
if lrc_rel:
lrc_uploaded = upload_lrc(abs_path.parent / str(lrc_rel), title, site)
lrc_filename = f"{title}.lrc"
if lrc_uploaded:
print(f" [lrc-uploaded] {lrc_filename}")
lrc_filename = f"{title}.lrc" if fm.get("lrc") else None
return build_wikitext(fm, body_wikitext, source_url, source_ref, lrc_filename)
page_content = build_wikitext(fm, body_wikitext, source_url, source_ref, lrc_filename)
def publish_one(abs_path: Path, post, page_content: str, site: mwclient.Site, source_ref: str) -> str:
"""Write one pre-rendered page. Returns 'published', 'noop', or 'skipped-manual'."""
fm = post.metadata
title = str(fm["title"])
lrc_rel = fm.get("lrc")
if lrc_rel and upload_lrc(abs_path.parent / str(lrc_rel), title, site):
print(f" [lrc-uploaded] {title}.lrc")
page = site.pages[title]
existing = page.text()
@@ -155,13 +182,29 @@ def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, sou
return "published"
def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]:
posts = {}
def discover_publishable(source_dir: Path) -> dict[Path, object]:
"""Every publishable .md in the repository, validated, keyed by absolute path.
The whole tree is scanned on every run, never a subset of it, however few
pages the run then publishes. Two things depend on seeing all of it:
- **link resolution**: a link to a sibling .md resolves through this set, so
a partial scan would resolve the same link differently depending on which
files a push happened to touch, and silently drop it when the target fell
outside the subset;
- **title collisions**: two files claiming one wiki page name must fail the
run wherever they sit in the tree, not only when both land in one push.
Files under wip/ and README.md are never published. Everything else needs an
explicit `wiki.publish: true`; a file is never published as a side effect of
something else linking to it.
"""
posts: dict[Path, object] = {}
titles: dict[str, Path] = {}
for path in files:
if not path.exists():
print(f" warn: {path} not found, skipping", file=sys.stderr)
for path in sorted(source_dir.rglob("*.md")):
rel = path.relative_to(source_dir)
if "wip" in rel.parts or path.name == "README.md":
continue
post = frontmatter.load(str(path))
@@ -175,10 +218,11 @@ def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]
title = str(post.metadata["title"])
if title in titles:
raise ValueError(
f"Duplicate wiki title '{title}': {path} and {titles[title]}"
f"Duplicate wiki title '{title}': {rel} and "
f"{titles[title].relative_to(source_dir)}"
)
titles[title] = path
posts[path] = post
posts[path.resolve()] = post
return posts
@@ -186,45 +230,56 @@ def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]
def main() -> None:
parser = argparse.ArgumentParser(description="Publish song .md files to MediaWiki.")
parser.add_argument("--source-dir", required=True, type=Path)
parser.add_argument("--files", nargs="*", default=[], help="Relative paths within source-dir")
parser.add_argument("--all", action="store_true", help="Publish all publishable files")
parser.add_argument(
"--files",
nargs="*",
default=[],
help="Changed files (relative to source-dir) to publish; the push diff",
)
parser.add_argument("--all", action="store_true", help="Publish every publishable file")
args = parser.parse_args()
source_dir = args.source_dir.resolve()
source_ref = os.environ.get("SOURCE_REF", "")
gitea_repo_url = os.environ.get("GITEA_REPO_URL", "").rstrip("/")
# The repository is always scanned in full, whatever gets published: link
# resolution and title-collision checks must see every publishable file, or
# the same link resolves differently depending on which files a push touched.
posts = discover_publishable(source_dir)
if not posts:
print("No publishable files found.")
return
title_index = build_title_index(posts)
print(f"Scanned repository: {len(posts)} publishable file(s).")
if args.all:
candidates = list(source_dir.rglob("*.md"))
selected = posts
elif args.files:
candidates = [source_dir / f for f in args.files]
seeds = {(source_dir / f).resolve() for f in args.files} & posts.keys()
if not seeds:
print("No changed file is publishable. Nothing to publish.")
return
selected = select_publish_set(posts, seeds)
else:
print("Nothing to do: pass --files or --all.")
return
files = [
p for p in candidates
if p.exists()
and p.suffix == ".md"
and "wip" not in p.relative_to(source_dir).parts
and p.name != "README.md"
]
if not files:
print("No publishable candidates after filtering.")
return
print(f"Validating {len(files)} file(s)...")
posts = load_and_validate(files, source_dir)
sibling_map = build_sibling_map(posts)
print("Validation passed.")
# Render every page before writing any of them, so a bad link fails the run
# with the wiki untouched rather than half-updated.
rendered = {
abs_path: render_page(abs_path, source_dir, post, source_ref, gitea_repo_url, title_index)
for abs_path, post in selected.items()
}
print(f"Rendered {len(rendered)} page(s). Publishing.")
site = connect_wiki()
counts = {"published": 0, "noop": 0, "skipped-manual": 0}
for abs_path, post in posts.items():
for abs_path, post in selected.items():
rel = abs_path.relative_to(source_dir)
outcome = publish_one(abs_path, source_dir, post, site, source_ref, gitea_repo_url, sibling_map)
outcome = publish_one(abs_path, post, rendered[abs_path], site, source_ref)
counts[outcome] += 1
if outcome == "skipped-manual":
print(f" [skipped-manual] {rel} ← page exists without auto-gen banner; delete or add banner to hand over to bot", file=sys.stderr)