novoyuuparosk-wiki/novoyuuparosk-auto-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

songs: replace the wiki.siblings macro with plain Markdown links

Browse Source 
The `wiki.siblings` bill of materials is gone. A related co-publication is
now nominated with an ordinary relative Markdown link — `[label](notes.md)` —
resolved at publish time against the target file's own `title` frontmatter.
No macro namespace, no per-file declaration, and the link still works in
Gitea's blob view and any editor preview.

The macro could not be made safe. Substitution was `str.replace(tag, …)` over
the whole rendered page: no word boundary, so a tag also occurring in lyrics or
a code block got rewritten; no collision check, so overlapping tags corrupted
each other in declaration order. SCHEMA.md documented the placeholder as
`[[TAG]]` while the code replaced the bare tag, so following the schema
produced `[[[[Title|label]]]]`.

Fixes a live corruption bug in the process. `sibling_map` was built only from
the files in the publish batch, and an unresolved sibling degraded to plain
text — so pushing `ses/pulse.md` alone republished it with its wikilink
silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the
workflow "always runs --all", which it never did on an ordinary push.

The repository is now always scanned in full, whatever a run publishes, so
link resolution and title-collision detection can never depend on which files
a push touched. Writes stay bounded to the push diff plus one hop in each link
direction: the page a changed page links to (so a link cannot point at an
unpublished page) and the pages linking to it (so an edited `title` does not
leave stale inbound links). Publishing is not instant; reading is cheap.

An unresolvable `.md` link is now a hard error, never graceful degradation, and
linking to a file still does not publish it — `wiki.publish: true` remains the
only gate. A leftover-link guard fails the run on any link shape the resolver
did not recognise, including a Markdown link written inside a verbatim
```columns fence, so a pandoc upgrade cannot silently publish literal text.

Also:

- render before write: every page in a run is rendered before the first is
  published, so a broken page leaves the wiki untouched instead of
  half-updated. `publish_one` is split into `render_page` (pure) and the write.
- pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale
  encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside
  subprocess's reader thread, `stdout` came back None, and the real error was
  lost behind a TypeError. Latent on the Debian runner via PEP 538 only.
- title collisions are detected repo-wide, not just within a push, and before
  any wiki contact.
- tests/, run by .gitea/workflows/test.yml in the publish runner image. The
  pandoc output contract the resolver depends on is now pinned by a test
  instead of by prose — which is how SCHEMA.md came to assert something false.
- drop an f-string with nothing to interpolate.
- quote the SOURCE_REF interpolation in the workflow instead of pasting an
  input straight into shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Wan Ziyu
2026-09-09 16:16:33 +09:00
parent 4e7d84d665
commit f5625cde98
10 changed files with 838 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipelines/songs/publish.py
+130 -75
View File
@@ -6,6 +6,7 @@ import hashlib
import io
import os
import sys
import urllib.parse
from datetime import datetime
from pathlib import Path
@@ -13,7 +14,14 @@ import frontmatter
import mwclient
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
from lib.wiki import AUTO_BANNER_PREFIX, connect_wiki, markdown_to_wikitext, strip_first_h1
from lib.wiki import (
AUTO_BANNER_PREFIX,
connect_wiki,
extract_md_link_targets,
markdown_to_wikitext,
resolve_md_links,
strip_first_h1,
)
def validate(fm: dict, path: Path) -> None:
@@ -35,47 +43,58 @@ def validate(fm: dict, path: Path) -> None:
if "categories" in fm and not isinstance(fm["categories"], list):
errors.append("'categories' must be a list")
for i, sib in enumerate(fm.get("wiki", {}).get("siblings") or []):
if not isinstance(sib, dict):
errors.append(f"wiki.siblings[{i}] must be a mapping")
continue
sib_path = sib.get("path")
sib_tag = sib.get("tag")
if sib_path and sib_tag:
if not (path.parent / str(sib_path)).resolve().exists():
errors.append(f"wiki.siblings[{i}]: path '{sib_path}' not found")
if errors:
raise ValueError(f"{path}: " + "; ".join(errors))
def build_sibling_map(posts: dict) -> dict[str, str]:
def build_title_index(posts: dict) -> dict[str, str]:
"""Map abs_path_str → wiki_title for all published posts."""
return {str(path): str(post.metadata["title"]) for path, post in posts.items()}
def apply_sibling_substitutions(wikitext: str, this_path: Path, fm_wiki: dict, sibling_map: dict[str, str]) -> str:
"""Replace [[TAG]] placeholders using the declared sibling bill of materials."""
for sib in (fm_wiki.get("siblings") or []):
if not isinstance(sib, dict):
continue
tag = sib.get("tag")
path = sib.get("path")
if not tag or not path:
continue
label = (sib.get("label") or "").strip()
abs_sib = str((this_path.parent / str(path)).resolve())
title = sibling_map.get(abs_sib)
if title and label:
replacement = f"[[{title}|{label}]]"
elif title:
replacement = f"[[{title}]]"
elif label:
replacement = label
else:
replacement = tag
wikitext = wikitext.replace(tag, replacement)
return wikitext
def build_link_graph(posts: dict) -> dict[Path, set[Path]]:
"""Map each publishable file to the publishable .md files it links to.
Read from the raw Markdown, so it costs no Pandoc runs. Targets outside the
publishable set are dropped here; `resolve_md_links` is what refuses them
loudly at render time.
"""
graph: dict[Path, set[Path]] = {}
for path, post in posts.items():
targets = set()
for raw in extract_md_link_targets(post.content):
rel_path = urllib.parse.unquote(raw.partition("#")[0])
if not rel_path.lower().endswith(".md"):
continue
resolved = (path.parent / rel_path).resolve()
if resolved in posts:
targets.add(resolved)
graph[path] = targets
return graph
def select_publish_set(posts: dict, seeds: set[Path]) -> dict[Path, object]:
"""The changed files, plus the pages their links touch in either direction.
Publishing is not instant, so only what a push actually affects is written.
Two extra hops are unavoidable, both one level deep:
- **outbound**: a page the changed page links to, so the wikilink cannot
point at a page that was never published (a red link);
- **inbound**: a page linking *to* a changed page, because if the changed
page's `title` moved, every link aimed at it is now stale.
Ordering follows `posts` so a run's output is stable regardless of which
files the diff reported.
"""
graph = build_link_graph(posts)
selected = set(seeds)
for path, targets in graph.items():
if path in seeds:
selected |= targets
elif targets & seeds:
selected.add(path)
return {path: post for path, post in posts.items() if path in selected}
def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: str, lrc_filename: str | None = None) -> str:
@@ -84,7 +103,7 @@ def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: st
cat_parts = []
if fm.get("album"):
cat_parts.append(f"[[Category:Songs]]")
cat_parts.append("[[Category:Songs]]")
cat_parts.append(f"[[Category:Album:{fm['album']}]]")
for cat in (fm.get("categories") or []):
cat_parts.append(f"[[Category:{cat}]]")
@@ -121,25 +140,33 @@ def upload_lrc(lrc_path: Path, title: str, site: mwclient.Site) -> bool:
return True
def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, source_ref: str, gitea_repo_url: str, sibling_map: dict[str, str]) -> str:
"""Returns 'published', 'noop', or 'skipped-manual'."""
def render_page(abs_path: Path, source_dir: Path, post, source_ref: str, gitea_repo_url: str, title_index: dict[str, str]) -> str:
"""Render one file to finished wikitext. Pure: touches no wiki, writes nothing.
Kept separate from the write so that every page in a run is rendered before
the first one is published. A bad link or a broken column fence then fails
the run with the wiki untouched, instead of half-updating it.
"""
rel = abs_path.relative_to(source_dir)
fm = post.metadata
title = str(fm["title"])
body_wikitext = markdown_to_wikitext(strip_first_h1(post.content))
body_wikitext = apply_sibling_substitutions(body_wikitext, abs_path, fm.get("wiki", {}), sibling_map)
body_wikitext = resolve_md_links(body_wikitext, abs_path, title_index)
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
lrc_filename = None
lrc_rel = fm.get("lrc")
if lrc_rel:
lrc_uploaded = upload_lrc(abs_path.parent / str(lrc_rel), title, site)
lrc_filename = f"{title}.lrc"
if lrc_uploaded:
print(f" [lrc-uploaded] {lrc_filename}")
lrc_filename = f"{title}.lrc" if fm.get("lrc") else None
return build_wikitext(fm, body_wikitext, source_url, source_ref, lrc_filename)
page_content = build_wikitext(fm, body_wikitext, source_url, source_ref, lrc_filename)
def publish_one(abs_path: Path, post, page_content: str, site: mwclient.Site, source_ref: str) -> str:
"""Write one pre-rendered page. Returns 'published', 'noop', or 'skipped-manual'."""
fm = post.metadata
title = str(fm["title"])
lrc_rel = fm.get("lrc")
if lrc_rel and upload_lrc(abs_path.parent / str(lrc_rel), title, site):
print(f" [lrc-uploaded] {title}.lrc")
page = site.pages[title]
existing = page.text()
@@ -155,13 +182,29 @@ def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, sou
return "published"
def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]:
posts = {}
def discover_publishable(source_dir: Path) -> dict[Path, object]:
"""Every publishable .md in the repository, validated, keyed by absolute path.
The whole tree is scanned on every run, never a subset of it, however few
pages the run then publishes. Two things depend on seeing all of it:
- **link resolution**: a link to a sibling .md resolves through this set, so
a partial scan would resolve the same link differently depending on which
files a push happened to touch, and silently drop it when the target fell
outside the subset;
- **title collisions**: two files claiming one wiki page name must fail the
run wherever they sit in the tree, not only when both land in one push.
Files under wip/ and README.md are never published. Everything else needs an
explicit `wiki.publish: true`; a file is never published as a side effect of
something else linking to it.
"""
posts: dict[Path, object] = {}
titles: dict[str, Path] = {}
for path in files:
if not path.exists():
print(f" warn: {path} not found, skipping", file=sys.stderr)
for path in sorted(source_dir.rglob("*.md")):
rel = path.relative_to(source_dir)
if "wip" in rel.parts or path.name == "README.md":
continue
post = frontmatter.load(str(path))
@@ -175,10 +218,11 @@ def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]
title = str(post.metadata["title"])
if title in titles:
raise ValueError(
f"Duplicate wiki title '{title}': {path} and {titles[title]}"
f"Duplicate wiki title '{title}': {rel} and "
f"{titles[title].relative_to(source_dir)}"
)
titles[title] = path
posts[path] = post
posts[path.resolve()] = post
return posts
@@ -186,45 +230,56 @@ def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]
def main() -> None:
parser = argparse.ArgumentParser(description="Publish song .md files to MediaWiki.")
parser.add_argument("--source-dir", required=True, type=Path)
parser.add_argument("--files", nargs="*", default=[], help="Relative paths within source-dir")
parser.add_argument("--all", action="store_true", help="Publish all publishable files")
parser.add_argument(
"--files",
nargs="*",
default=[],
help="Changed files (relative to source-dir) to publish; the push diff",
)
parser.add_argument("--all", action="store_true", help="Publish every publishable file")
args = parser.parse_args()
source_dir = args.source_dir.resolve()
source_ref = os.environ.get("SOURCE_REF", "")
gitea_repo_url = os.environ.get("GITEA_REPO_URL", "").rstrip("/")
# The repository is always scanned in full, whatever gets published: link
# resolution and title-collision checks must see every publishable file, or
# the same link resolves differently depending on which files a push touched.
posts = discover_publishable(source_dir)
if not posts:
print("No publishable files found.")
return
title_index = build_title_index(posts)
print(f"Scanned repository: {len(posts)} publishable file(s).")
if args.all:
candidates = list(source_dir.rglob("*.md"))
selected = posts
elif args.files:
candidates = [source_dir / f for f in args.files]
seeds = {(source_dir / f).resolve() for f in args.files} & posts.keys()
if not seeds:
print("No changed file is publishable. Nothing to publish.")
return
selected = select_publish_set(posts, seeds)
else:
print("Nothing to do: pass --files or --all.")
return
files = [
p for p in candidates
if p.exists()
and p.suffix == ".md"
and "wip" not in p.relative_to(source_dir).parts
and p.name != "README.md"
]
if not files:
print("No publishable candidates after filtering.")
return
print(f"Validating {len(files)} file(s)...")
posts = load_and_validate(files, source_dir)
sibling_map = build_sibling_map(posts)
print("Validation passed.")
# Render every page before writing any of them, so a bad link fails the run
# with the wiki untouched rather than half-updated.
rendered = {
abs_path: render_page(abs_path, source_dir, post, source_ref, gitea_repo_url, title_index)
for abs_path, post in selected.items()
}
print(f"Rendered {len(rendered)} page(s). Publishing.")
site = connect_wiki()
counts = {"published": 0, "noop": 0, "skipped-manual": 0}
for abs_path, post in posts.items():
for abs_path, post in selected.items():
rel = abs_path.relative_to(source_dir)
outcome = publish_one(abs_path, source_dir, post, site, source_ref, gitea_repo_url, sibling_map)
outcome = publish_one(abs_path, post, rendered[abs_path], site, source_ref)
counts[outcome] += 1
if outcome == "skipped-manual":
print(f" [skipped-manual] {rel} ← page exists without auto-gen banner; delete or add banner to hand over to bot", file=sys.stderr)