novoyuuparosk-wiki/novoyuuparosk-auto-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

songs: replace the wiki.siblings macro with plain Markdown links

Browse Source 
The `wiki.siblings` bill of materials is gone. A related co-publication is
now nominated with an ordinary relative Markdown link — `[label](notes.md)` —
resolved at publish time against the target file's own `title` frontmatter.
No macro namespace, no per-file declaration, and the link still works in
Gitea's blob view and any editor preview.

The macro could not be made safe. Substitution was `str.replace(tag, …)` over
the whole rendered page: no word boundary, so a tag also occurring in lyrics or
a code block got rewritten; no collision check, so overlapping tags corrupted
each other in declaration order. SCHEMA.md documented the placeholder as
`[[TAG]]` while the code replaced the bare tag, so following the schema
produced `[[[[Title|label]]]]`.

Fixes a live corruption bug in the process. `sibling_map` was built only from
the files in the publish batch, and an unresolved sibling degraded to plain
text — so pushing `ses/pulse.md` alone republished it with its wikilink
silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the
workflow "always runs --all", which it never did on an ordinary push.

The repository is now always scanned in full, whatever a run publishes, so
link resolution and title-collision detection can never depend on which files
a push touched. Writes stay bounded to the push diff plus one hop in each link
direction: the page a changed page links to (so a link cannot point at an
unpublished page) and the pages linking to it (so an edited `title` does not
leave stale inbound links). Publishing is not instant; reading is cheap.

An unresolvable `.md` link is now a hard error, never graceful degradation, and
linking to a file still does not publish it — `wiki.publish: true` remains the
only gate. A leftover-link guard fails the run on any link shape the resolver
did not recognise, including a Markdown link written inside a verbatim
```columns fence, so a pandoc upgrade cannot silently publish literal text.

Also:

- render before write: every page in a run is rendered before the first is
  published, so a broken page leaves the wiki untouched instead of
  half-updated. `publish_one` is split into `render_page` (pure) and the write.
- pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale
  encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside
  subprocess's reader thread, `stdout` came back None, and the real error was
  lost behind a TypeError. Latent on the Debian runner via PEP 538 only.
- title collisions are detected repo-wide, not just within a push, and before
  any wiki contact.
- tests/, run by .gitea/workflows/test.yml in the publish runner image. The
  pandoc output contract the resolver depends on is now pinned by a test
  instead of by prose — which is how SCHEMA.md came to assert something false.
- drop an f-string with nothing to interpolate.
- quote the SOURCE_REF interpolation in the workflow instead of pasting an
  input straight into shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Wan Ziyu
2026-09-09 16:16:33 +09:00
parent 4e7d84d665
commit f5625cde98
10 changed files with 838 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/wiki.py
+123 -1
View File
@@ -4,6 +4,8 @@ import html
import os
import re
import subprocess
import urllib.parse
from pathlib import Path
from urllib.parse import urlparse
import mwclient
@@ -22,6 +24,38 @@ _BOLD_ITALIC_RE = re.compile(r"\*\*\*(.+?)\*\*\*")
_BOLD_RE = re.compile(r"\*\*(.+?)\*\*")
_ITALIC_RE = re.compile(r"\*(.+?)\*")
# Pandoc's mediawiki writer renders a *relative* Markdown link target as a
# wikilink carrying the raw path — [label](notes.md) becomes [[notes.md|label]] —
# and an absolute URL as [url label]. Both shapes are matched here so that a
# pandoc version which classifies relative targets the other way still resolves.
# The alternation is scanned left to right, so a [[...]] is consumed whole and
# the single-bracket branch never re-matches its interior.
_MD_LINK_RE = re.compile(
r"\[\[(?P<wiki_target>[^\[\]|]+?)(?:\|(?P<wiki_label>[^\[\]]*))?\]\]"
r"|\[(?P<ext_target>[^\[\]\s|]+)(?:[ \t]+(?P<ext_label>[^\[\]]*))?\]"
)
# Inline Markdown link target, read from the *source* body before Pandoc runs:
# the "…](target)" tail of [label](target). Used for the link graph only.
_MD_SOURCE_LINK_RE = re.compile(r"\]\(\s*<?([^()<>\s]+)>?")
# Anything still naming a .md file in a link position after resolution means the
# renderer failed to understand it. That aborts the run; it never reaches the wiki.
_UNRESOLVED_LINK_RES = (
(
re.compile(r"\[\[[^\[\]]*\.md(?:#[^\[\]]*)?(?:\|[^\[\]]*)?\]\]", re.IGNORECASE),
"unresolved wikilink",
),
(
re.compile(r"\[[^\[\]\s|]*\.md[^\[\]]*\]", re.IGNORECASE),
"unresolved external-style link",
),
(
re.compile(r"\]\([^()\s]*\.md[^()]*\)", re.IGNORECASE),
"raw Markdown link (Markdown is not processed inside a ```columns fence)",
),
)
def _md_emphasis_to_wikitext(text: str) -> str:
text = _BOLD_ITALIC_RE.sub(r"'''''\1'''''", text)
@@ -65,12 +99,100 @@ def expand_columns(wikitext: str) -> str:
return _COLUMNS_BLOCK_RE.sub(render, wikitext)
def extract_md_link_targets(markdown_body: str) -> list[str]:
"""Raw targets of inline Markdown links in an unrendered body.
Used to build the link graph *before* Pandoc runs, so the publisher can work
out which pages a changed page points at (and which point back) without
rendering the whole repository. Targets are returned verbatim, including any
``#fragment`` and percent-encoding; the caller resolves them.
"""
return _MD_SOURCE_LINK_RE.findall(markdown_body)
def assert_no_unresolved_md_links(wikitext: str, source_path: Path) -> None:
"""Abort if any link still names a .md file after resolution.
Guards against a link shape the resolver does not recognise: a pandoc version
that writes relative targets differently, or a Markdown link buried in a
verbatim ```columns fence. Without this, such a link would reach the wiki as
literal text, silently corrupting a published page.
"""
for pattern, kind in _UNRESOLVED_LINK_RES:
found = pattern.search(wikitext)
if found:
raise ValueError(
f"{source_path}: {kind} still references a .md file after "
f"resolution: {found.group(0)!r}"
)
def resolve_md_links(wikitext: str, source_path: Path, title_index: dict[str, str]) -> str:
"""Rewrite links to sibling ``.md`` files into wikilinks to their wiki pages.
A related co-publication is nominated with an ordinary relative Markdown
link — ``see [dev notes](pulse_dev.md)`` — which stays a working link in
Gitea's blob view and in any Markdown preview. Here the ``.md`` target is
resolved against ``title_index`` (absolute path string → wiki page title) and
replaced with ``[[Wiki page title|label]]``.
``title_index`` must cover every publishable file in the repository, not only
the batch being published, or the same link resolves differently depending on
which files a push happened to touch.
A ``.md`` target that cannot be resolved raises ``ValueError``: either the
file does not exist, or it exists but is not published. Both are authoring
mistakes, and both fail the run rather than silently degrade the page. Link
targets that do not name a ``.md`` file are left exactly as Pandoc wrote them.
"""
def resolve(match: re.Match) -> str:
groups = match.groupdict()
if groups["wiki_target"] is not None:
target, label = groups["wiki_target"], groups["wiki_label"]
else:
target, label = groups["ext_target"], groups["ext_label"]
raw_path, _, fragment = target.partition("#")
rel_path = urllib.parse.unquote(raw_path)
if not rel_path.lower().endswith(".md"):
return match.group(0)
resolved = (source_path.parent / rel_path).resolve()
title = title_index.get(str(resolved))
if title is None:
reason = (
"file exists but is not published (no 'wiki.publish: true', "
"or it lives under wip/)"
if resolved.exists()
else "file does not exist"
)
raise ValueError(
f"{source_path}: link to '{target}' cannot be resolved - {reason}"
)
anchor = f"#{fragment}" if fragment else ""
# A label that repeats the path (pandoc's form for an unlabelled link) or
# repeats the page title needs no pipe: [[Title]] already displays it.
if label is None or label.strip() in (target, title):
return f"[[{title}{anchor}]]"
return f"[[{title}{anchor}|{label}]]"
resolved = _MD_LINK_RE.sub(resolve, wikitext)
assert_no_unresolved_md_links(resolved, source_path)
return resolved
def markdown_to_wikitext(body: str) -> str:
# encoding is pinned rather than left to text=True, which would decode
# pandoc's output with the ambient locale encoding. The lyrics are Japanese
# and Chinese; on any non-UTF-8 locale that decode dies inside subprocess's
# reader thread, stdout comes back None, and the real error is lost.
result = subprocess.run(
["pandoc", "-f", "markdown", "-t", "mediawiki"],
input=body,
capture_output=True,
text=True,
encoding="utf-8",
)
if result.returncode != 0:
raise RuntimeError(f"pandoc failed: {result.stderr.strip()}")