novoyuuparosk-wiki/novoyuuparosk-auto-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

songs: replace the wiki.siblings macro with plain Markdown links

Browse Source 
The `wiki.siblings` bill of materials is gone. A related co-publication is
now nominated with an ordinary relative Markdown link — `[label](notes.md)` —
resolved at publish time against the target file's own `title` frontmatter.
No macro namespace, no per-file declaration, and the link still works in
Gitea's blob view and any editor preview.

The macro could not be made safe. Substitution was `str.replace(tag, …)` over
the whole rendered page: no word boundary, so a tag also occurring in lyrics or
a code block got rewritten; no collision check, so overlapping tags corrupted
each other in declaration order. SCHEMA.md documented the placeholder as
`[[TAG]]` while the code replaced the bare tag, so following the schema
produced `[[[[Title|label]]]]`.

Fixes a live corruption bug in the process. `sibling_map` was built only from
the files in the publish batch, and an unresolved sibling degraded to plain
text — so pushing `ses/pulse.md` alone republished it with its wikilink
silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the
workflow "always runs --all", which it never did on an ordinary push.

The repository is now always scanned in full, whatever a run publishes, so
link resolution and title-collision detection can never depend on which files
a push touched. Writes stay bounded to the push diff plus one hop in each link
direction: the page a changed page links to (so a link cannot point at an
unpublished page) and the pages linking to it (so an edited `title` does not
leave stale inbound links). Publishing is not instant; reading is cheap.

An unresolvable `.md` link is now a hard error, never graceful degradation, and
linking to a file still does not publish it — `wiki.publish: true` remains the
only gate. A leftover-link guard fails the run on any link shape the resolver
did not recognise, including a Markdown link written inside a verbatim
```columns fence, so a pandoc upgrade cannot silently publish literal text.

Also:

- render before write: every page in a run is rendered before the first is
  published, so a broken page leaves the wiki untouched instead of
  half-updated. `publish_one` is split into `render_page` (pure) and the write.
- pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale
  encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside
  subprocess's reader thread, `stdout` came back None, and the real error was
  lost behind a TypeError. Latent on the Debian runner via PEP 538 only.
- title collisions are detected repo-wide, not just within a push, and before
  any wiki contact.
- tests/, run by .gitea/workflows/test.yml in the publish runner image. The
  pandoc output contract the resolver depends on is now pinned by a test
  instead of by prose — which is how SCHEMA.md came to assert something false.
- drop an f-string with nothing to interpolate.
- quote the SOURCE_REF interpolation in the workflow instead of pasting an
  input straight into shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Wan Ziyu
2026-09-09 16:16:33 +09:00
parent 4e7d84d665
commit f5625cde98
10 changed files with 838 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipelines/songs/SCHEMA.md
+84 -69
View File
@@ -2,7 +2,7 @@
Source-file contract for the songs pipeline. Files in `novoyuuparosk-wiki/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar, except that `[[TAG]]` placeholders declared via `wiki.siblings` are substituted with resolved wikilinks after rendering.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar. There is no macro syntax and nothing to declare: a link to another song page is an ordinary relative Markdown link, resolved to a wikilink at publish time.
## File layout
@@ -12,20 +12,16 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
# pulse under latent semantic envelopes
... body markdown, PULSE_DEV used inline ...
... body markdown; a related page is nominated as [development notes](pulse_dev.md) ...
```
The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc, then applies sibling substitutions on the wikitext output.
The renderer parses the frontmatter, validates required fields, hands the body (with the first `# Heading` line removed) to Pandoc, then resolves `.md` links in the wikitext output to the wiki page titles of their targets.
## Fields
@@ -81,54 +77,54 @@ Example: in `ses/pulse.md`, `lrc: pulse.lrc` declares that `ses/pulse.lrc` is th
If the LRC file is missing or not valid UTF-8, the upload is skipped with a warning — the page still publishes.
#### `wiki.siblings` (list of sibling declarations)
Bill of materials for inter-page wikilinks. Each entry declares a sibling `.md` file and a tag that can be used as a `[[TAG]]` placeholder anywhere in the body.
```yaml
wiki:
siblings:
- path: pulse_dev.md # relative to this file's directory
tag: PULSE_DEV # placeholder used in body as [[PULSE_DEV]]
label: development notes # optional display text
```
**Fields per entry:**
| Field | Required for resolution | Purpose |
|---|---|---|
| `path` | yes | Relative path to the sibling `.md` file |
| `tag` | yes | Placeholder identifier; pipeline does a case-sensitive string replace of `TAG` in the rendered wikitext |
| `label` | no | Display text for the resolved link |
An entry missing either `path` or `tag` is silently skipped — no substitution is attempted for that entry.
**Resolution:**
The pipeline resolves each declared sibling's `path` to its wiki page title by looking it up in the set of all publishable files processed in the same run. Substitution is a case-sensitive string replace of the bare tag. Substitution rules:
| Condition | `TAG` becomes |
|---|---|
| `path` resolves to a published page, `label` present | `[[wiki page title\|label]]` |
| `path` resolves to a published page, no `label` | `[[wiki page title]]` |
| `path` not in published set (file exists but not published), `label` present | `label` (plain text) |
| `path` not in published set, no `label` | `TAG` (plain text) |
Declared siblings whose file doesn't exist on disk are a **validation error**. Siblings that exist but aren't published (no `wiki.publish: true`) degrade gracefully per the table above.
The siblings list is a bill of materials — declaring a sibling has no effect unless its `[[TAG]]` appears in the body. Unused entries are inert.
**`--files` mode**: sibling resolution is limited to the files in that batch. Siblings pointing to files outside the batch degrade to plain text. This is acceptable since the pipeline always runs `--all` in the workflow.
## Body
Plain markdown. The renderer applies these transformations before and after Pandoc:
1. **Pre-Pandoc**: strip the first-line `# Heading` if present.
2. **Post-Pandoc**: expand `columns` fenced blocks into side-by-side wikitext (see below).
3. **Post-Pandoc**: substitute `TAG` placeholders declared in `wiki.siblings`.
3. **Post-Pandoc**: resolve links to sibling `.md` files into wikilinks (see below).
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, sibling resolution) around Pandoc's output.
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, link resolution) around Pandoc's output.
### Nominating a related co-publication
Link to it the ordinary way, with a relative Markdown link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
There is nothing to declare in the frontmatter. The link stays a working link in Gitea's blob view, in an editor preview, and anywhere else the raw file is read — and on the wiki it becomes a wikilink to the target's page:
```wikitext
For more behind-the-scene shenanigans, see [[pulse under latent semantic envelopes - development notes|dev gibberish]].
```
**How the target becomes a page name.** The path is resolved relative to the linking file's directory, and the target file's own `title` frontmatter field is used verbatim as the wikilink target — the same string the bot writes that page to. There is no second name format to keep in sync: rename a page by editing its `title`, and every link aimed at it follows on the next publish. MediaWiki handles the first-letter capitalisation and the space-to-underscore in URLs itself.
| Written in the body | Rendered wikitext |
|---|---|
| `[dev gibberish](pulse_dev.md)` | `[[<target's title>\|dev gibberish]]` |
| `[pulse_dev.md](pulse_dev.md)` (label same as path) | `[[<target's title>]]` |
| `[notes](../ses/pulse_dev.md)` | `[[<target's title>\|notes]]` |
| `[notes](pulse_dev.md#Section)` | `[[<target's title>#Section\|notes]]` |
| `[site](https://example.com)` | `[https://example.com site]` (untouched) |
**A `.md` link that cannot be resolved fails the run.** Both cases abort with a non-zero exit before anything is published:
| Condition | Result |
|---|---|
| Target file does not exist | **error** — `file does not exist` |
| Target exists but is not published (no `wiki.publish: true`, or under `wip/`) | **error** — `file exists but is not published` |
Linking to a file does **not** publish it. Publication is always the target's own explicit `wiki.publish: true`; nothing is published as a side effect of being linked. If you want the link, opt the target in.
**Scope of a run.** A push publishes the files it changed, plus — one hop in each direction — any page a changed page links to, and any page that links to a changed page. The second is what keeps inbound links correct when a page's `title` moves. Regardless of how few pages a run publishes, the whole repository is always scanned first, so link resolution and title-collision checks see every publishable file.
**Anchors** (`pulse_dev.md#Section`) are passed through verbatim. MediaWiki derives its own section anchors from heading text, so an anchor that doesn't match a heading on the target page lands the reader at the top of it rather than at the section. Nothing validates this.
**Links inside a `columns` fence do not work.** The fence is verbatim by design (see below); Pandoc does not process Markdown in it, so a link written there would reach the wiki as literal text. The renderer detects this and fails the run rather than publishing it. Put the link in ordinary body prose.
### Columns (side-by-side) shorthand
@@ -157,29 +153,43 @@ The renderer turns this into a flexbox row of `<poem>` columns (one per `===`-de
## Renderer behaviour
For each `.md` with `wiki.publish: true` and not under `wip/`:
A run happens in three phases, in this order, and the wiki is not contacted until the third. Anything wrong in phase 1 or 2 aborts with a non-zero exit and the wiki untouched — never half-updated.
1. Parse and validate frontmatter.
2. Strip the leading `# Heading` from the body if present.
3. Pipe the body through `pandoc -f markdown -t mediawiki`, then expand any `columns` fenced blocks into side-by-side `<poem>` columns.
4. Substitute `TAG` placeholders with resolved `[[Page|Display]]` wikilinks (or fallback text).
5. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
6. If `lrc` is declared: upload the LRC file if its SHA1 has changed; append a `[[Media:…]]` link.
7. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
9. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
10. Otherwise, write the page with an edit summary referencing the source commit.
**Phase 1 — scan and validate the whole repository.** For every `.md` outside `wip/` (and excluding `README.md`):
1. Parse the frontmatter; skip the file silently unless `wiki.publish: true`.
2. Validate the fields (see *Validation errors* below).
3. Register `title` and fail on a collision with any other publishable file, anywhere in the tree.
4. Build the path → page-title index every `.md` link resolves through.
**Phase 2 — render every page that this run will publish.** For each selected file:
1. Strip the leading `# Heading` from the body if present.
2. Pipe the body through `pandoc -f markdown -t mediawiki`, then expand any `columns` fenced blocks into side-by-side `<poem>` columns.
3. Resolve `.md` links to `[[Page title|label]]` wikilinks; fail on any that cannot be resolved, and on any link shape that survived rendering unresolved.
4. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
5. If `lrc` is declared, append a `[[Media:…]]` link.
6. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
**Phase 3 — write.** For each rendered page:
1. If `lrc` is declared: upload the LRC file if its SHA1 has changed.
2. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
3. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
4. Otherwise, write the page with an edit summary referencing the source commit.
## Implemented scope
- Frontmatter parsing and validation
- Repo-wide page-title collision detection, before any wiki write
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Banner template injection
- LRC file upload and footer link
- Sibling placeholder resolution (`[[TAG]]` → `[[wiki page title|label]]`)
- Relative `.md` link resolution (`[label](notes.md)` → `[[wiki page title|label]]`), repo-wide and batch-independent
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]`) when `album` is set
- Additional categories via `categories` list
- Render-before-write: a broken page fails the run instead of half-updating the wiki
- Idempotent writes (no-op skip when content matches)
- Manual-page protection (skip pages without auto-gen banner)
- Files without `wiki.publish: true` silently skipped
@@ -199,16 +209,18 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
```
### A companion/sibling document (no album)
Its body nominates the companion page with a plain relative link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
### A companion document (no album)
```yaml
---
@@ -249,8 +261,11 @@ Files without a `wiki.publish: true` frontmatter field are silently skipped —
- `wiki.publish` present but not a boolean
- `release_date` (if present) does not parse as YYYY-MM-DD
- `categories` (if present) is not a list
- `wiki.siblings` entry that has both `path` and `tag` declared, but the `path` file does not exist on disk
- `wiki.siblings` entry that is not a mapping
- Two publishable source files declare the same `title` (page-name collision)
- Two publishable source files declare the same `title` (page-name collision) — checked across the whole repository, not just the files a push touched
- A `.md` link whose target file does not exist
- A `.md` link whose target exists but is not published
- A `.md` link that survives rendering unresolved — a link shape the renderer does not recognise, or a Markdown link written inside a `columns` fence
Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.
Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken.
Frontmatter keys the schema does not define are ignored, so a leftover `wiki.siblings` block from the retired macro system is inert rather than an error. Delete them; they do nothing.