novoyuuparosk-auto-wiki/pipelines/songs/SCHEMA.md

# SCHEMA — song source files

Source-file contract for the songs pipeline. Files in `novoyuuparosk-wiki/ncmr-songs` must follow this contract to be picked up by the auto-publisher.

The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar. There is no macro syntax and nothing to declare: a link to another song page is an ordinary relative Markdown link, resolved to a wikilink at publish time.

## File layout

```markdown
---
title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
  publish: true
release_date: 2026-06-18
lrc: pulse.lrc
---

# pulse under latent semantic envelopes

... body markdown; a related page is nominated as [development notes](pulse_dev.md) ...
```

The renderer parses the frontmatter, validates required fields, hands the body (with the first `# Heading` line removed) to Pandoc, then resolves `.md` links in the wikitext output to the wiki page titles of their targets.

## Fields

### Required

#### `title` (string)

The MediaWiki page name that the bot writes or updates. MediaWiki capitalises the first letter automatically; lowercase input is fine.

Example: `title: pulse under latent semantic envelopes` → wiki page `Pulse under latent semantic envelopes` at URL `/wiki/Pulse_under_latent_semantic_envelopes`.

The body should contain a matching `# Title` first-line heading for human readability when viewing the source file directly (in an editor, on Gitea's blob view, etc.). The renderer strips this first-line heading during conversion — the wiki page already supplies its own title.

#### `wiki.publish` (boolean)

Publishing gate. `true` means the pipeline writes this file to the wiki. `false` means the file is ignored.

The `wiki` key is a mapping rather than a flat field so future toggles (`wiki.protected`, `wiki.summary_template`, etc.) can be added without restructuring.

Files under `wip/` are also skipped, regardless of `wiki.publish`.

### Optional

#### `album` (string)

The full album name. Drives `[[Category:Songs]]` and `[[Category:Album:<album name>]]` injection at the bottom of the rendered wiki page.

Example: `album: Shirakaba Express Service` → category tags `[[Category:Songs]]` and `[[Category:Album:Shirakaba Express Service]]`.

Pages without an album (companion documents, placeholders, songs without an assigned album) simply omit this field — neither category is injected unless `categories` provides one.

#### `categories` (list of strings)

Additional wiki categories to inject, beyond the album category. Each entry becomes `[[Category:<value>]]`.

```yaml
categories:
  - Song development notes
  - Work in progress
```

`album` and `categories` are additive — a page can have both.

#### `release_date` (ISO 8601 date)

The song's release date in `YYYY-MM-DD` format. Skipped if absent. Currently informational; may drive sort order on future auto-generated indexes.

#### `lrc` (string, relative path)

Pointer to a companion LRC file, relative to the `.md` file's location. The pipeline uploads the file to the wiki as `<title>.lrc` (if the content has changed) and appends a `[[Media:<title>.lrc|Synced lyrics (.lrc)]]` link to the page footer.

Example: in `ses/pulse.md`, `lrc: pulse.lrc` declares that `ses/pulse.lrc` is the companion synced-lyrics file.

If the LRC file is missing or not valid UTF-8, the upload is skipped with a warning — the page still publishes.

## Body

Plain markdown. The renderer applies these transformations before and after Pandoc:

1. **Pre-Pandoc**: strip the first-line `# Heading` if present.
2. **Post-Pandoc**: expand `columns` fenced blocks into side-by-side wikitext (see below).
3. **Post-Pandoc**: resolve links to sibling `.md` files into wikilinks (see below).

Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, link resolution) around Pandoc's output.

### Nominating a related co-publication

Link to it the ordinary way, with a relative Markdown link:

```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```

There is nothing to declare in the frontmatter. The link stays a working link in Gitea's blob view, in an editor preview, and anywhere else the raw file is read — and on the wiki it becomes a wikilink to the target's page:

```wikitext
For more behind-the-scene shenanigans, see [[pulse under latent semantic envelopes - development notes|dev gibberish]].
```

**How the target becomes a page name.** The path is resolved relative to the linking file's directory, and the target file's own `title` frontmatter field is used verbatim as the wikilink target — the same string the bot writes that page to. There is no second name format to keep in sync: rename a page by editing its `title`, and every link aimed at it follows on the next publish. MediaWiki handles the first-letter capitalisation and the space-to-underscore in URLs itself.

| Written in the body | Rendered wikitext |
|---|---|
| `[dev gibberish](pulse_dev.md)` | `[[<target's title>\|dev gibberish]]` |
| `[pulse_dev.md](pulse_dev.md)` (label same as path) | `[[<target's title>]]` |
| `[notes](../ses/pulse_dev.md)` | `[[<target's title>\|notes]]` |
| `[notes](pulse_dev.md#Section)` | `[[<target's title>#Section\|notes]]` |
| `[site](https://example.com)` | `[https://example.com site]` (untouched) |

**A `.md` link that cannot be resolved fails the run.** Both cases abort with a non-zero exit before anything is published:

| Condition | Result |
|---|---|
| Target file does not exist | **error** — `file does not exist` |
| Target exists but is not published (no `wiki.publish: true`, or under `wip/`) | **error** — `file exists but is not published` |

Linking to a file does **not** publish it. Publication is always the target's own explicit `wiki.publish: true`; nothing is published as a side effect of being linked. If you want the link, opt the target in.

**Scope of a run.** A push publishes the files it changed, plus — one hop in each direction — any page a changed page links to, and any page that links to a changed page. The second is what keeps inbound links correct when a page's `title` moves. Regardless of how few pages a run publishes, the whole repository is always scanned first, so link resolution and title-collision checks see every publishable file.

**Anchors** (`pulse_dev.md#Section`) are passed through verbatim. MediaWiki derives its own section anchors from heading text, so an anchor that doesn't match a heading on the target page lands the reader at the top of it rather than at the section. Nothing validates this.

**Links inside a `columns` fence do not work.** The fence is verbatim by design (see below); Pandoc does not process Markdown in it, so a link written there would reach the wiki as literal text. The renderer detects this and fails the run rather than publishing it. Put the link in ordinary body prose.

### Columns (side-by-side) shorthand

For parallel content — e.g. an original and its translation — write a fenced code block tagged `columns` and separate the columns with a line containing only `===`:

````markdown
```columns
**原題**

一行目
二行目
===
**Title**

first line
second line
```
````

The renderer turns this into a flexbox row of `<poem>` columns (one per `===`-delimited section). Notes:

- **Line breaks and blank lines are preserved verbatim** — that's the point of using a fenced block; Pandoc passes the body through untouched, and each column is wrapped in `<poem>` so MediaWiki keeps the line breaks.
- **N columns**: use N−1 `===` separators. Two is the common case (original / translation).
- **Emphasis in Markdown**: write `*italic*`, `**bold**`, or `***bold-italic***` (asterisk style, single line) — these are converted to wikitext. Raw inline HTML such as `<b>…</b>` or `<br>` also still works (it's HTML-unescaped on the way out), but you shouldn't need it.
- This is a shared transform (`lib/wiki.py`), so it works for any pipeline, not just songs. Column width/gap styling currently lives in that transform.

## Renderer behaviour

A run happens in three phases, in this order, and the wiki is not contacted until the third. Anything wrong in phase 1 or 2 aborts with a non-zero exit and the wiki untouched — never half-updated.

**Phase 1 — scan and validate the whole repository.** For every `.md` outside `wip/` (and excluding `README.md`):

1. Parse the frontmatter; skip the file silently unless `wiki.publish: true`.
2. Validate the fields (see *Validation errors* below).
3. Register `title` and fail on a collision with any other publishable file, anywhere in the tree.
4. Build the path → page-title index every `.md` link resolves through.

**Phase 2 — render every page that this run will publish.** For each selected file:

1. Strip the leading `# Heading` from the body if present.
2. Pipe the body through `pandoc -f markdown -t mediawiki`, then expand any `columns` fenced blocks into side-by-side `<poem>` columns.
3. Resolve `.md` links to `[[Page title|label]]` wikilinks; fail on any that cannot be resolved, and on any link shape that survived rendering unresolved.
4. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
5. If `lrc` is declared, append a `[[Media:…]]` link.
6. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.

**Phase 3 — write.** For each rendered page:

1. If `lrc` is declared: upload the LRC file if its SHA1 has changed.
2. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
3. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
4. Otherwise, write the page with an edit summary referencing the source commit.

## Implemented scope

- Frontmatter parsing and validation
- Repo-wide page-title collision detection, before any wiki write
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Banner template injection
- LRC file upload and footer link
- Relative `.md` link resolution (`[label](notes.md)` → `[[wiki page title|label]]`), repo-wide and batch-independent
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]`) when `album` is set
- Additional categories via `categories` list
- Render-before-write: a broken page fails the run instead of half-updating the wiki
- Idempotent writes (no-op skip when content matches)
- Manual-page protection (skip pages without auto-gen banner)
- Files without `wiki.publish: true` silently skipped

## Not in scope (intentional)

- Multi-page aggregation: no album index pages generated; each file → its own wiki page. Album landing pages are wiki category pages, written freeform by the user.
- Language-aware rendering: the body conveys what languages exist; the renderer doesn't introspect.

## Examples

### A released song

```yaml
---
title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
  publish: true
release_date: 2026-06-18
lrc: pulse.lrc
---
```

Its body nominates the companion page with a plain relative link:

```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```

### A companion document (no album)

```yaml
---
title: pulse under latent semantic envelopes - development notes
wiki:
  publish: true
categories:
  - Song development notes
---
```

### A placeholder (not published)

```yaml
---
title: (placeholder)
album: kairo
wiki:
  publish: false
---
```

### A WIP file (in `wip/`, skipped regardless)

```yaml
---
title: 海淀
wiki:
  publish: false
---
```

## Validation errors the renderer must produce

Files without a `wiki.publish: true` frontmatter field are silently skipped — no error. The pipeline fails fast and loud only on files that are opted in (`wiki.publish: true`) with invalid data:

- Missing or empty `title` on a file with `wiki.publish: true`
- `wiki.publish` present but not a boolean
- `release_date` (if present) does not parse as YYYY-MM-DD
- `categories` (if present) is not a list
- Two publishable source files declare the same `title` (page-name collision) — checked across the whole repository, not just the files a push touched
- A `.md` link whose target file does not exist
- A `.md` link whose target exists but is not published
- A `.md` link that survives rendering unresolved — a link shape the renderer does not recognise, or a Markdown link written inside a `columns` fence

Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken.

Frontmatter keys this schema does not define are ignored rather than rejected.