novoyuuparosk-wiki/novoyuuparosk-auto-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'automation/songs' (#3) from automation/songs into main

Browse Source 
Reviewed-on: mikkeli/novoyuuparosk-auto-wiki#3
This commit was merged in pull request #3.
This commit is contained in:
Mikkeli Matlock
2026-06-09 15:52:21 +00:00
parent ef8ba3f01f f2fe4da667
commit dd6b8491d8
3 changed files with 177 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipelines/songs/README.md
+10 -6
View File
@@ -12,17 +12,18 @@ Implemented:
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`) + album category injection
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
- Album category injection (`[[Category:Album:<name>]]`); additional categories via `categories` list
- LRC file upload (idempotent SHA1 check) + footer link
- Inter-page sibling placeholder resolution (`[[TAG]]` → `[[wiki page title|label]]`)
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
- Files without `wiki.publish: true` silently skipped
- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
Deferred (schema reserves the fields; renderer doesn't yet act on them):
Not in scope (intentional):
- LRC parsing and embedding
- Inter-page sibling placeholder resolution
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
- Auto-generated album index pages (intentionally NOT done — user writes album category-page descriptions freeform)
- Auto-generated album index pages — user writes album category-page descriptions freeform
## Source repo and branch convention
@@ -79,7 +80,10 @@ The pipeline always runs `--all`: every file with `wiki.publish: true` (not unde
| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
| Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 |
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
| Album landing pages | Auto-injected `[[Category:<album>]]`; the wiki category page is user-written and not overwritten | 2026-06-09 |
| Album categories | Auto-injected `[[Category:Album:<name>]]`; category page is user-written and not overwritten | 2026-06-09 |
| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:<value>]]` | 2026-06-10 |
| `album` optional | Companion/sibling pages don't need an album; omitting the field emits no album category | 2026-06-10 |
| Sibling resolution | Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc | 2026-06-10 |
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
| Publish mode | Always `--all`; no diff detection — small repo, simpler than fragile git-diff gating | 2026-06-09 |
pipelines/songs/SCHEMA.md
+108 -53
View File
@@ -2,7 +2,7 @@
Source-file contract for the songs pipeline. Files in `mikkeli/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar, except that `[[TAG]]` placeholders declared via `wiki.siblings` are substituted with resolved wikilinks after rendering.
## File layout
@@ -12,15 +12,20 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
release_date: 2026-06-09
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
# pulse under latent semantic envelopes
... body markdown ...
... body markdown, PULSE_DEV used inline ...
```
The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc.
The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc, then applies sibling substitutions on the wikitext output.
## Fields
@@ -34,56 +39,95 @@ Example: `title: pulse under latent semantic envelopes` → wiki page `Pulse und
The body should contain a matching `# Title` first-line heading for human readability when viewing the source file directly (in an editor, on Gitea's blob view, etc.). The renderer strips this first-line heading during conversion — the wiki page already supplies its own title.
#### `album` (string)
The full album name. Drives `[[Category:<album name>]]` injection at the bottom of the rendered wiki page. MediaWiki auto-creates the category page when at least one page references it.
The pipeline **does not** create or update a dedicated album info/index page. Album landing pages are the wiki's category pages, which you write freeform yourself.
Example: `album: Shirakaba Express Service` → category tag `[[Category:Shirakaba Express Service]]`.
For files where no album applies (placeholders, songs without an assigned album), use a sensible literal like `(unassigned)` and accept that they'll get a `[[Category:(unassigned)]]` tag. The field is required to keep the schema's contract simple.
#### `wiki.publish` (boolean)
Publishing gate. `true` means the pipeline writes this file to the wiki. `false` means the file is ignored.
The `wiki` key is a mapping rather than a flat field so future toggles (`wiki.protected`, `wiki.summary_template`, etc.) can be added without restructuring. None are defined yet.
The `wiki` key is a mapping rather than a flat field so future toggles (`wiki.protected`, `wiki.summary_template`, etc.) can be added without restructuring.
Files under `wip/` are also skipped, regardless of `wiki.publish`.
### Optional
#### `album` (string)
The full album name. Drives `[[Category:Album:<album name>]]` injection at the bottom of the rendered wiki page.
Example: `album: Shirakaba Express Service` → category tag `[[Category:Album:Shirakaba Express Service]]`.
Pages without an album (companion documents, placeholders, songs without an assigned album) simply omit this field — no category is injected unless `categories` provides one.
#### `categories` (list of strings)
Additional wiki categories to inject, beyond the album category. Each entry becomes `[[Category:<value>]]`.
```yaml
categories:
- Song development notes
- Work in progress
```
`album` and `categories` are additive — a page can have both.
#### `release_date` (ISO 8601 date)
The song's release date in `YYYY-MM-DD` format. Skipped if absent. Currently informational; may drive sort order on future auto-generated indexes.
#### `lrc` (string, relative path)
Pointer to a companion LRC file, relative to the `.md` file's location.
Pointer to a companion LRC file, relative to the `.md` file's location. The pipeline uploads the file to the wiki as `<title>.lrc` (if the content has changed) and appends a `[[Media:<title>.lrc|Synced lyrics (.lrc)]]` link to the page footer.
Example: in `ses/tunnels.md`, `lrc: tunnels.lrc` declares that `ses/tunnels.lrc` is the companion synced-lyrics file.
Example: in `ses/pulse.md`, `lrc: pulse.lrc` declares that `ses/pulse.lrc` is the companion synced-lyrics file.
**Deferred for v1**: the field is permitted by the schema but the renderer doesn't yet read or embed LRC contents.
If the LRC file is missing or not valid UTF-8, the upload is skipped with a warning — the page still publishes.
#### `siblings` (mapping of placeholder → relative path)
#### `wiki.siblings` (list of sibling declarations)
Placeholder-to-file mapping for inter-page wikilinks.
Bill of materials for inter-page wikilinks. Each entry declares a sibling `.md` file and a tag that can be used as a `[[TAG]]` placeholder anywhere in the body.
```yaml
wiki:
siblings:
PULSE_DEV: pulse_dev.md
- path: pulse_dev.md # relative to this file's directory
tag: PULSE_DEV # placeholder used in body as [[PULSE_DEV]]
label: development notes # optional display text
```
In the body markdown, write `[[PULSE_DEV]]` for a wikilink that displays the target's title, or `[[PULSE_DEV|custom display]]` for an aliased display. Resolution uses the target file's `title:` frontmatter as the wiki page name.
**Fields per entry:**
**Deferred for v1**: the field is permitted by the schema but placeholders pass through untouched.
| Field | Required for resolution | Purpose |
|---|---|---|
| `path` | yes | Relative path to the sibling `.md` file |
| `tag` | yes | Placeholder identifier; pipeline does a case-sensitive string replace of `TAG` in the rendered wikitext |
| `label` | no | Display text for the resolved link |
An entry missing either `path` or `tag` is silently skipped — no substitution is attempted for that entry.
**Resolution:**
The pipeline resolves each declared sibling's `path` to its wiki page title by looking it up in the set of all publishable files processed in the same run. Substitution is a case-sensitive string replace of the bare tag. Substitution rules:
| Condition | `TAG` becomes |
|---|---|
| `path` resolves to a published page, `label` present | `[[wiki page title\|label]]` |
| `path` resolves to a published page, no `label` | `[[wiki page title]]` |
| `path` not in published set (file exists but not published), `label` present | `label` (plain text) |
| `path` not in published set, no `label` | `TAG` (plain text) |
Declared siblings whose file doesn't exist on disk are a **validation error**. Siblings that exist but aren't published (no `wiki.publish: true`) degrade gracefully per the table above.
The siblings list is a bill of materials — declaring a sibling has no effect unless its `[[TAG]]` appears in the body. Unused entries are inert.
**`--files` mode**: sibling resolution is limited to the files in that batch. Siblings pointing to files outside the batch degrade to plain text. This is acceptable since the pipeline always runs `--all` in the workflow.
## Body
Plain markdown. The renderer applies a single transformation before handing the body to Pandoc: the first-line `# Heading`, if present, is stripped.
Plain markdown. The renderer applies two transformations before and after Pandoc:
Do not embed wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body. Use markdown idioms only; the renderer adds the metadata-derived bits (banner, categories, sibling resolution) around Pandoc's output.
1. **Pre-Pandoc**: strip the first-line `# Heading` if present.
2. **Post-Pandoc**: substitute `TAG` placeholders declared in `wiki.siblings`.
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, sibling resolution) around Pandoc's output.
## Renderer behaviour
@@ -91,29 +135,33 @@ For each `.md` with `wiki.publish: true` and not under `wip/`:
1. Parse and validate frontmatter.
2. Strip the leading `# Heading` from the body if present.
3. Replace `[[PLACEHOLDER]]` markers with safe tokens. *(Deferred v1; no-op until siblings are implemented.)*
4. Pipe the body through `pandoc -f markdown -t mediawiki`.
5. Re-substitute sibling tokens with resolved `[[Page|Display]]` wikilinks. *(Deferred v1.)*
6. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
7. Append the album category: `[[Category:<album>]]`.
3. Pipe the body through `pandoc -f markdown -t mediawiki`.
4. Substitute `TAG` placeholders with resolved `[[Page|Display]]` wikilinks (or fallback text).
5. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
6. If `lrc` is declared: upload the LRC file if its SHA1 has changed; append a `[[Media:…]]` link.
7. Append category tags: `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
9. Otherwise, write the page with an edit summary referencing the source commit.
9. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
10. Otherwise, write the page with an edit summary referencing the source commit.
## v1 scope
## Implemented scope
**Implemented:**
- Frontmatter parsing and validation for `title`, `album`, `wiki.publish`, `release_date`
- Markdown body → wikitext via Pandoc
- Frontmatter parsing and validation
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Banner template injection (requires `Template:Auto-generated` to exist on the wiki — see the pipeline `README.md` for the one-off setup)
- Album category injection
- Idempotent writes (no-op skip)
- Banner template injection
- LRC file upload and footer link
- Sibling placeholder resolution (`[[TAG]]` → `[[wiki page title|label]]`)
- Album category injection (`[[Category:Album:<name>]]`)
- Additional categories via `categories` list
- Idempotent writes (no-op skip when content matches)
- Manual-page protection (skip pages without auto-gen banner)
- Files without `wiki.publish: true` silently skipped
**Deferred:**
- `lrc` field consumption
- `siblings` placeholder resolution
- Multi-page aggregation (no album index pages generated; each file → its own wiki page)
- Language-aware rendering (the body conveys what languages exist; the renderer doesn't introspect)
## Not in scope (intentional)
- Multi-page aggregation: no album index pages generated; each file → its own wiki page. Album landing pages are wiki category pages, written freeform by the user.
- Language-aware rendering: the body conveys what languages exist; the renderer doesn't introspect.
## Examples
@@ -125,22 +173,28 @@ title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
release_date: 2026-06-09
siblings:
- path: pulse_dev.md
tag: PULSE_DEV
label: development notes
release_date: 2026-06-18
lrc: pulse.lrc
---
```
### A dev sister file (not published)
### A companion/sibling document (no album)
```yaml
---
title: pulse under latent semantic envelopes — personal dev notes
album: Shirakaba Express Service
title: pulse under latent semantic envelopes - development notes
wiki:
publish: false
publish: true
categories:
- Song development notes
---
```
### A placeholder file (no real song yet)
### A placeholder (not published)
```yaml
---
@@ -151,12 +205,11 @@ wiki:
---
```
### A WIP file (in `wip/`, schema still applies but the file is skipped regardless)
### A WIP file (in `wip/`, skipped regardless)
```yaml
---
title: 海淀
album: (unassigned)
wiki:
publish: false
---
@@ -166,10 +219,12 @@ wiki:
Files without a `wiki.publish: true` frontmatter field are silently skipped — no error. The pipeline fails fast and loud only on files that are opted in (`wiki.publish: true`) with invalid data:
- Missing required field (`title` or `album`) on a file with `wiki.publish: true`
- Missing or empty `title` on a file with `wiki.publish: true`
- `wiki.publish` present but not a boolean
- `release_date` (if present) does not parse as YYYY-MM-DD
- `siblings` (if present) references a file that doesn't exist
- `categories` (if present) is not a list
- `wiki.siblings` entry that has both `path` and `tag` declared, but the `path` file does not exist on disk
- `wiki.siblings` entry that is not a mapping
- Two publishable source files declare the same `title` (page-name collision)
Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.
pipelines/songs/publish.py
+57 -7
View File
@@ -17,10 +17,9 @@ import mwclient
def validate(fm: dict, path: Path) -> None:
errors = []
for field in ("title", "album"):
val = fm.get(field)
val = fm.get("title")
if not val or not str(val).strip():
errors.append(f"missing or empty '{field}'")
errors.append("missing or empty 'title'")
if not isinstance(fm.get("wiki", {}).get("publish"), bool):
errors.append("'wiki.publish' must be a boolean (true/false), not a string")
@@ -31,6 +30,19 @@ def validate(fm: dict, path: Path) -> None:
except ValueError:
errors.append("'release_date' must be YYYY-MM-DD")
if "categories" in fm and not isinstance(fm["categories"], list):
errors.append("'categories' must be a list")
for i, sib in enumerate(fm.get("wiki", {}).get("siblings") or []):
if not isinstance(sib, dict):
errors.append(f"wiki.siblings[{i}] must be a mapping")
continue
sib_path = sib.get("path")
sib_tag = sib.get("tag")
if sib_path and sib_tag:
if not (path.parent / str(sib_path)).resolve().exists():
errors.append(f"wiki.siblings[{i}]: path '{sib_path}' not found")
if errors:
raise ValueError(f"{path}: " + "; ".join(errors))
@@ -59,11 +71,47 @@ def markdown_to_wikitext(body: str) -> str:
return result.stdout
def build_sibling_map(posts: dict) -> dict[str, str]:
"""Map abs_path_str → wiki_title for all published posts."""
return {str(path): str(post.metadata["title"]) for path, post in posts.items()}
def apply_sibling_substitutions(wikitext: str, this_path: Path, fm_wiki: dict, sibling_map: dict[str, str]) -> str:
"""Replace [[TAG]] placeholders using the declared sibling bill of materials."""
for sib in (fm_wiki.get("siblings") or []):
if not isinstance(sib, dict):
continue
tag = sib.get("tag")
path = sib.get("path")
if not tag or not path:
continue
label = (sib.get("label") or "").strip()
abs_sib = str((this_path.parent / str(path)).resolve())
title = sibling_map.get(abs_sib)
if title and label:
replacement = f"[[{title}|{label}]]"
elif title:
replacement = f"[[{title}]]"
elif label:
replacement = label
else:
replacement = tag
wikitext = wikitext.replace(tag, replacement)
return wikitext
def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: str, lrc_filename: str | None = None) -> str:
banner = f"{{{{Auto-generated|source={source_url}|commit={source_ref}}}}}"
category = f"[[Category:{fm['album']}]]"
lrc_line = f"[[Media:{lrc_filename}|Synced lyrics (.lrc)]]\n" if lrc_filename else ""
return f"{banner}\n\n{body_wikitext}{lrc_line}{category}\n"
cat_parts = []
if fm.get("album"):
cat_parts.append(f"[[Category:Album:{fm['album']}]]")
for cat in (fm.get("categories") or []):
cat_parts.append(f"[[Category:{cat}]]")
categories = "\n".join(cat_parts) + "\n" if cat_parts else ""
return f"{banner}\n\n{body_wikitext}{lrc_line}{categories}"
def connect_wiki() -> mwclient.Site:
@@ -110,13 +158,14 @@ def upload_lrc(lrc_path: Path, title: str, site: mwclient.Site) -> bool:
AUTO_BANNER_PREFIX = "{{Auto-generated"
def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, source_ref: str, gitea_repo_url: str) -> str:
def publish_one(abs_path: Path, source_dir: Path, post, site: mwclient.Site, source_ref: str, gitea_repo_url: str, sibling_map: dict[str, str]) -> str:
"""Returns 'published', 'noop', or 'skipped-manual'."""
rel = abs_path.relative_to(source_dir)
fm = post.metadata
title = str(fm["title"])
body_wikitext = markdown_to_wikitext(strip_first_h1(post.content))
body_wikitext = apply_sibling_substitutions(body_wikitext, abs_path, fm.get("wiki", {}), sibling_map)
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
lrc_filename = None
@@ -204,6 +253,7 @@ def main() -> None:
print(f"Validating {len(files)} file(s)...")
posts = load_and_validate(files, source_dir)
sibling_map = build_sibling_map(posts)
print("Validation passed.")
site = connect_wiki()
@@ -211,7 +261,7 @@ def main() -> None:
counts = {"published": 0, "noop": 0, "skipped-manual": 0}
for abs_path, post in posts.items():
rel = abs_path.relative_to(source_dir)
outcome = publish_one(abs_path, source_dir, post, site, source_ref, gitea_repo_url)
outcome = publish_one(abs_path, source_dir, post, site, source_ref, gitea_repo_url, sibling_map)
counts[outcome] += 1
if outcome == "skipped-manual":
print(f" [skipped-manual] {rel} ← page exists without auto-gen banner; delete or add banner to hand over to bot", file=sys.stderr)