Merge pull request 'automation/songs' (#1) from automation/songs into main
Just get it done
This commit was merged in pull request #1.
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
SOURCE_REF:
|
||||
description: "Commit SHA that was pushed to ncmr-songs"
|
||||
required: true
|
||||
type: string
|
||||
SOURCE_BASE_REF:
|
||||
description: "Commit SHA before the push (all-zeros = first push)"
|
||||
required: true
|
||||
type: string
|
||||
FULL_PUBLISH:
|
||||
description: "Re-publish every file regardless of diff"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
container:
|
||||
image: python:3.12-slim
|
||||
|
||||
steps:
|
||||
- name: Install system deps
|
||||
run: |
|
||||
apt-get update -qq
|
||||
apt-get install -y --no-install-recommends git pandoc ca-certificates
|
||||
|
||||
- name: Checkout ncmr-songs
|
||||
env:
|
||||
FAPAT: ${{ secrets.FAPAT }}
|
||||
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
|
||||
run: |
|
||||
git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/ncmr-songs" ncmr-songs
|
||||
git -C ncmr-songs checkout ${{ inputs.SOURCE_REF }}
|
||||
|
||||
- name: Checkout auto-wiki
|
||||
env:
|
||||
FAPAT: ${{ secrets.FAPAT }}
|
||||
URL_TO_GITEA: ${{ vars.URL_TO_GITEA }}
|
||||
run: git clone "http://mikkeli:${FAPAT}@${URL_TO_GITEA#http://}/mikkeli/novoyuuparosk-auto-wiki" auto-wiki
|
||||
|
||||
- name: Install Python deps
|
||||
run: pip install --no-cache-dir -r auto-wiki/pipelines/songs/requirements.txt
|
||||
|
||||
- name: Detect changed files
|
||||
id: diff
|
||||
run: |
|
||||
BASE="${{ inputs.SOURCE_BASE_REF }}"
|
||||
HEAD="${{ inputs.SOURCE_REF }}"
|
||||
# All-zeros base means first push — publish everything
|
||||
if [ -z "$BASE" ] || echo "$BASE" | grep -qE '^0+$'; then
|
||||
echo "mode=all" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
git -C ncmr-songs diff --name-only "$BASE" "$HEAD" -- '*.md' \
|
||||
| grep -v '^wip/' \
|
||||
| grep -v '^README\.md' \
|
||||
> /tmp/changed.txt 2>/dev/null || true
|
||||
echo "Changed files:"
|
||||
cat /tmp/changed.txt
|
||||
echo "mode=incremental" >> "$GITHUB_OUTPUT"
|
||||
echo "files=$(tr '\n' ' ' < /tmp/changed.txt)" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Publish
|
||||
env:
|
||||
WIKI_API_URL: ${{ vars.WIKI_API_URL }}
|
||||
WIKI_BASE_URL: ${{ vars.WIKI_BASE_URL }}
|
||||
WIKI_BOT_USER: ${{ secrets.WIKI_BOT_USER }}
|
||||
WIKI_BOT_PASSWORD: ${{ secrets.WIKI_BOT_PASSWORD }}
|
||||
SOURCE_REF: ${{ inputs.SOURCE_REF }}
|
||||
GITEA_REPO_URL: ${{ vars.URL_TO_GITEA }}/mikkeli/ncmr-songs
|
||||
run: |
|
||||
MODE="${{ steps.diff.outputs.mode }}"
|
||||
if [ "$MODE" = "all" ] || [ "${{ inputs.FULL_PUBLISH }}" = "true" ]; then
|
||||
python auto-wiki/pipelines/songs/render.py \
|
||||
--source-dir ncmr-songs \
|
||||
--all
|
||||
else
|
||||
FILES="${{ steps.diff.outputs.files }}"
|
||||
if [ -z "$FILES" ]; then
|
||||
echo "No changed .md files detected. Nothing to publish."
|
||||
exit 0
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
python auto-wiki/pipelines/songs/render.py \
|
||||
--source-dir ncmr-songs \
|
||||
--files $FILES
|
||||
fi
|
||||
@@ -0,0 +1,2 @@
|
||||
# claude local settings
|
||||
.claude/
|
||||
@@ -1,3 +1,94 @@
|
||||
# novoyuuparosk-auto-wiki
|
||||
|
||||
CI/CD or in English auto-apply pipelines for the wiki
|
||||
CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from upstream content repos.
|
||||
|
||||
## Pipelines
|
||||
|
||||
| Path | Source repo | Purpose | Status |
|
||||
|---|---|---|---|
|
||||
| [`pipelines/songs/`](pipelines/songs/) | `mikkeli/ncmr-songs` | Song lyric pages | in development (v1, no executable code yet) |
|
||||
|
||||
Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime, decisions). This root README covers only what's cross-cutting.
|
||||
|
||||
## Architecture
|
||||
|
||||
Hybrid layout. The Gitea Actions trigger has to live in the source repo (Gitea only fires workflows from `.gitea/workflows/` of the pushed-to repo); the real logic lives here. Source-repo workflows are thin stubs that call reusable workflows defined here.
|
||||
|
||||
```
|
||||
<source-repo>/
|
||||
.gitea/workflows/<name>.yml <- thin stub, calls into this repo
|
||||
|
||||
novoyuuparosk-auto-wiki/ <- this repo
|
||||
.gitea/workflows/<pipeline>.yml <- reusable workflows (the actual logic)
|
||||
pipelines/<pipeline>/ <- per-pipeline code, schema, templates
|
||||
lib/ <- shared modules (MediaWiki client, etc.)
|
||||
```
|
||||
|
||||
## Wiki
|
||||
|
||||
- Base URL: https://wiki.novoyuuparosk.org
|
||||
- MediaWiki API: https://wiki.novoyuuparosk.org/api.php *(confirmed)*
|
||||
|
||||
## Bot identity
|
||||
|
||||
MediaWiki BotPassword issued for user `Dubrowski`, bot name `giteaAutomaton`.
|
||||
|
||||
Login as `Dubrowski@giteaAutomaton` with password `d8jua48t65jgjp3dfcqhfg7257tri6ui`.
|
||||
(Legacy form: username `Dubrowski`, password `giteaAutomaton@d8jua48t65jgjp3dfcqhfg7257tri6ui`.)
|
||||
|
||||
Plaintext here is acceptable for the current phase (private repo, home-Pi LAN-only Gitea). Rotate before any of those preconditions change. Note that `git log` retains this string forever, so rotation requires a wiki-side BotPassword regeneration regardless of what happens to this file.
|
||||
|
||||
## Runner infrastructure
|
||||
|
||||
One `act_runner` instance serves all pipelines. Runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) inside the same `docker-compose` stack that hosts the Gitea instance. Job execution is via the host Docker socket — runner is a container, jobs spawn as sibling containers.
|
||||
|
||||
`act_runner` build: `linux-arm64`, from the `gitea/act_runner` Docker image.
|
||||
|
||||
## Gitea Actions setup (cross-cutting)
|
||||
|
||||
Secrets and variables are scoped to user `mikkeli` (no orgs on this instance), inherited by all repos under that account.
|
||||
|
||||
**Secrets:**
|
||||
- `WIKI_BOT_USER` = `Dubrowski@giteaAutomaton`
|
||||
- `WIKI_BOT_PASSWORD` = the value from *Bot identity* above
|
||||
- `FAPAT` = Full-Access PAT under `mikkeli`, used by source-repo shim workflows to clone this repo at workflow time
|
||||
|
||||
**Variables:**
|
||||
- `WIKI_BASE_URL` = `https://wiki.novoyuuparosk.org`
|
||||
- `WIKI_API_URL` = `https://wiki.novoyuuparosk.org/api.php`
|
||||
|
||||
## Branch naming
|
||||
|
||||
- This repo: `automation/<pipeline-name>` for pipeline-development branches (e.g., `automation/songs`).
|
||||
- Source repos: each pipeline's README defines the source-side branch convention (e.g., `autowiki/<song-slug>` in `ncmr-songs`).
|
||||
|
||||
## Decisions log (cross-cutting)
|
||||
|
||||
| Decision | Value | Date |
|
||||
|---|---|---|
|
||||
| Architecture | Hybrid: stub in source repo, logic in this repo via reusable workflows | 2026-06-09 |
|
||||
| Workflow pattern | Gitea reusable workflows (`workflow_call`); requires Gitea ≥ 1.20 — confirmed 1.25+ | 2026-06-09 |
|
||||
| Runner execution | Docker, added as a service to the existing Gitea docker-compose | 2026-06-09 |
|
||||
| Secret/runner scope | User-level on `mikkeli` (no orgs on this instance) | 2026-06-09 |
|
||||
| MediaWiki API path | `api.php` (classic action API) | 2026-06-09 |
|
||||
| Branch naming (this repo) | `automation/<pipeline>` for pipeline-development branches | 2026-06-09 |
|
||||
|
||||
Per-pipeline decisions live in each pipeline's README.
|
||||
|
||||
## Setup checklist (cross-cutting)
|
||||
|
||||
Via the Gitea web UI logged in as `mikkeli`:
|
||||
|
||||
- [v] User-scoped secrets and variables set per *Gitea Actions setup* above
|
||||
- [v] `WIKI_BOT_USER`
|
||||
- [v] `WIKI_BOT_PASSWORD`
|
||||
- [v] `FAPAT` (Full-Access PAT — value not stored in this README; saved directly into the Gitea secret. Regenerate if lost.)
|
||||
- [v] `WIKI_BASE_URL`
|
||||
- [v] `WIKI_API_URL`
|
||||
|
||||
With Pi access:
|
||||
|
||||
- [ ] Add `act_runner` service to the existing Gitea docker-compose
|
||||
- [ ] Generate a runner registration token at `/-/admin/actions/runners` (or `/user/settings/actions/runners` if user-scoped runners are exposed), bake into the compose env, `docker compose up -d act_runner`, confirm "online" in the Gitea UI
|
||||
|
||||
Per-pipeline setup lives in each pipeline's README. Start with [`pipelines/songs/`](pipelines/songs/).
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# pipelines/songs
|
||||
|
||||
Publishes lyric pages from `mikkeli/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
|
||||
|
||||
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
|
||||
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
|
||||
|
||||
## v1 scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
|
||||
- Pandoc-based markdown → wikitext body rendering
|
||||
- First-line h1 stripping
|
||||
- Auto-generated banner + album category injection
|
||||
- MediaWiki bot API write with idempotency (no-op edits skipped)
|
||||
|
||||
Deferred (schema reserves the fields; renderer doesn't yet act on them):
|
||||
|
||||
- LRC parsing and embedding
|
||||
- Inter-page sibling placeholder resolution
|
||||
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
|
||||
- Auto-generated album index pages (intentionally NOT done — user writes album category-page descriptions freeform)
|
||||
|
||||
## Source repo and branch convention
|
||||
|
||||
Source repo: `mikkeli/ncmr-songs`. The pipeline triggers on push to `master`.
|
||||
|
||||
**Branch convention in `ncmr-songs`**: per-song short-lived branches named `autowiki/<song-slug>` (e.g., `autowiki/pulse`). Create when staging edits, merge to master when ready to publish, delete after one cycle.
|
||||
|
||||
**Excluded paths** (won't trigger the pipeline):
|
||||
|
||||
- `wip/**`
|
||||
- `README.md`, `.gitignore`, `.claude/**`
|
||||
|
||||
## Invocation
|
||||
|
||||
The pipeline is invoked via a reusable Gitea Actions workflow (`.gitea/workflows/publish-songs.yml` in this repo). The `ncmr-songs` repo holds a thin stub workflow (`.gitea/workflows/publish.yml`) that calls into this one on push to `master`.
|
||||
|
||||
### Workflow inputs and secrets
|
||||
|
||||
| Name | Kind | Source | Purpose |
|
||||
|---|---|---|---|
|
||||
| `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
|
||||
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL (used in the banner link) |
|
||||
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login |
|
||||
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
|
||||
| `SOURCE_DIR` | workflow input | from the stub | Path to checked-out `ncmr-songs` working tree |
|
||||
| `SOURCE_REF` | workflow input | from the stub | The pushed-to commit SHA |
|
||||
| `SOURCE_BASE_REF` | workflow input | from the stub | The SHA prior to the push (for diff-based change detection) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Python 3.12+
|
||||
- Pandoc 2.x or newer (apt-installable in the runner container)
|
||||
- `mwclient` or `requests` for the MediaWiki API
|
||||
- `PyYAML` for frontmatter parsing
|
||||
- `python-frontmatter` (convenience wrapper around PyYAML for markdown frontmatter)
|
||||
|
||||
`requirements.txt` will be added when the implementation lands.
|
||||
|
||||
## One-off wiki setup
|
||||
|
||||
Before the first run, two things must exist on the wiki:
|
||||
|
||||
1. **`Template:Auto-generated`** — the banner injected at the top of every auto-published page. Wikitext for this template is included in the implementation step (not yet written).
|
||||
2. **The bot user has edit rights** for the namespace(s) the pipeline writes to. The default main namespace is fine; verify by attempting a manual edit via the bot account before relying on the pipeline.
|
||||
|
||||
## Modes
|
||||
|
||||
- **Incremental** (default, triggered by push to `master`): publishes only files changed between `SOURCE_BASE_REF` and `SOURCE_REF`.
|
||||
- **Full** (`--all` flag, triggered by `workflow_dispatch`): re-publishes every publishable file in the source tree. Use after template or renderer changes.
|
||||
|
||||
## Decisions log (songs pipeline)
|
||||
|
||||
| Decision | Value | Date |
|
||||
|---|---|---|
|
||||
| Metadata source | YAML frontmatter inside each `.md` (see [SCHEMA.md](SCHEMA.md)) | 2026-06-09 |
|
||||
| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
|
||||
| Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 |
|
||||
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
|
||||
| Album landing pages | Auto-injected `[[Category:<album>]]`; the wiki category page is user-written and not overwritten by the pipeline | 2026-06-09 |
|
||||
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
|
||||
|
||||
## Status
|
||||
|
||||
Schema and pipeline overview committed. No executable code yet.
|
||||
|
||||
**Next**: implement renderer (Python module), workflow YAMLs (this repo + `ncmr-songs` stub), `Template:Auto-generated` wikitext, and end-to-end dry-run on `ses/pulse.md`.
|
||||
@@ -0,0 +1,177 @@
|
||||
# SCHEMA — song source files
|
||||
|
||||
Source-file contract for the songs pipeline. Files in `mikkeli/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
|
||||
|
||||
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar.
|
||||
|
||||
## File layout
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: pulse under latent semantic envelopes
|
||||
album: Shirakaba Express Service
|
||||
wiki:
|
||||
publish: true
|
||||
release_date: 2026-06-09
|
||||
---
|
||||
|
||||
# pulse under latent semantic envelopes
|
||||
|
||||
... body markdown ...
|
||||
```
|
||||
|
||||
The renderer parses the frontmatter, validates required fields, then hands the body (with the first `# Heading` line removed) to Pandoc.
|
||||
|
||||
## Fields
|
||||
|
||||
### Required
|
||||
|
||||
#### `title` (string)
|
||||
|
||||
The MediaWiki page name that the bot writes or updates. MediaWiki capitalises the first letter automatically; lowercase input is fine.
|
||||
|
||||
Example: `title: pulse under latent semantic envelopes` → wiki page `Pulse under latent semantic envelopes` at URL `/wiki/Pulse_under_latent_semantic_envelopes`.
|
||||
|
||||
The body should contain a matching `# Title` first-line heading for human readability when viewing the source file directly (in an editor, on Gitea's blob view, etc.). The renderer strips this first-line heading during conversion — the wiki page already supplies its own title.
|
||||
|
||||
#### `album` (string)
|
||||
|
||||
The full album name. Drives `[[Category:<album name>]]` injection at the bottom of the rendered wiki page. MediaWiki auto-creates the category page when at least one page references it.
|
||||
|
||||
The pipeline **does not** create or update a dedicated album info/index page. Album landing pages are the wiki's category pages, which you write freeform yourself.
|
||||
|
||||
Example: `album: Shirakaba Express Service` → category tag `[[Category:Shirakaba Express Service]]`.
|
||||
|
||||
For files where no album applies (placeholders, songs without an assigned album), use a sensible literal like `(unassigned)` and accept that they'll get a `[[Category:(unassigned)]]` tag. The field is required to keep the schema's contract simple.
|
||||
|
||||
#### `wiki.publish` (boolean)
|
||||
|
||||
Publishing gate. `true` means the pipeline writes this file to the wiki. `false` means the file is ignored.
|
||||
|
||||
The `wiki` key is a mapping rather than a flat field so future toggles (`wiki.protected`, `wiki.summary_template`, etc.) can be added without restructuring. None are defined yet.
|
||||
|
||||
Files under `wip/` are also skipped, regardless of `wiki.publish`.
|
||||
|
||||
### Optional
|
||||
|
||||
#### `release_date` (ISO 8601 date)
|
||||
|
||||
The song's release date in `YYYY-MM-DD` format. Skipped if absent. Currently informational; may drive sort order on future auto-generated indexes.
|
||||
|
||||
#### `lrc` (string, relative path)
|
||||
|
||||
Pointer to a companion LRC file, relative to the `.md` file's location.
|
||||
|
||||
Example: in `ses/tunnels.md`, `lrc: tunnels.lrc` declares that `ses/tunnels.lrc` is the companion synced-lyrics file.
|
||||
|
||||
**Deferred for v1**: the field is permitted by the schema but the renderer doesn't yet read or embed LRC contents.
|
||||
|
||||
#### `siblings` (mapping of placeholder → relative path)
|
||||
|
||||
Placeholder-to-file mapping for inter-page wikilinks.
|
||||
|
||||
```yaml
|
||||
siblings:
|
||||
PULSE_DEV: pulse_dev.md
|
||||
```
|
||||
|
||||
In the body markdown, write `[[PULSE_DEV]]` for a wikilink that displays the target's title, or `[[PULSE_DEV|custom display]]` for an aliased display. Resolution uses the target file's `title:` frontmatter as the wiki page name.
|
||||
|
||||
**Deferred for v1**: the field is permitted by the schema but placeholders pass through untouched.
|
||||
|
||||
## Body
|
||||
|
||||
Plain markdown. The renderer applies a single transformation before handing the body to Pandoc: the first-line `# Heading`, if present, is stripped.
|
||||
|
||||
Do not embed wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]` not declared via `siblings`, etc.) in the body. Use markdown idioms only; the renderer adds the metadata-derived bits (banner, categories, sibling resolution) around Pandoc's output.
|
||||
|
||||
## Renderer behaviour
|
||||
|
||||
For each `.md` with `wiki.publish: true` and not under `wip/`:
|
||||
|
||||
1. Parse and validate frontmatter.
|
||||
2. Strip the leading `# Heading` from the body if present.
|
||||
3. Replace `[[PLACEHOLDER]]` markers with safe tokens. *(Deferred v1; no-op until siblings are implemented.)*
|
||||
4. Pipe the body through `pandoc -f markdown -t mediawiki`.
|
||||
5. Re-substitute sibling tokens with resolved `[[Page|Display]]` wikilinks. *(Deferred v1.)*
|
||||
6. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|generated_at=<ISO timestamp>}}`.
|
||||
7. Append the album category: `[[Category:<album>]]`.
|
||||
8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
|
||||
9. Otherwise, write the page with an edit summary referencing the source commit.
|
||||
|
||||
## v1 scope
|
||||
|
||||
**Implemented:**
|
||||
- Frontmatter parsing and validation for `title`, `album`, `wiki.publish`, `release_date`
|
||||
- Markdown body → wikitext via Pandoc
|
||||
- First-line h1 stripping
|
||||
- Banner template injection (requires `Template:Auto-generated` to exist on the wiki — see the pipeline `README.md` for the one-off setup)
|
||||
- Album category injection
|
||||
- Idempotent writes (no-op skip)
|
||||
|
||||
**Deferred:**
|
||||
- `lrc` field consumption
|
||||
- `siblings` placeholder resolution
|
||||
- Multi-page aggregation (no album index pages generated; each file → its own wiki page)
|
||||
- Language-aware rendering (the body conveys what languages exist; the renderer doesn't introspect)
|
||||
|
||||
## Examples
|
||||
|
||||
### A released song
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: pulse under latent semantic envelopes
|
||||
album: Shirakaba Express Service
|
||||
wiki:
|
||||
publish: true
|
||||
release_date: 2026-06-09
|
||||
---
|
||||
```
|
||||
|
||||
### A dev sister file (not published)
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: pulse under latent semantic envelopes — personal dev notes
|
||||
album: Shirakaba Express Service
|
||||
wiki:
|
||||
publish: false
|
||||
---
|
||||
```
|
||||
|
||||
### A placeholder file (no real song yet)
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: (placeholder)
|
||||
album: kairo
|
||||
wiki:
|
||||
publish: false
|
||||
---
|
||||
```
|
||||
|
||||
### A WIP file (in `wip/`, schema still applies but the file is skipped regardless)
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: 海淀
|
||||
album: (unassigned)
|
||||
wiki:
|
||||
publish: false
|
||||
---
|
||||
```
|
||||
|
||||
## Validation errors the renderer must produce
|
||||
|
||||
The pipeline fails fast and loud on the following:
|
||||
|
||||
- Missing or unparseable frontmatter block
|
||||
- Missing required field (`title`, `album`, `wiki.publish`)
|
||||
- `title` is empty or whitespace-only
|
||||
- `wiki.publish` is not a boolean
|
||||
- `release_date` (if present) does not parse as YYYY-MM-DD
|
||||
- `siblings` (if present) references a file that doesn't exist
|
||||
- Two source files declare the same `title` (page-name collision)
|
||||
|
||||
Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.
|
||||
@@ -0,0 +1,183 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Songs pipeline renderer — converts source .md files to MediaWiki pages."""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import frontmatter
|
||||
import mwclient
|
||||
|
||||
|
||||
def validate(fm: dict, path: Path) -> None:
|
||||
errors = []
|
||||
|
||||
for field in ("title", "album"):
|
||||
val = fm.get(field)
|
||||
if not val or not str(val).strip():
|
||||
errors.append(f"missing or empty '{field}'")
|
||||
|
||||
wiki = fm.get("wiki")
|
||||
if not isinstance(wiki, dict) or "publish" not in wiki:
|
||||
errors.append("missing 'wiki.publish'")
|
||||
elif not isinstance(wiki["publish"], bool):
|
||||
errors.append("'wiki.publish' must be a boolean (true/false), not a string")
|
||||
|
||||
if "release_date" in fm and fm["release_date"]:
|
||||
try:
|
||||
datetime.strptime(str(fm["release_date"]), "%Y-%m-%d")
|
||||
except ValueError:
|
||||
errors.append("'release_date' must be YYYY-MM-DD")
|
||||
|
||||
if errors:
|
||||
raise ValueError(f"{path}: " + "; ".join(errors))
|
||||
|
||||
|
||||
def strip_first_h1(text: str) -> str:
|
||||
"""Remove the first '# Heading' line and any immediately following blank line."""
|
||||
lines = text.split("\n")
|
||||
for i, line in enumerate(lines):
|
||||
if line.strip().startswith("# "):
|
||||
del lines[i]
|
||||
if i < len(lines) and lines[i].strip() == "":
|
||||
del lines[i]
|
||||
break
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def markdown_to_wikitext(body: str) -> str:
|
||||
result = subprocess.run(
|
||||
["pandoc", "-f", "markdown", "-t", "mediawiki"],
|
||||
input=body,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"pandoc failed: {result.stderr.strip()}")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def build_wikitext(fm: dict, body_wikitext: str, source_url: str, source_ref: str) -> str:
|
||||
banner = f"{{{{Auto-generated|source={source_url}|commit={source_ref}}}}}"
|
||||
category = f"[[Category:{fm['album']}]]"
|
||||
return f"{banner}\n\n{body_wikitext}\n{category}\n"
|
||||
|
||||
|
||||
def connect_wiki() -> mwclient.Site:
|
||||
from urllib.parse import urlparse
|
||||
|
||||
api_url = os.environ["WIKI_API_URL"]
|
||||
parsed = urlparse(api_url)
|
||||
host = parsed.netloc
|
||||
# Strip api.php to get the wiki root path (e.g. "/" or "/w/")
|
||||
path = parsed.path[: parsed.path.rfind("/") + 1] or "/"
|
||||
site = mwclient.Site(host, path=path, scheme=parsed.scheme)
|
||||
site.login(os.environ["WIKI_BOT_USER"], os.environ["WIKI_BOT_PASSWORD"])
|
||||
return site
|
||||
|
||||
|
||||
def publish_one(rel: Path, post, site: mwclient.Site, source_ref: str, gitea_repo_url: str) -> str:
|
||||
"""Returns 'published', 'noop', or 'skipped'."""
|
||||
fm = post.metadata
|
||||
|
||||
if not fm["wiki"]["publish"]:
|
||||
return "skipped"
|
||||
|
||||
body_wikitext = markdown_to_wikitext(strip_first_h1(post.content))
|
||||
source_url = f"{gitea_repo_url}/src/commit/{source_ref}/{rel}"
|
||||
page_content = build_wikitext(fm, body_wikitext, source_url, source_ref)
|
||||
|
||||
page = site.pages[fm["title"]]
|
||||
if page.text() == page_content:
|
||||
return "noop"
|
||||
|
||||
ref_short = source_ref[:8] if source_ref else "unknown"
|
||||
page.save(page_content, summary=f"Auto-published from {ref_short} (songs pipeline)")
|
||||
return "published"
|
||||
|
||||
|
||||
def load_and_validate(files: list[Path], source_dir: Path) -> dict[Path, object]:
|
||||
posts = {}
|
||||
titles: dict[str, Path] = {}
|
||||
|
||||
for path in files:
|
||||
if not path.exists():
|
||||
print(f" warn: {path} not found, skipping", file=sys.stderr)
|
||||
continue
|
||||
|
||||
post = frontmatter.load(str(path))
|
||||
validate(post.metadata, path)
|
||||
|
||||
if post.metadata["wiki"]["publish"]:
|
||||
title = str(post.metadata["title"])
|
||||
if title in titles:
|
||||
raise ValueError(
|
||||
f"Duplicate wiki title '{title}': {path} and {titles[title]}"
|
||||
)
|
||||
titles[title] = path
|
||||
|
||||
posts[path] = post
|
||||
|
||||
return posts
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Publish song .md files to MediaWiki.")
|
||||
parser.add_argument("--source-dir", required=True, type=Path)
|
||||
parser.add_argument("--files", nargs="*", default=[], help="Relative paths within source-dir")
|
||||
parser.add_argument("--all", action="store_true", help="Publish all publishable files")
|
||||
args = parser.parse_args()
|
||||
|
||||
source_dir = args.source_dir.resolve()
|
||||
source_ref = os.environ.get("SOURCE_REF", "")
|
||||
gitea_repo_url = os.environ.get("GITEA_REPO_URL", "").rstrip("/")
|
||||
|
||||
if args.all:
|
||||
candidates = list(source_dir.rglob("*.md"))
|
||||
elif args.files:
|
||||
candidates = [source_dir / f for f in args.files]
|
||||
else:
|
||||
print("Nothing to do: pass --files or --all.")
|
||||
return
|
||||
|
||||
files = [
|
||||
p for p in candidates
|
||||
if p.exists()
|
||||
and p.suffix == ".md"
|
||||
and "wip" not in p.relative_to(source_dir).parts
|
||||
and p.name != "README.md"
|
||||
]
|
||||
|
||||
if not files:
|
||||
print("No publishable candidates after filtering.")
|
||||
return
|
||||
|
||||
print(f"Validating {len(files)} file(s)...")
|
||||
posts = load_and_validate(files, source_dir)
|
||||
print("Validation passed.")
|
||||
|
||||
site = connect_wiki()
|
||||
|
||||
counts = {"published": 0, "noop": 0, "skipped": 0}
|
||||
for abs_path, post in posts.items():
|
||||
rel = abs_path.relative_to(source_dir)
|
||||
outcome = publish_one(rel, post, site, source_ref, gitea_repo_url)
|
||||
counts[outcome] += 1
|
||||
print(f" [{outcome}] {rel}")
|
||||
|
||||
print(
|
||||
f"\nDone: {counts['published']} published, "
|
||||
f"{counts['noop']} unchanged, "
|
||||
f"{counts['skipped']} skipped (wiki.publish=false)."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e:
|
||||
print(f"ERROR: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,3 @@
|
||||
python-frontmatter>=1.1.0
|
||||
mwclient>=0.10.1
|
||||
PyYAML>=6.0
|
||||
@@ -0,0 +1,9 @@
|
||||
{{#if:{{{source|}}}|<div style="background:#f8f9fa;border:1px solid #a2a9b1;padding:0.4em 0.8em;margin-bottom:1em;font-size:0.85em;color:#54595d;">⚠️ This page is automatically published from source. Manual edits will be overwritten on the next pipeline run. • Source: [{{{source}}} view on Gitea] • Generated: {{{generated_at}}}</div>}}
|
||||
<noinclude>
|
||||
== Usage ==
|
||||
Applied automatically by the songs pipeline. Parameters:
|
||||
* <code>source</code> — URL to the source file on Gitea
|
||||
* <code>generated_at</code> — ISO 8601 timestamp of the publishing run
|
||||
|
||||
[[Category:Auto-generated templates]]
|
||||
</noinclude>
|
||||
Reference in New Issue
Block a user