docs: update all docs to reflect v1 live state

- template_auto_generated.wikitext: generated_at → commit parameter - SCHEMA.md: same banner fix; clarify that frontmatter-less files are silently skipped, not errors - pipelines/songs/README.md: full rewrite — v1 live status, self- contained workflow pattern, correct env table, modes, one-off setup marked done, full decisions log - README.md: architecture reflects actual pattern (no workflow_call), URL_TO_GITEA variable added, setup checklist marked complete, host network mode documented Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-09 18:39:13 +09:00
parent cf6567cd77
commit afc6a62791
4 changed files with 62 additions and 54 deletions
@@ -12,8 +12,10 @@ Implemented:
 - YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
 - Pandoc-based markdown → wikitext body rendering
 - First-line h1 stripping
- Auto-generated banner + album category injection
- MediaWiki bot API write with idempotency (no-op edits skipped)
+- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`) + album category injection
+- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
+- Files without `wiki.publish: true` silently skipped
+- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot

 Deferred (schema reserves the fields; renderer doesn't yet act on them):

@@ -35,41 +37,39 @@ Source repo: `mikkeli/ncmr-songs`. The pipeline triggers on push to `master`.

 ## Invocation

-The pipeline is invoked via a reusable Gitea Actions workflow (`.gitea/workflows/publish-songs.yml` in this repo). The `ncmr-songs` repo holds a thin stub workflow (`.gitea/workflows/publish.yml`) that calls into this one on push to `master`.
+The pipeline runs as a self-contained Gitea Actions workflow in `mikkeli/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.

-### Workflow inputs and secrets
+Triggers:
+- `push` to `master` (path-filtered as above)
+- `workflow_dispatch` — manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit
+
+### Environment (secrets and variables)

 | Name | Kind | Source | Purpose |
 |---|---|---|---|
 | `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
-| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL (used in the banner link) |
-| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login |
+| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
+| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Dubrowski@giteaAutomaton`) |
 | `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
-| `SOURCE_DIR` | workflow input | from the stub | Path to checked-out `ncmr-songs` working tree |
-| `SOURCE_REF` | workflow input | from the stub | The pushed-to commit SHA |
-| `SOURCE_BASE_REF` | workflow input | from the stub | The SHA prior to the push (for diff-based change detection) |
+| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Gitea instance base URL for cloning |
+| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
+| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary |
+| `GITEA_REPO_URL` | env (set in workflow) | composed from vars | Source-file URL base for the banner link |

 ## Dependencies

- Python 3.12+
- Pandoc 2.x or newer (apt-installable in the runner container)
- `mwclient` or `requests` for the MediaWiki API
- `PyYAML` for frontmatter parsing
- `python-frontmatter` (convenience wrapper around PyYAML for markdown frontmatter)
-
-`requirements.txt` will be added when the implementation lands.
-
-## One-off wiki setup
-
-Before the first run, two things must exist on the wiki:
-
-1. **`Template:Auto-generated`** — the banner injected at the top of every auto-published page. Wikitext for this template is included in the implementation step (not yet written).
-2. **The bot user has edit rights** for the namespace(s) the pipeline writes to. The default main namespace is fine; verify by attempting a manual edit via the bot account before relying on the pipeline.
+- Python 3.12+ (job container: `python:3.12-slim`)
+- Pandoc (apt-installed in the job container)
+- `python-frontmatter`, `mwclient`, `PyYAML` — see `requirements.txt`

 ## Modes

- **Incremental** (default, triggered by push to `master`): publishes only files changed between `SOURCE_BASE_REF` and `SOURCE_REF`.
- **Full** (`--all` flag, triggered by `workflow_dispatch`): re-publishes every publishable file in the source tree. Use after template or renderer changes.
+The pipeline always runs `--all`: every file with `wiki.publish: true` (not under `wip/`) is checked against the current wiki content. Writes only happen when content has changed (commit SHA in the banner drives idempotency). This is appropriate for the current repo size.
+
+## One-off wiki setup
+
+- [x] **`Template:Auto-generated`** — exists on the wiki. Wikitext source: `template_auto_generated.wikitext` in this directory.
+- [x] **Bot user in `bot` group** — grants `skipcaptcha`; verified working.

 ## Decisions log (songs pipeline)

@@ -79,11 +79,13 @@ Before the first run, two things must exist on the wiki:
 | Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
 | Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 |
 | Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
-| Album landing pages | Auto-injected `[[Category:<album>]]`; the wiki category page is user-written and not overwritten by the pipeline | 2026-06-09 |
+| Album landing pages | Auto-injected `[[Category:<album>]]`; the wiki category page is user-written and not overwritten | 2026-06-09 |
 | Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
+| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
+| Publish mode | Always `--all`; no diff detection — small repo, simpler than fragile git-diff gating | 2026-06-09 |
+| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 |
+| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 |

 ## Status

-Schema and pipeline overview committed. No executable code yet.
-
-**Next**: implement renderer (Python module), workflow YAMLs (this repo + `ncmr-songs` stub), `Template:Auto-generated` wikitext, and end-to-end dry-run on `ses/pulse.md`.
+v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org.
@@ -94,7 +94,7 @@ For each `.md` with `wiki.publish: true` and not under `wip/`:
 3. Replace `[[PLACEHOLDER]]` markers with safe tokens. *(Deferred v1; no-op until siblings are implemented.)*
 4. Pipe the body through `pandoc -f markdown -t mediawiki`.
 5. Re-substitute sibling tokens with resolved `[[Page|Display]]` wikilinks. *(Deferred v1.)*
-6. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|generated_at=<ISO timestamp>}}`.
+6. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
 7. Append the album category: `[[Category:<album>]]`.
 8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
 9. Otherwise, write the page with an edit summary referencing the source commit.
@@ -164,14 +164,12 @@ wiki:

 ## Validation errors the renderer must produce

-The pipeline fails fast and loud on the following:
+Files without a `wiki.publish: true` frontmatter field are silently skipped — no error. The pipeline fails fast and loud only on files that are opted in (`wiki.publish: true`) with invalid data:

- Missing or unparseable frontmatter block
- Missing required field (`title`, `album`, `wiki.publish`)
- `title` is empty or whitespace-only
- `wiki.publish` is not a boolean
+- Missing required field (`title` or `album`) on a file with `wiki.publish: true`
+- `wiki.publish` present but not a boolean
 - `release_date` (if present) does not parse as YYYY-MM-DD
 - `siblings` (if present) references a file that doesn't exist
- Two source files declare the same `title` (page-name collision)
+- Two publishable source files declare the same `title` (page-name collision)

 Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched.
@@ -1,9 +1,9 @@
-{{#if:{{{source|}}}|<div style="background:#f8f9fa;border:1px solid #a2a9b1;padding:0.4em 0.8em;margin-bottom:1em;font-size:0.85em;color:#54595d;">&#x26A0;&#xFE0F; This page is automatically published from source. Manual edits will be overwritten on the next pipeline run. &nbsp;&bull;&nbsp; Source: [{{{source}}} view on Gitea] &nbsp;&bull;&nbsp; Generated: {{{generated_at}}}</div>}}
+{{#if:{{{source|}}}|<div style="background:#f8f9fa;border:1px solid #a2a9b1;padding:0.4em 0.8em;margin-bottom:1em;font-size:0.85em;color:#54595d;">&#x26A0;&#xFE0F; This page is automatically published from source. Manual edits will be overwritten on the next pipeline run. &nbsp;&bull;&nbsp; Source: [{{{source}}} view on Gitea] &nbsp;&bull;&nbsp; Commit: {{{commit}}}</div>}}
 <noinclude>
 == Usage ==
 Applied automatically by the songs pipeline. Parameters:
-* <code>source</code> — URL to the source file on Gitea
-* <code>generated_at</code> — ISO 8601 timestamp of the publishing run
+* <code>source</code> — URL to the source file on Gitea at the publishing commit
+* <code>commit</code> — commit SHA of the source that produced this page

 [[Category:Auto-generated templates]]
 </noinclude>