diff --git a/README.md b/README.md index adeef8e..53b3b9c 100644 --- a/README.md +++ b/README.md @@ -6,24 +6,26 @@ CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from u | Path | Source repo | Purpose | Status | |---|---|---|---| -| [`pipelines/songs/`](pipelines/songs/) | `mikkeli/ncmr-songs` | Song lyric pages | in development (v1, no executable code yet) | +| [`pipelines/songs/`](pipelines/songs/) | `mikkeli/ncmr-songs` | Song lyric pages | v1 live | Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime, decisions). This root README covers only what's cross-cutting. ## Architecture -Hybrid layout. The Gitea Actions trigger has to live in the source repo (Gitea only fires workflows from `.gitea/workflows/` of the pushed-to repo); the real logic lives here. Source-repo workflows are thin stubs that call reusable workflows defined here. +Hybrid layout. The Gitea Actions trigger must live in the source repo (Gitea only fires workflows from `.gitea/workflows/` of the pushed-to repo); the rendering logic lives here. The source-repo workflow is self-contained but clones this repo at runtime to get the renderer. ``` / - .gitea/workflows/.yml <- thin stub, calls into this repo + .gitea/workflows/.yml <- workflow: installs deps, clones this repo, runs renderer novoyuuparosk-auto-wiki/ <- this repo - .gitea/workflows/.yml <- reusable workflows (the actual logic) + .gitea/workflows/.yml <- reusable workflow stubs (kept for reference; not actively called) pipelines// <- per-pipeline code, schema, templates lib/ <- shared modules (MediaWiki client, etc.) ``` +Note: `workflow_call` across private repos was abandoned — the auto-generated run token is scoped to the triggering repo only and cannot clone a private callee. The source-repo workflow clones this repo directly using `FAPAT`. + ## Wiki - Base URL: https://wiki.novoyuuparosk.org @@ -44,6 +46,8 @@ One `act_runner` instance serves all pipelines. Runs on a Pi 5 (Raspberry Pi OS `act_runner` build: `linux-arm64`, from the `gitea/act_runner` Docker image. +Container network mode: `host` — required so job containers can reach `localhost:3005` (Gitea) and resolve mDNS hostnames. + ## Gitea Actions setup (cross-cutting) Secrets and variables are scoped to user `mikkeli` (no orgs on this instance), inherited by all repos under that account. @@ -51,11 +55,12 @@ Secrets and variables are scoped to user `mikkeli` (no orgs on this instance), i **Secrets:** - `WIKI_BOT_USER` = `Dubrowski@giteaAutomaton` - `WIKI_BOT_PASSWORD` = the value from *Bot identity* above -- `FAPAT` = Full-Access PAT under `mikkeli`, used by source-repo shim workflows to clone this repo at workflow time +- `FAPAT` = Full-Access PAT under `mikkeli`, used by source-repo workflows to clone this repo at runtime **Variables:** - `WIKI_BASE_URL` = `https://wiki.novoyuuparosk.org` - `WIKI_API_URL` = `https://wiki.novoyuuparosk.org/api.php` +- `URL_TO_GITEA` = Gitea instance base URL (e.g. `http://localhost:3005`). Named with `URL_TO_` prefix — Gitea blocks variable names starting with `GITEA_` or `GITHUB_`. ## Branch naming @@ -66,12 +71,14 @@ Secrets and variables are scoped to user `mikkeli` (no orgs on this instance), i | Decision | Value | Date | |---|---|---| -| Architecture | Hybrid: stub in source repo, logic in this repo via reusable workflows | 2026-06-09 | -| Workflow pattern | Gitea reusable workflows (`workflow_call`); requires Gitea ≥ 1.20 — confirmed 1.25+ | 2026-06-09 | +| Architecture | Hybrid: source-repo workflow clones this repo at runtime for the renderer | 2026-06-09 | +| Workflow pattern | Self-contained (not `workflow_call`) — cross-repo `workflow_call` blocked by token scoping on private repos | 2026-06-09 | | Runner execution | Docker, added as a service to the existing Gitea docker-compose | 2026-06-09 | +| Runner network mode | `host` — job containers need to reach Gitea on localhost | 2026-06-09 | | Secret/runner scope | User-level on `mikkeli` (no orgs on this instance) | 2026-06-09 | | MediaWiki API path | `api.php` (classic action API) | 2026-06-09 | | Branch naming (this repo) | `automation/` for pipeline-development branches | 2026-06-09 | +| Variable naming | `URL_TO_GITEA` not `GITEA_URL` — Gitea blocks `GITEA_`/`GITHUB_` prefixes | 2026-06-09 | Per-pipeline decisions live in each pipeline's README. @@ -79,16 +86,17 @@ Per-pipeline decisions live in each pipeline's README. Via the Gitea web UI logged in as `mikkeli`: -- [v] User-scoped secrets and variables set per *Gitea Actions setup* above - - [v] `WIKI_BOT_USER` - - [v] `WIKI_BOT_PASSWORD` - - [v] `FAPAT` (Full-Access PAT — value not stored in this README; saved directly into the Gitea secret. Regenerate if lost.) - - [v] `WIKI_BASE_URL` - - [v] `WIKI_API_URL` +- [x] User-scoped secrets and variables set per *Gitea Actions setup* above + - [x] `WIKI_BOT_USER` + - [x] `WIKI_BOT_PASSWORD` + - [x] `FAPAT` (Full-Access PAT — value not stored in this README; saved directly into the Gitea secret. Regenerate if lost.) + - [x] `WIKI_BASE_URL` + - [x] `WIKI_API_URL` + - [x] `URL_TO_GITEA` With Pi access: -- [ ] Add `act_runner` service to the existing Gitea docker-compose -- [ ] Generate a runner registration token at `/-/admin/actions/runners` (or `/user/settings/actions/runners` if user-scoped runners are exposed), bake into the compose env, `docker compose up -d act_runner`, confirm "online" in the Gitea UI +- [x] Add `act_runner` service to the existing Gitea docker-compose +- [x] Generate a runner registration token at `/-/admin/actions/runners`, bake into the compose env, `docker compose up -d act_runner`, confirm "online" in the Gitea UI Per-pipeline setup lives in each pipeline's README. Start with [`pipelines/songs/`](pipelines/songs/). diff --git a/pipelines/songs/README.md b/pipelines/songs/README.md index 5c0af42..6c831cf 100644 --- a/pipelines/songs/README.md +++ b/pipelines/songs/README.md @@ -12,8 +12,10 @@ Implemented: - YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md) - Pandoc-based markdown → wikitext body rendering - First-line h1 stripping -- Auto-generated banner + album category injection -- MediaWiki bot API write with idempotency (no-op edits skipped) +- Auto-generated banner (`{{Auto-generated|source=...|commit=}}`) + album category injection +- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output +- Files without `wiki.publish: true` silently skipped +- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot Deferred (schema reserves the fields; renderer doesn't yet act on them): @@ -35,41 +37,39 @@ Source repo: `mikkeli/ncmr-songs`. The pipeline triggers on push to `master`. ## Invocation -The pipeline is invoked via a reusable Gitea Actions workflow (`.gitea/workflows/publish-songs.yml` in this repo). The `ncmr-songs` repo holds a thin stub workflow (`.gitea/workflows/publish.yml`) that calls into this one on push to `master`. +The pipeline runs as a self-contained Gitea Actions workflow in `mikkeli/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer. -### Workflow inputs and secrets +Triggers: +- `push` to `master` (path-filtered as above) +- `workflow_dispatch` — manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit + +### Environment (secrets and variables) | Name | Kind | Source | Purpose | |---|---|---|---| | `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint | -| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL (used in the banner link) | -| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login | +| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL | +| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Dubrowski@giteaAutomaton`) | | `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value | -| `SOURCE_DIR` | workflow input | from the stub | Path to checked-out `ncmr-songs` working tree | -| `SOURCE_REF` | workflow input | from the stub | The pushed-to commit SHA | -| `SOURCE_BASE_REF` | workflow input | from the stub | The SHA prior to the push (for diff-based change detection) | +| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Gitea instance base URL for cloning | +| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos | +| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary | +| `GITEA_REPO_URL` | env (set in workflow) | composed from vars | Source-file URL base for the banner link | ## Dependencies -- Python 3.12+ -- Pandoc 2.x or newer (apt-installable in the runner container) -- `mwclient` or `requests` for the MediaWiki API -- `PyYAML` for frontmatter parsing -- `python-frontmatter` (convenience wrapper around PyYAML for markdown frontmatter) - -`requirements.txt` will be added when the implementation lands. - -## One-off wiki setup - -Before the first run, two things must exist on the wiki: - -1. **`Template:Auto-generated`** — the banner injected at the top of every auto-published page. Wikitext for this template is included in the implementation step (not yet written). -2. **The bot user has edit rights** for the namespace(s) the pipeline writes to. The default main namespace is fine; verify by attempting a manual edit via the bot account before relying on the pipeline. +- Python 3.12+ (job container: `python:3.12-slim`) +- Pandoc (apt-installed in the job container) +- `python-frontmatter`, `mwclient`, `PyYAML` — see `requirements.txt` ## Modes -- **Incremental** (default, triggered by push to `master`): publishes only files changed between `SOURCE_BASE_REF` and `SOURCE_REF`. -- **Full** (`--all` flag, triggered by `workflow_dispatch`): re-publishes every publishable file in the source tree. Use after template or renderer changes. +The pipeline always runs `--all`: every file with `wiki.publish: true` (not under `wip/`) is checked against the current wiki content. Writes only happen when content has changed (commit SHA in the banner drives idempotency). This is appropriate for the current repo size. + +## One-off wiki setup + +- [x] **`Template:Auto-generated`** — exists on the wiki. Wikitext source: `template_auto_generated.wikitext` in this directory. +- [x] **Bot user in `bot` group** — grants `skipcaptcha`; verified working. ## Decisions log (songs pipeline) @@ -79,11 +79,13 @@ Before the first run, two things must exist on the wiki: | Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 | | Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 | | Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 | -| Album landing pages | Auto-injected `[[Category:]]`; the wiki category page is user-written and not overwritten by the pipeline | 2026-06-09 | +| Album landing pages | Auto-injected `[[Category:]]`; the wiki category page is user-written and not overwritten | 2026-06-09 | | Source-repo branch convention | `autowiki/` short-lived branches in `ncmr-songs` | 2026-06-09 | +| Idempotency | Commit SHA in banner (`commit=`), not timestamp — stable across identical re-runs | 2026-06-09 | +| Publish mode | Always `--all`; no diff detection — small repo, simpler than fragile git-diff gating | 2026-06-09 | +| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 | +| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 | ## Status -Schema and pipeline overview committed. No executable code yet. - -**Next**: implement renderer (Python module), workflow YAMLs (this repo + `ncmr-songs` stub), `Template:Auto-generated` wikitext, and end-to-end dry-run on `ses/pulse.md`. +v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org. diff --git a/pipelines/songs/SCHEMA.md b/pipelines/songs/SCHEMA.md index dd4d84d..a5865f0 100644 --- a/pipelines/songs/SCHEMA.md +++ b/pipelines/songs/SCHEMA.md @@ -94,7 +94,7 @@ For each `.md` with `wiki.publish: true` and not under `wip/`: 3. Replace `[[PLACEHOLDER]]` markers with safe tokens. *(Deferred v1; no-op until siblings are implemented.)* 4. Pipe the body through `pandoc -f markdown -t mediawiki`. 5. Re-substitute sibling tokens with resolved `[[Page|Display]]` wikilinks. *(Deferred v1.)* -6. Prepend the auto-generated banner: `{{Auto-generated|source=|generated_at=}}`. +6. Prepend the auto-generated banner: `{{Auto-generated|source=|commit=}}`. 7. Append the album category: `[[Category:]]`. 8. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean). 9. Otherwise, write the page with an edit summary referencing the source commit. @@ -164,14 +164,12 @@ wiki: ## Validation errors the renderer must produce -The pipeline fails fast and loud on the following: +Files without a `wiki.publish: true` frontmatter field are silently skipped — no error. The pipeline fails fast and loud only on files that are opted in (`wiki.publish: true`) with invalid data: -- Missing or unparseable frontmatter block -- Missing required field (`title`, `album`, `wiki.publish`) -- `title` is empty or whitespace-only -- `wiki.publish` is not a boolean +- Missing required field (`title` or `album`) on a file with `wiki.publish: true` +- `wiki.publish` present but not a boolean - `release_date` (if present) does not parse as YYYY-MM-DD - `siblings` (if present) references a file that doesn't exist -- Two source files declare the same `title` (page-name collision) +- Two publishable source files declare the same `title` (page-name collision) Validation runs before any wiki API calls. A failed validation aborts the run with a non-zero exit and leaves the wiki untouched. diff --git a/pipelines/songs/template_auto_generated.wikitext b/pipelines/songs/template_auto_generated.wikitext index e7706b5..b9a364d 100644 --- a/pipelines/songs/template_auto_generated.wikitext +++ b/pipelines/songs/template_auto_generated.wikitext @@ -1,9 +1,9 @@ -{{#if:{{{source|}}}|
⚠️ This page is automatically published from source. Manual edits will be overwritten on the next pipeline run.  •  Source: [{{{source}}} view on Gitea]  •  Generated: {{{generated_at}}}
}} +{{#if:{{{source|}}}|
⚠️ This page is automatically published from source. Manual edits will be overwritten on the next pipeline run.  •  Source: [{{{source}}} view on Gitea]  •  Commit: {{{commit}}}
}} == Usage == Applied automatically by the songs pipeline. Parameters: -* source — URL to the source file on Gitea -* generated_at — ISO 8601 timestamp of the publishing run +* source — URL to the source file on Gitea at the publishing commit +* commit — commit SHA of the source that produced this page [[Category:Auto-generated templates]]