libretro/wiki/release-process.md

# Release Process

This page documents the CI/CD pipeline: what each workflow does, how releases
are built, and how to run the process manually.

## CI workflows overview

The project uses 3 GitHub Actions workflows. All use only official GitHub
actions (`actions/checkout`, `actions/setup-python`, `actions/upload-pages-artifact`,
`actions/deploy-pages`). No third-party actions.

Budget target: ~175 minutes/month on the GitHub free tier.

| Workflow | File | Trigger |
|----------|------|---------|
| Build & Release | `build.yml` | Manual dispatch only |
| Deploy Site | `deploy-site.yml` | Push to main (platforms, emulators, provenance, wiki, scripts, database.json, mkdocs.yml), manual |
| PR Validation | `validate.yml` | PR touching `bios/**`, `platforms/**` or `emulators/**` |

Upstream BIOS lists are not scraped on a schedule. A maintainer runs the
scrapers by hand (see [adding a scraper](adding-a-scraper.md)), reviews the
diff, and commits the refreshed platform YAML.

## build.yml - Build & Release

Releasing is deliberate. Pushing never cuts one: somebody decides the
collection is worth publishing and dispatches the workflow.

**Trigger.** `workflow_dispatch` only, with an optional `force_release` flag to
bypass the rate limit.

**Concurrency.** Group `build`, queued rather than cancelled: a run that is
already uploading assets must finish.

**Steps:**

1. Checkout, Python 3.12, install `pyyaml`
2. Run `test_e2e`
3. Rate limit check: skip if last release was less than 7 days ago (unless
   `force_release` is set)
4. Restore large files from the `large-files` release into `.cache/large/`
5. Refresh data directories (`refresh_data_dirs.py`)
6. Build packs (`generate_pack.py --all --output-dir dist/`)
7. Split any pack over 2 GB into `.zip.001`, `.zip.002`, ... volumes. GitHub
   caps release assets at 2 GB, and the `.001` convention is what 7-Zip and
   PeaZip open directly (they reject `.partNN` names as corrupt)
8. Create GitHub release with tag `v{YYYY.MM.DD}` (appends `.N` suffix if
   a same-day release already exists)
9. Clean up old releases, keeping the 3 most recent plus `large-files`

**Release notes** include file count, total size, per-pack sizes, the extract
path per platform, and the last 15 non-merge commits touching `bios/` or
`platforms/`.

**Pack variants.** The workflow builds full packs only. Releases that also
carry `*_Platform_BIOS_Pack.zip` had a second `--source platform` run added by
hand, as in the manual process below. See
[advanced usage](advanced-usage.md#pack-source-variants).

## deploy-site.yml - Deploy Documentation Site

**Trigger.** Push to `main` when any of these paths change: `platforms/`,
`emulators/`, `provenance/`, `wiki/`, `scripts/generate_site.py`,
`scripts/generate_readme.py`, `scripts/verify.py`, `scripts/common.py`,
`database.json`, `mkdocs.yml`. Also manual dispatch.

The list is the set of inputs the site is generated from. Adding a new input to
`generate_site.py` means adding its path here, or the site silently goes stale.

**Steps:**

1. Checkout, Python 3.12
2. Install `pyyaml`, `mkdocs-material>=9.7.5,<10`, `pymdown-extensions>=10.14`
3. Restore large files from the `large-files` release, refresh data directories
4. Run `generate_site.py` (converts YAML data into MkDocs pages and rewrites
   `mkdocs.yml`)
5. Run `generate_readme.py` (rebuilds README.md and CONTRIBUTING.md)
6. `mkdocs build --strict` to produce the static site
7. Run `validate_site.py` on the rendered HTML (metadata, headings, image
   alternatives, duplicate ids, local links and fragments)
8. Require the committed README and CONTRIBUTING to match what the generator
   just produced. `write_if_changed()` compares content with the timestamp line
   stripped, so a run that only moves the clock leaves the files untouched and
   the check stays meaningful
9. Upload artifact, deploy to GitHub Pages

Data contracts are validated with `scripts/validate_schemas.py` before the site
is generated: `database.json`, the install and target manifests, the site API
envelopes and the stats file, plus the semantic invariants those schemas cannot
express (declared totals matching their lists, no destination both installed
and omitted).

The site is deployed via the `github-pages` environment using the official
`actions/deploy-pages` action. Pages deployments are queued rather than
cancelled (`cancel-in-progress: false`): cancelling one mid-flight leaves the
deployment stuck and the next runs time out waiting on it.

`--strict` turns MkDocs warnings into failures, so a broken internal link or a
dangling anchor fails the build instead of shipping. The `validation:` block in
`mkdocs.yml` is what promotes unrecognized links and missing anchors to
warnings in the first place.

The theme version is pinned on both sides: `>=9.7.5` because that is the
release which caps `mkdocs < 2` (MkDocs 2.0 ships without a license), `<10`
so a major theme release cannot change the site without a deliberate bump.

## validate.yml - PR Validation

**Trigger.** Pull requests that modify `bios/**`, `platforms/**`,
`emulators/**`, `schemas/**`, `scripts/**`, `tests/**` or `install.py`.

**Concurrency.** Per-PR group, cancel in-progress.

Four parallel jobs:

**validate-bios.** Diffs the PR to find changed BIOS files, runs
`validate_pr.py --markdown` on each, and posts the validation report as a PR
comment (hash verification, database match status).

**validate-configs.** Runs `python scripts/validate_schemas.py --source-only`,
which validates every platform YAML against `schemas/platform.schema.json` and
every emulator profile against `schemas/emulator.schema.json`. Both schemas set
`additionalProperties: false`, so a typo in a field name fails the job instead
of being silently ignored.

**run-tests.** Runs `python -m unittest discover tests -v`. Must pass before
merge.

**label-pr.** Auto-labels the PR based on changed paths:

| Path pattern | Label |
|-------------|-------|
| `bios/` | `bios` |
| `bios/{Manufacturer}/` | `system:{manufacturer}` |
| `platforms/` | `platform-config` |
| `scripts/` | `automation` |

## Large files management

Files larger than 50 MB are stored as assets on a permanent GitHub release
named `large-files` (to keep the git repository lightweight).

Examples: PS3UPDAT.PUP, PSVUPDAT.PUP, PSP2UPDAT.PUP, the DSi NAND images,
maclc3.zip, Firmware.19.0.0.zip (Switch), the QEMU EDK2 firmware, the ScummVM
data bundle, the EasyRPG soundfont, the Dolphin/Ishiiruka SD card images, and
the arcade sets over 100 MB. `.gitignore` is the authoritative list: every
`bios/` path listed there is a release asset.

**Storage.** Listed in `.gitignore` so they stay out of git history. The
`large-files` release is excluded from cleanup (the build workflow only
deletes version-tagged releases).

**Build-time restore.** The build workflow downloads all assets from
`large-files` into `.cache/large/` and copies them to their expected paths
before pack generation.

**Upload.** To add or update a large file:

```bash
gh release upload large-files "bios/Sony/PS3/PS3UPDAT.PUP#PS3UPDAT.PUP"
```

**Local cache.** `generate_pack.py` calls `fetch_large_file()` which downloads
from the release and caches in `.cache/large/` for subsequent runs.

## Manual release process

When `build.yml` is disabled, build and release manually:

```bash
# Run the full pipeline (DB + verify + packs + manifests + integrity + docs)
python scripts/pipeline.py

# Or step by step:
python scripts/generate_db.py --force --bios-dir bios --output database.json
python scripts/verify.py --all
python scripts/generate_pack.py --all --output-dir dist/                    # full packs
python scripts/generate_pack.py --all --source platform --output-dir dist/  # platform packs
python scripts/generate_pack.py --all --verify-packs --output-dir dist/

# Split anything over 2 GB (GitHub asset cap)
for f in dist/*.zip; do
  [ "$(stat -c%s "$f")" -gt 2000000000 ] || continue
  split --bytes=1900M --numeric-suffixes=1 --suffix-length=3 "$f" "$f." && rm "$f"
done

# Create the release
DATE=$(date +%Y.%m.%d)
gh release create "v${DATE}" dist/*.zip* \
  --title "BIOS Pack v${DATE}" \
  --notes "Release notes here" \
  --latest
```

The two `generate_pack.py` runs are what puts both `*_BIOS_Pack.zip` and
`*_Platform_BIOS_Pack.zip` on the release. `--all-variants` builds all six
combinations instead, which is more than a release needs.

Run the pipeline online for a release: `--offline` skips the data directory
refresh and the MAME/FBNeo hash refresh, so the packs would ship stale data
directories.

The workflow carries no push trigger, so there is nothing to disable and no
guard to remove. Cutting a release means dispatching `build.yml` from the
Actions tab, or:

```bash
gh workflow run build.yml
gh workflow run build.yml -f force_release=true   # within 7 days of the last
```

The rate limit refuses a second release inside seven days unless
`force_release` is set, which is what keeps a stray dispatch from publishing
twice in a day.