mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-04-13 20:32:32 -05:00
docs: add wiki pages for all audiences, fix .old.yml leak
9 new wiki pages: getting-started, faq, troubleshooting, advanced-usage, verification-modes, adding-a-platform, adding-a-scraper, testing-guide, release-process. Updated architecture.md with mermaid diagrams, tools.md with full pipeline and target/exporter sections, profiling.md with missing fields, index.md with glossary and nav links. Expanded CONTRIBUTING.md from stub to full contributor guide. Filter .old.yml from load_emulator_profiles, generate_db alias collection, and generate_readme counts. Fix BizHawk sha1 mode in tools.md, fix RetroPie path, fix export_truth.py typos.
This commit is contained in:
Abdessamad Derraz
158
wiki/release-process.md
Normal file
158
wiki/release-process.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# Release Process
|
||||
|
||||
This page documents the CI/CD pipeline: what each workflow does, how releases
|
||||
are built, and how to run the process manually.
|
||||
|
||||
## CI workflows overview
|
||||
|
||||
The project uses 4 GitHub Actions workflows. All use only official GitHub
|
||||
actions (`actions/checkout`, `actions/setup-python`, `actions/upload-pages-artifact`,
|
||||
`actions/deploy-pages`). No third-party actions.
|
||||
|
||||
Budget target: ~175 minutes/month on the GitHub free tier.
|
||||
|
||||
| Workflow | File | Trigger |
|
||||
|----------|------|---------|
|
||||
| Build & Release | `build.yml` | Push to `bios/**` or `platforms/**`, manual dispatch |
|
||||
| Deploy Site | `deploy-site.yml` | Push to main (platforms, emulators, wiki, scripts, database.json, mkdocs.yml), manual |
|
||||
| PR Validation | `validate.yml` | PR touching `bios/**` or `platforms/**` |
|
||||
| Weekly Sync | `watch.yml` | Cron Monday 06:00 UTC, manual dispatch |
|
||||
|
||||
## build.yml - Build & Release
|
||||
|
||||
Currently disabled (`if: false` on the release job) until pack generation is
|
||||
validated in production.
|
||||
|
||||
**Trigger.** Push to `main` on `bios/**` or `platforms/**` paths, or manual
|
||||
`workflow_dispatch` with optional `force_release` flag to bypass rate limiting.
|
||||
|
||||
**Concurrency.** Group `build`, cancel in-progress.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Checkout, Python 3.12, install `pyyaml`
|
||||
2. Run `test_e2e`
|
||||
3. Rate limit check: skip if last release was less than 7 days ago (unless
|
||||
`force_release` is set)
|
||||
4. Restore large files from the `large-files` release into `.cache/large/`
|
||||
5. Refresh data directories (`refresh_data_dirs.py`)
|
||||
6. Build packs (`generate_pack.py --all --output-dir dist/`)
|
||||
7. Create GitHub release with tag `v{YYYY.MM.DD}` (appends `.N` suffix if
|
||||
a same-day release already exists)
|
||||
8. Clean up old releases, keeping the 3 most recent plus `large-files`
|
||||
|
||||
**Release notes** include file count, total size, per-pack sizes, and the last
|
||||
15 non-merge commits touching `bios/` or `platforms/`.
|
||||
|
||||
## deploy-site.yml - Deploy Documentation Site
|
||||
|
||||
**Trigger.** Push to `main` when any of these paths change: `platforms/`,
|
||||
`emulators/`, `wiki/`, `scripts/generate_site.py`, `scripts/generate_readme.py`,
|
||||
`scripts/verify.py`, `scripts/common.py`, `database.json`, `mkdocs.yml`.
|
||||
Also manual dispatch.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Checkout, Python 3.12
|
||||
2. Install `pyyaml`, `mkdocs-material`, `pymdown-extensions`
|
||||
3. Run `generate_site.py` (converts YAML data into MkDocs pages)
|
||||
4. Run `generate_readme.py` (rebuilds README.md and CONTRIBUTING.md)
|
||||
5. `mkdocs build` to produce the static site
|
||||
6. Upload artifact, deploy to GitHub Pages
|
||||
|
||||
The site is deployed via the `github-pages` environment using the official
|
||||
`actions/deploy-pages` action.
|
||||
|
||||
## validate.yml - PR Validation
|
||||
|
||||
**Trigger.** Pull requests that modify `bios/**` or `platforms/**`.
|
||||
|
||||
**Concurrency.** Per-PR group, cancel in-progress.
|
||||
|
||||
Four parallel jobs:
|
||||
|
||||
**validate-bios.** Diffs the PR to find changed BIOS files, runs
|
||||
`validate_pr.py --markdown` on each, and posts the validation report as a PR
|
||||
comment (hash verification, database match status).
|
||||
|
||||
**validate-configs.** Validates all platform YAML files against
|
||||
`schemas/platform.schema.json` using `jsonschema`. Fails if any config does
|
||||
not match the schema.
|
||||
|
||||
**run-tests.** Runs `python -m unittest tests.test_e2e -v`. Must pass before
|
||||
merge.
|
||||
|
||||
**label-pr.** Auto-labels the PR based on changed paths:
|
||||
|
||||
| Path pattern | Label |
|
||||
|-------------|-------|
|
||||
| `bios/` | `bios` |
|
||||
| `bios/{Manufacturer}/` | `system:{manufacturer}` |
|
||||
| `platforms/` | `platform-config` |
|
||||
| `scripts/` | `automation` |
|
||||
|
||||
## watch.yml - Weekly Platform Sync
|
||||
|
||||
**Trigger.** Cron schedule every Monday at 06:00 UTC, or manual dispatch.
|
||||
|
||||
**Flow:**
|
||||
|
||||
1. Scrape live upstream sources (System.dat, batocera-systems, es_bios.xml,
|
||||
etc.) and regenerate platform YAML configs
|
||||
2. Auto-fetch missing BIOS files
|
||||
3. Refresh data directories
|
||||
4. Run dedup
|
||||
5. Regenerate `database.json`
|
||||
6. Create or update a PR with labels `automated` and `platform-update`
|
||||
|
||||
The PR contains all changes from the scrape cycle. A maintainer reviews and
|
||||
merges.
|
||||
|
||||
## Large files management
|
||||
|
||||
Files larger than 50 MB are stored as assets on a permanent GitHub release
|
||||
named `large-files` (to keep the git repository lightweight).
|
||||
|
||||
Known large files: PS3UPDAT.PUP, PSVUPDAT.PUP, PSP2UPDAT.PUP, dsi_nand.bin,
|
||||
maclc3.zip, Firmware.19.0.0.zip (Switch).
|
||||
|
||||
**Storage.** Listed in `.gitignore` so they stay out of git history. The
|
||||
`large-files` release is excluded from cleanup (the build workflow only
|
||||
deletes version-tagged releases).
|
||||
|
||||
**Build-time restore.** The build workflow downloads all assets from
|
||||
`large-files` into `.cache/large/` and copies them to their expected paths
|
||||
before pack generation.
|
||||
|
||||
**Upload.** To add or update a large file:
|
||||
|
||||
```bash
|
||||
gh release upload large-files "bios/Sony/PS3/PS3UPDAT.PUP#PS3UPDAT.PUP"
|
||||
```
|
||||
|
||||
**Local cache.** `generate_pack.py` calls `fetch_large_file()` which downloads
|
||||
from the release and caches in `.cache/large/` for subsequent runs.
|
||||
|
||||
## Manual release process
|
||||
|
||||
When `build.yml` is disabled, build and release manually:
|
||||
|
||||
```bash
|
||||
# Run the full pipeline (DB + verify + packs + consistency check)
|
||||
python scripts/pipeline.py --offline
|
||||
|
||||
# Or step by step:
|
||||
python scripts/generate_db.py --force --bios-dir bios --output database.json
|
||||
python scripts/verify.py --all
|
||||
python scripts/generate_pack.py --all --output-dir dist/
|
||||
|
||||
# Create the release
|
||||
DATE=$(date +%Y.%m.%d)
|
||||
gh release create "v${DATE}" dist/*.zip \
|
||||
--title "BIOS Pack v${DATE}" \
|
||||
--notes "Release notes here" \
|
||||
--latest
|
||||
```
|
||||
|
||||
To re-enable automated releases, remove the `if: false` guard from the
|
||||
`release` job in `build.yml`.
|
||||
Reference in New Issue
Block a user