feat: build the site with Astro Starlight

generate_site.py writes the Starlight project in website/ and nothing
else: mkdocs.yml, the MkDocs hooks, theme overrides and stylesheet go,
and the wiki is written in Starlight's syntax. Both workflows set up
Node with the official actions/setup-node pinned by commit, run npm ci
from the committed lockfile, build with the strict link validator and
check the result with validate_site.py.

The search weighs a page by what it answers: Start here pages first,
a system page's cards and file names before a guide citing them, deep
guides, launchers, aliases and frozen copies after. Status badges,
hash blocks and the export change lists leave the index. The ranking
sits at the centre of a range where the 13 queries of search_gate.py
all answer, in the search box itself.

839 pages, every URL and 45,611 ids match the MkDocs build. The tests
read the Starlight content and syntax.
This commit is contained in:
Abdessamad Derraz committed 2026-10-22 18:47:22 +02:00
1 parent abe240073c
commit 123d698e43
64 files changed
+742 -3118

No files matched your search

+1 -1
View File
@@ -652,7 +652,7 @@ This executes in sequence:
5. Consistency check: verify and the pack agree on counts and on the paths each archive holds
6. Pack integrity - extract ZIPs and verify hashes per platform mode
7. `generate_readme.py` - regenerate README and CONTRIBUTING
8. `generate_site.py` - regenerate documentation site and `mkdocs.yml`
8. `generate_site.py` - regenerate the documentation site and its navigation
A new platform is archived-by-default nowhere: `status: active` puts it in
`list_platforms.py`, the CI matrix, and the scrapers. Set
+5 -3
View File
@@ -39,7 +39,8 @@ data/ cached data directories (not BIOS, fetched at build)
schemas/ JSON schemas for platform and emulator YAML (checked in CI)
tests/ test suite with synthetic fixtures
wiki/ hand-written documentation sources
docs/ generated MkDocs site (gitignored, rebuilt in CI)
website/ Astro Starlight project; pages, data and navigation
are generated into it (gitignored, rebuilt in CI)
_mame_clones.json MAME parent/clone set mappings
database.json file index built from bios/ (SHA1 primary key)
dist/ generated packs (gitignored)
@@ -55,7 +56,8 @@ heading above the match. Every file card is therefore a level 3 heading with
the card's anchor, under its console or a Files heading, and every platform
system a heading of its own: a file name typed in the search opens that card,
not the top of the page. On pages of cards (systems, emulators, platforms) the
table of contents stops at level 2 (`scripts/mkdocs_hooks.py`).
table of contents stops at level 2, and a page of cards under one or two
headings has none (`scripts/sitestarlight.py`).
The browser downloads and indexes the whole search index before it shows a
first result, so the index holds names and prose only. Hashes, lists of other
@@ -450,7 +452,7 @@ pattern and how to add a test.
| Workflow | File | Trigger | Role |
|----------|------|---------|------|
| Deploy Site | `deploy-site.yml` | push to main on any input of the site build, and manual | validate contracts, generate site, build with MkDocs, validate rendered HTML, deploy to Pages |
| Deploy Site | `deploy-site.yml` | push to main on any input of the site build, and manual | validate contracts, generate site, build with Astro Starlight, validate rendered HTML, deploy to Pages |
| Validation | `validate.yml` | pull request and push to main on the collection, profiles, schemas, scripts, tests, the installer and every input of the site build | schema check and full test suite on both, BIOS hash validation and auto-label on pull requests |
Each workflow's `paths:` list is the exact set of files that starts it.
+3 -2
View File
@@ -21,6 +21,7 @@ See the project's [README](https://github.com/monster-penguin/BIOS-Preservation-
---
!!! note "Add your project"
:::note[Add your project]
Open an issue with a link to your repository.
:::
Open an issue with a link to your repository.
+4 -3
View File
@@ -104,7 +104,8 @@ and name the file. A file is removed when the owner of its rights asks.
Files disappear when stores close, like the 3DS and Wii U shops in 2023. We keep
them while they can still be found.
!!! tip "Next"
:::tip[Next]
- [Get your files](../which-pack.md)
- [Words used on this site](glossary.md)
:::
- [Get your files](../which-pack.md)
- [Words used on this site](glossary.md)
+35 -26
View File
@@ -15,39 +15,47 @@ From an empty BIOS folder to games that start.
A **terminal** is a window where you type commands. Paste the line for your
system, then press Enter.
=== "Windows"
<Tabs syncKey="os">
<TabItem label="Windows">
Open the Start menu. Type **PowerShell** and open it.
Open the Start menu. Type **PowerShell** and open it.
```powershell
irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex
```
```powershell
irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex
```
=== "Mac"
</TabItem>
<TabItem label="Mac">
Open **Terminal**. It is in Applications, then Utilities.
Open **Terminal**. It is in Applications, then Utilities.
```sh
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
```
```sh
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
```
=== "Linux and Steam Deck"
</TabItem>
<TabItem label="Linux and Steam Deck">
On a Steam Deck, switch to Desktop Mode and open **Konsole**.
On a Steam Deck, switch to Desktop Mode and open **Konsole**.
```sh
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
```
```sh
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
```
=== "Android"
</TabItem>
<TabItem label="Android">
Install **Termux**, a terminal app. Run these three lines in it.
Install **Termux**, a terminal app. Run these three lines in it.
```sh
pkg install python
termux-setup-storage
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
```
</TabItem>
</Tabs>
```sh
pkg install python
termux-setup-storage
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
```
On Batocera, Recalbox and other game systems, follow
[your setup's page](../which-pack.md).
@@ -145,8 +153,9 @@ drive.
See [When a game does not start](troubleshooting.md).
!!! tip "Next"
:::tip[Next]
- [Your setup's page](../which-pack.md): its folder and how it checks files
- [When a game does not start](troubleshooting.md)
- Details: [Reading the installer's output, line by line](installer.md#reading-a-run)
:::
- [Your setup's page](../which-pack.md): its folder and how it checks files
- [When a game does not start](troubleshooting.md)
- Details: [Reading the installer's output, line by line](installer.md#reading-a-run)
+1 -1
View File
@@ -7,7 +7,7 @@ nav: false
Three pages take you from nothing to games that start.
[Get your files](../which-pack.md){ .md-button .md-button--primary }
<a class="rb-button rb-button-primary" href="../which-pack.md">Get your files</a>
## What is a BIOS?
+5 -4
View File
@@ -43,8 +43,9 @@ nav_title: Glossary
**Game data**
: Files from your own copy of a PC game, like Doom. Not a BIOS.
!!! tip "Next"
:::tip[Next]
- [What is a BIOS?](what-are-bios-files.md)
- [Get your files](../which-pack.md)
- Reference: [Terms of the reference pages](reference-terms.md)
:::
- [What is a BIOS?](what-are-bios-files.md)
- [Get your files](../which-pack.md)
- Reference: [Terms of the reference pages](reference-terms.md)
+4 -3
View File
@@ -41,7 +41,8 @@ This reflects the project's good-faith understanding, not legal advice.
A rights holder, or anyone acting for one, can [open an issue](https://github.com/Abdess/retrobios/issues) identifying the file by path or hash. A file whose removal is requested by its rights holder is removed from the repository and from the next release. No formal notice is required and none will be demanded. State the file and the basis of the claim. The same channel is open to anyone else with a concern about a specific file.
!!! tip "Next"
:::tip[Next]
- [FAQ](faq.md)
- [What is a BIOS?](what-are-bios-files.md)
:::
- [FAQ](faq.md)
- [What is a BIOS?](what-are-bios-files.md)
+4 -3
View File
@@ -149,7 +149,8 @@ everyday words (BIOS, emulator, core, hash) are in the [glossary](glossary.md).
takes a file from (the collection, a data folder, a release asset) is its
source, not its provenance.
!!! tip "Next"
:::tip[Next]
- [How each platform checks files](verification-modes.md)
- [Glossary](glossary.md)
:::
- [How each platform checks files](verification-modes.md)
- [Glossary](glossary.md)
+28 -21
View File
@@ -18,8 +18,8 @@ Budget target: ~175 minutes/month on the GitHub free tier.
| Workflow | File | Trigger |
|----------|------|---------|
| Deploy Site | `deploy-site.yml` | Push to main (platforms, emulators, provenance, wiki, scripts, database.json, mkdocs.yml), manual |
| Validation | `validate.yml` | PR and push to main touching `bios/**`, `platforms/**`, `emulators/**`, `schemas/**`, `scripts/**`, `tests/**`, `install.py`, `install.sh`, `install.ps1`, or any other input of the site build (`wiki/**`, `docs_assets/**`, `provenance/**`, `exports/**`, `install/**`, `database.json`, `release.json`, `derivable.json`, `mkdocs.yml`) |
| Deploy Site | `deploy-site.yml` | Push to main (platforms, emulators, provenance, wiki, scripts, database.json, website), manual |
| Validation | `validate.yml` | PR and push to main touching `bios/**`, `platforms/**`, `emulators/**`, `schemas/**`, `scripts/**`, `tests/**`, `install.py`, `install.sh`, `install.ps1`, or any other input of the site build (`wiki/**`, `docs_assets/**`, `provenance/**`, `exports/**`, `install/**`, `database.json`, `release.json`, `derivable.json`, `website/**`) |
Upstream BIOS lists are not scraped on a schedule. A maintainer runs the
scrapers by hand (see [adding a scraper](adding-a-scraper.md)), reviews the
@@ -32,7 +32,8 @@ hosted runner should rebuild and re-upload.
**Trigger.** Push to `main` when any of these paths change: `platforms/`,
`emulators/`, `provenance/`, `wiki/`, `schemas/`, `tests/`, `docs_assets/`,
`install/`, `install.py`, `database.json`, `release.json`, `mkdocs.yml`, the
`install/`, `install.py`, `database.json`, `release.json`, `website/` (the
Starlight project: configuration, components, styles, lockfile), the
workflow itself, and every script the build runs or imports. Also manual
dispatch.
@@ -45,19 +46,21 @@ is missing.
### Steps
1. Checkout, Python 3.12
2. Install `pyyaml`, `jsonschema[format-nongpl]==4.26.0`,
`mkdocs>=1.6,<2`, `mkdocs-material>=9.7.5,<10`,
`pymdown-extensions>=10.14`
2. Install `pyyaml` and `jsonschema[format-nongpl]==4.26.0`, then Node 24 with
the official `actions/setup-node` (npm cache keyed on the lockfile) and
`npm ci --prefix website`
3. Run `validate_schemas.py`: the data contracts, checked before anything is
generated (see below)
4. Restore large files from the `large-files` release, refresh data directories
5. Run `generate_site.py` (converts YAML data into MkDocs pages and rewrites
`mkdocs.yml`)
5. Run `generate_site.py`: it writes the pages, the navigation and the data
files into the Astro Starlight project in `website/`
6. Run `generate_readme.py` (rebuilds README.md and CONTRIBUTING.md)
7. `mkdocs build --strict` to produce the static site
8. Run `validate_site.py` on the rendered HTML (metadata, headings, image
alternatives, duplicate ids, local links and fragments, page and search
index budgets)
7. `npm run build --prefix website` to produce the static site in
`website/dist/`. The build runs `starlight-links-validator` with its
strict defaults, so a dead link or anchor fails it
8. Run `validate_site.py --site-dir website/dist` on the rendered HTML
(metadata, headings, image alternatives, duplicate ids, every local link
and fragment again, page and search index budgets)
9. Require the committed README and CONTRIBUTING to match what the generator
just produced. `write_if_changed()` compares content with the timestamp line
stripped, so a run that only moves the clock leaves the files untouched and
@@ -77,14 +80,18 @@ The site is deployed via the `github-pages` environment using the official
cancelled (`cancel-in-progress: false`): cancelling one mid-flight leaves the
deployment stuck and the next runs time out waiting on it.
`--strict` turns MkDocs warnings into failures, so a broken internal link or a
dangling anchor fails the build instead of shipping. The `validation:` block in
`mkdocs.yml` is what promotes unrecognized links and missing anchors to
warnings in the first place.
Two link checks run, and both fail the build: Starlight's validator inside
`astro build`, and `validate_site.py` over the rendered files. They read
different things (the content collection, the final HTML with its navigation
and breadcrumbs), and each has caught what the other passed.
The theme version is pinned on both sides: `>=9.7.5` because that is the
release which caps `mkdocs < 2` (MkDocs 2.0 ships without a license), `<10`
so a major theme release cannot change the site without a deliberate bump.
Astro, Starlight and their plugins are pinned to exact versions in
`website/package.json`, resolved from the committed `website/package-lock.json`
by `npm ci`. Starlight is below 1.0 and Astro shipped two majors in 2026, so an
upgrade is its own change, built and measured with the scripts in
`website/tools/` (search queries, Lighthouse, axe, screenshots) before it
merges. Node runs only inside the site build: `install.py` and the build
scripts stay Python.
## validate.yml - Validation
@@ -119,7 +126,7 @@ declares while schemas and tests pass it. This job confronts it with its
upstream. An edited profile is not held to it.
**run-tests.** Runs `python -m unittest discover -s tests -t . -v`, then builds the
site as Deploy Site does (`generate_site.py`, `mkdocs build --strict`,
site as Deploy Site does (`generate_site.py`, `npm run build --prefix website`,
`validate_site.py`) whatever the suite returned. Must pass before a merge, and
again on the commit a direct push puts at the head of main.
@@ -257,7 +264,7 @@ python scripts/check_release.py --tag "v${DATE}" --sums dist/SHA256SUMS.txt \
# and pack_size describe the packs just built. git status names anything
# else the run rewrote.
git status --short
git add release.json README.md mkdocs.yml install/*.json install/targets/*.json
git add release.json README.md install/*.json install/targets/*.json
git commit -m "chore: record release v${DATE}" && git push
# 6. Keep only the new release plus large-files: an older pack carries hashes
+4 -3
View File
@@ -143,7 +143,8 @@ Running `verify.py --emulator <core> --verbose` shows the emulator-level ground
It reaches only as far as the profiles do. Files no profile states a value for are platform-only, and nothing can be said about their content. The footer of the platform report gives that ratio as `Ground truth: N/M files have emulator validation`.
!!! tip "Next"
:::tip[Next]
- [Reference index](index.md), every reference page in one list
- For contributors: [How to contribute](../contributing.md)
:::
- [Reference index](index.md), every reference page in one list
- For contributors: [How to contribute](../contributing.md)
+19 -3
View File
@@ -287,7 +287,23 @@ The test suite is one layer of verification. The full quality gate is:
3. No unexpected CRITICAL entries in the verify output
4. Pack file counts match verification file counts (consistency check)
5. Pack integrity passes (every declared file extractable with correct hash)
6. `mkdocs build --strict` and `python scripts/validate_site.py` pass
6. `npm run build --prefix website` and `python scripts/validate_site.py` pass
A change to the site engine, its version, its search ranking or the page
weights is measured on a served build before it lands, with the scripts in
`website/tools/` (they need `npm install --no-save playwright-core axe-core
lighthouse` and Chrome):
- `search_gate.py`: the queries a visitor types and the page that must answer
each, read from the site's own search box
- `lighthouse.py`: Lighthouse mobile on a list of pages, the heaviest ones
included
- `axe.mjs`: WCAG 2.2 AA in light and dark
- `check_tools.mjs`: the hash lookup, the file picker, a hash typed in the
search, breadcrumbs, previous and next across tabs, glossary terms, tables
on a phone and the 404 page, in a real browser
- `serve.mjs`: the build served under /retrobios/ with gzip, as GitHub Pages
serves it
A change that passes the tests but breaks the pipeline is not merged. A new
CRITICAL entry in the verify output after a change is explained before the
@@ -380,7 +396,7 @@ nav_title: Save files # only when the menu names it otherwise than its headin
- `level` picks the tab and the label shown at the top of the page.
- `order` sorts the tab. Pages are spaced by ten, so a new page fits between
two without renumbering. The generated pages (systems, emulators, the gaps
and data pages) hold fixed places among them, set in `generate_mkdocs_nav`.
and data pages) hold fixed places among them, set in `generate_nav`.
- `nav_title` is the menu label. Without it the menu shows the page's `# `
heading. It is not `title`, which both MkDocs and Astro Starlight read as
the page's own title.
@@ -419,7 +435,7 @@ The `validate.yml` workflow runs `python -m unittest discover -s tests -t . -v`
roads into main: every pull request, and every direct push, that touches
`bios/`, `platforms/`, `emulators/`, `schemas/`, `scripts/`, `tests/`,
`install.py` or an input of the site build such as `wiki/`. The same job then
builds the site with `mkdocs build --strict`, so a page that would fail the
builds the site with `npm run build --prefix website`, so a page that would fail the
deploy fails the pull request. Work reaches main by push as often as by pull request, so a suite
wired to pull requests alone would guard the road nobody takes. The test job
(`run-tests`) runs in parallel with schema validation, and on a pull request
+3 -3
View File
@@ -51,7 +51,7 @@ Pipeline steps:
| 6 | Pack integrity (extract + hash) | `--skip-packs` |
| 7 | Generate README | `--skip-docs` |
| 8 | Generate site | `--skip-docs` |
| 8b | Render the site with `mkdocs build --strict`, then `validate_site.py`, as the deploy does | `--skip-docs`, or no `mkdocs` installed |
| 8b | Render the site with `npm run build --prefix website`, then `validate_site.py`, as the deploy does | `--skip-docs`, or Node not set up (`npm ci --prefix website`) |
Pipeline flags:
@@ -692,7 +692,7 @@ python scripts/refresh_stale.py --only platforms,targets --jobs 6
| `provenance_report.py` | Dump-catalog coverage and acquisition targets (see above) |
| `find_pinned_revision.py` | Walk the upstream commits that touched a file and return the revision whose bytes match a platform's pinned md5 (`--write` saves them) |
| `generate_readme.py` | Generate README.md and CONTRIBUTING.md from database |
| `generate_site.py` | Generate all MkDocs site pages (this documentation) |
| `generate_site.py` | Generate the site (this documentation) into the Starlight project in `website/` |
| `validate_site.py` | Validate rendered metadata, heading order, image alternatives, JSON-LD, local resources, links and fragments, and the page size and search index budgets |
| `romset_recipes.py` | Identify which emulator version an arcade archive matches, and rebuild a pinned archive from ROMs already held |
| `scraper/romset_dat_importer.py` | Fetch and import per-set recipes from MAME `-listxml` or FBNeo DATs into `recipes/`. `--fetch` alone takes the newest MAME release, and FBNeo DATs are refreshed by git blob sha. `--fetch <revision>` reads FBNeo at that tag, branch or commit |
@@ -732,7 +732,7 @@ Further options, by script:
| `split_pack.py` | `--max-size 1900M`, the ceiling per part (K, M, G, default 2G) |
| `torrentzip.py` | `--check ZIP` reports conformance, `--rebuild ZIP -o OUT` normalizes an archive |
| `validate_pr.py` | `--changed` takes the BIOS files the branch changes, `--markdown`, `--json` |
| `validate_site.py` | `--site-dir` (rendered site), `--config` (mkdocs.yml), `--max-errors N` |
| `validate_site.py` | `--site-dir` (rendered site, default `website/dist`), `--base` (path the site is served under), `--max-errors N` |
| `scraper/coreinfo_scraper.py` | `--compare-db database.json` lists the `.info` firmware the database lacks |
| `scraper/mame_hash_scraper.py` | `--release mameNNNN --profile FILE --source-commit SHA --core NAME --built-by PLATFORM` writes the profile of a MAME build a platform compiles. `--devices --listxml FILE` documents in `mame.yml` every device set with dumped ROMs that the release's `-listxml` names (`isdevice="yes"`), with `--dry-run` to preview |
| `scraper/romset_dat_importer.py` | `--cache-dir` (downloaded listxml and DATs) |
+4 -3
View File
@@ -78,7 +78,8 @@ Other messages: Details: [When something goes wrong](installer.md#when-something
<span id="hash-mismatch-untested"></span><span id="pack-is-missing-files"></span><span id="verifypy-reports-errors"></span>
!!! tip "Next"
:::tip[Next]
- [Your setup's page](../which-pack.md): its folder and how it checks files
- [FAQ](faq.md)
:::
- [Your setup's page](../which-pack.md): its folder and how it checks files
- [FAQ](faq.md)
+5 -4
View File
@@ -9,7 +9,7 @@ nav_title: What BIOS files are
A console has a small program inside it. It starts first, then starts the game.
This program is the **BIOS**, also called **firmware**.
[Get your files](../which-pack.md){ .md-button .md-button--primary }
<a class="rb-button rb-button-primary" href="../which-pack.md">Get your files</a>
An **emulator** is an app that acts like a console. For many consoles, it needs
a copy of the BIOS. That copy is a file: a **BIOS file**.
@@ -59,7 +59,8 @@ then glitch or do not start. The real file works best, so we include it.
Laws differ by country. [Read the project's position](faq.md#is-this-legal).
!!! tip "Next"
:::tip[Next]
- [Get your files](../which-pack.md)
- [Words used on this site](glossary.md)
:::
- [Get your files](../which-pack.md)
- [Words used on this site](glossary.md)