chore: merge site and docs fixes, third pass

This commit is contained in:
Abdessamad Derraz committed 2026-11-06 06:38:47 +01:00
commit 19898cb6ea
46 files changed
+38582 -163

No files matched your search

+8 -5
View File
@@ -7,7 +7,7 @@ order: 20
Adding a platform gives its users a pack and a one-line install with the files
their setup checks, in the place it checks them, plus every file its emulators
need that its own list leaves out. The work is to read how the platform
need that its own list does not name. The work is to read how the platform
declares and checks BIOS files, then teach this project to read that
declaration. A platform can be a frontend like Batocera, a manager like EmuDeck
or a firmware database like BizHawk.
@@ -732,10 +732,13 @@ This executes in sequence, numbered as the run prints them (`STEP_LABELS` in
`--offline` skips steps 2 to 2b2.
A new platform is archived-by-default nowhere: `status: active` puts it in
`list_platforms.py`, the CI matrix, and the scrapers. Set
`status: archived` to keep the configuration without the automation. Archived
platforms are excluded unless `--all` or `--include-archived` is passed.
A new platform is `status: active`. `status: archived` keeps the platform
out of the runs that cover every platform (`verify.py --all`,
`generate_pack.py --all`, `slots.py`, `pipeline.py`, `list_platforms.py`)
unless `--include-archived` is passed. The installer manifests, the site, the
native export and `check_freshness.py` cover archived platforms as well. No
tool scrapes a platform on its own, archived or not: the `schedule:` field is
the cadence a maintainer keeps by hand.
A new platform has no release ZIP: `release_pack: true` is set for the setups
most people download, and the installer serves every other one from its
+5 -3
View File
@@ -52,6 +52,7 @@ website/ Astro Starlight project; pages, data and navigation
_mame_clones.json MAME parent/clone set mappings
database.json file index built from bios/ (SHA1 primary key)
derivable.json files a held dump already contains, from provenance_report.py
retired.json files that left the collection, written by generate_db.py
release.json files, sizes and assets of each pack of the last release
allowed_signers public key that signs SHA256SUMS.txt
dist/ generated packs (gitignored)
@@ -196,9 +197,10 @@ whoever asked for it:
- A pack built with `--emulator` answers to that emulator, and takes its file.
`scripts/slots.py` prints each contested destination and the reason for its
winner (`python scripts/slots.py --platform retrobat`). `generate_pack.py` and
`verify.py` both decide through `slots.arbitrate()`, so the pack and the report
agree.
winner (`python scripts/slots.py --platform retrobat`), as `slots.arbitrate()`
decides it. `generate_pack.py`, the install manifest and `verify.py` all read
`slots.slot_choices()` through `slots.pack_overrides()`, which applies the same
order, so the pack, the installer and the report agree.
A platform YAML is regenerated by its scraper and never edited by hand: an
edit would be lost at the next scrape, and the file would stop describing the
+2 -2
View File
@@ -77,8 +77,8 @@ setup's files and checks each one. `--dry-run` shows the list first.
## Fill an SD card from a computer?
The interface finds a muOS, Onion, spruceOS, MinUI, KNULLI or ROCKNIX card by
itself. Name your setup and the card's BIOS folder. For RetroArch on a Switch,
The interface finds a muOS, Onion, spruceOS, MinUI, KNULLI, ROCKNIX, Lakka or
MiSTer card. Else name your setup and its BIOS folder. For RetroArch on a Switch,
with the card in drive `E:`, run:
<!-- os-tabs windows: --platform retroarch --dest E:\retroarch\cores\system -->
+6 -3
View File
@@ -65,7 +65,7 @@ Every module in `tests/`, by the area it guards.
| Pack probe | the cores a platform runs for a system, staging grown to a fixed point, runs mapped to pack and manifest entries, the line and its schema, the probe lock per run, that no page shows its lines. One class builds a real pack narrowed to PlayStation and lays it out both ways, from the archive and with `install.py --dest`, and skips without the collection | `test_pack_probe` |
| Automation runner, disk cleanup, asset sync and bit-flip repair | fake jobs, git repositories with a bare origin and session worktrees, a committed file with one bit flipped, canned CI runs in place of `gh` | `test_autorun` |
| Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` |
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_cited_lines`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_pin_build`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_arcade_boards`, `test_site_availability`, `test_site_bootstrap`, `test_site_built_gate`, `test_site_components`, `test_site_console_folders`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_css`, `test_site_density_cards`, `test_site_emulator_page`, `test_site_emulators`, `test_site_entry`, `test_site_entry_tables`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_integration`, `test_site_nav`, `test_site_picker`, `test_site_platform_density`, `test_site_platform_hashes`, `test_site_reference`, `test_site_render`, `test_site_search`, `test_site_system_links`, `test_site_system_names`, `test_site_system_sections`, `test_site_systems`, `test_site_validation`, `test_site_weight`, `test_site_wiki`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_version_basis`, `test_workflow_paths` |
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_cited_lines`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_pin_build`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_retired_files`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_arcade_boards`, `test_site_availability`, `test_site_bootstrap`, `test_site_built_gate`, `test_site_components`, `test_site_console_folders`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_css`, `test_site_density_cards`, `test_site_emulator_page`, `test_site_emulators`, `test_site_entry`, `test_site_entry_tables`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_integration`, `test_site_nav`, `test_site_picker`, `test_site_platform_density`, `test_site_platform_hashes`, `test_site_reference`, `test_site_render`, `test_site_search`, `test_site_system_links`, `test_site_system_names`, `test_site_system_sections`, `test_site_systems`, `test_site_validation`, `test_site_weight`, `test_site_wiki`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_version_basis`, `test_workflow_paths` |
| Games on the card | small DATs, listxml and setup sources, a fake fetcher, fixture install lists, and the committed data against its schemas | `test_arcade_parents`, `test_game_catalogs`, `test_game_headers`, `test_game_regions`, `test_games_index`, `test_romfolders_card`, `test_romfolders_es` |
## Test architecture
@@ -463,9 +463,12 @@ refactor: extract shared utils to common.py
Files are staged by name, never with `git add -A` or `git add .`, which sweep in
deletions and work that is not yours. Generated artifacts (`database.json`,
`install/*.json`, `README.md`, `CONTRIBUTING.md`, `exports/`) are committed
`retired.json`, `derivable.json`, `install/*.json`, `README.md`, `CONTRIBUTING.md`, `exports/`,
the generated tables and run samples of `wiki/installer.md`) are committed
with the data they derive from, never alone: a manifest committed before the
files it points at gives the installer a 404.
files it points at gives the installer a 404. The run samples count what the
manifests hold, so `generate_installer_doc.py` runs after every manifest
regeneration and its output goes in the same commit.
## CI integration
+12 -7
View File
@@ -243,6 +243,7 @@ python scripts/generate_pack.py --all --offline # cache-only, never open t
# Install manifests (consumed by install.py)
python scripts/generate_pack.py --all --manifest --output-dir install/
python scripts/generate_pack.py --manifest-targets --output-dir install/targets/
python scripts/generate_installer_doc.py # run samples of wiki/installer.md, from the manifests
```
Packs include platform baseline files plus files required by the platform's cores.
@@ -364,9 +365,11 @@ test reads the original and the updated System.dat and es_bios.xml back to
check that the list is the diff.
No entry the platform declares is removed. One the truth does not pair is
looked for in the profiles of every core the platform runs: by name or
alias, by the basename or tail of a `path:`, by any hash (archive members
included), under a `type: directory` entry or a data directory. An entry
looked for, in this order: by the tail of a `path:` in the profiles of every
core the platform runs, by a name the platform's own launch code reads (its
`launchers:` records, such as the 22 names Batocera's DuckStation generator
tries), then in those profiles by name or alias, by any hash (archive
members included), under a `type: directory` entry or a data directory. An entry
none of them answers is written to `exports/<platform>.json` under
`removal_candidates`, with the cores examined for its system and a reason.
The verdict is `undetermined` when a core the platform names has no
@@ -481,7 +484,7 @@ python scripts/profile_sync.py --all --stale-before 2026-06-01 --limit 20 --tria
python scripts/profile_sync.py --emulator vice --full-diff --ref vice/src/c64/c64rom.h --context 6
python scripts/profile_sync.py --emulator vice --strict
python scripts/profile_sync.py --changed --strict
python scripts/profile_sync.py --added --base upstream/main --strict
python scripts/profile_sync.py --added --strict
```
Per part of a ref: `ANCHORED` (same content, same lines), `SHIFTED` (same
@@ -694,8 +697,9 @@ directories are checked against their upstream, dump catalogs and romset
snapshots against the catalog they were read from (Redump DAT versions, the
No-Intro daily rebuild, the newest TOSEC pack, the newest MAME release, the
git blobs of the FBNeo DATs), the game catalogs of `games/catalogs/` against
the No-Intro pack and the Redump datfiles they were read from, and the CI
toolchain against PyPI and the actions' latest releases.
the No-Intro pack and the Redump datfiles they were read from, the CI
toolchain against PyPI and the actions' latest releases, and the site's exact
npm pins in `website/package.json` against the npm registry.
The `native` area needs no network. It reads the cache `export_native.py`
patches from and reports an original that is absent, that came from another
@@ -1330,7 +1334,8 @@ before a release (wiki/release-process.md). Its lines stay in
| Script | Purpose |
|--------|---------|
| `common.py` | Shared library: hash computation, file resolution, platform config loading, emulator profiles, target filtering |
| `dedup.py` | Deduplicate `bios/` (`--dry-run`, `--bios-dir`): byte-identical copies are deleted and one kept. Identical MAME sets under other names stay one file, named in `_mame_clones.json`. Trees addressed by path are left alone (`NODEDUP_DIRS`): BBK, Bungie, C-Dogs SDL, Ikemen GO, PinMAME, RPG Maker, ScummVM, Softdisk, TheXTech, splitwolf, Theme Hospital, c64-emu, opentyrian, tzdata |
| `retired_files.py` | The record of files that left the collection, `retired.json`, which `generate_db.py` appends to and the site's lookup reads. `--backfill` rebuilds it from the history of `database.json` since content addresses were published (`--output` names another file) |
| `dedup.py` | Deduplicate `bios/` (`--dry-run`, `--bios-dir`): byte-identical copies are deleted and one kept. Identical MAME sets under other names stay one file, named in `_mame_clones.json`. Trees addressed by path are left alone (`NODEDUP_DIRS`), under a folder of that name at any depth: BBK, Bungie, C-Dogs SDL, Ikemen GO, mame, PinMAME, RPG Maker, ScummVM, Softdisk, TheXTech, splitwolf, Theme Hospital, c64-emu, opentyrian, tzdata |
| `identity.py` | One content filed under two identities: groups `bios/` by content and reports a group that crosses manufacturers outside `KNOWN_SHARED` (`--strict` exits 1, pipeline step 3a) |
| `slots.py` | Destinations the platform and a core both claim, and how the pack settles each (`--platform`, `--include-archived`, `--target`, `--json`, `--strict`, and pipeline step 3b passes the run's own `--include-archived` and `--target`) |
| `region_audit.py` | Confront every profile `region:` with the regions No-Intro names for the same dump (`--json`, `--db`, `--emulators-dir`). An opinion, never an authority: the code decides, and a disagreement is a line to read in the emulator's source |