feat: publish the games index with install lists

This commit is contained in:
Abdessamad Derraz committed 2026-11-03 17:45:03 +01:00
1 parent eaf0fbdd19
commit c2ad67e4fa
128 files changed
+1234 -4

No files matched your search

+40
View File
@@ -47,6 +47,7 @@ Pipeline steps:
| 4b | Generate install manifests | `--skip-packs` |
| 4c | Generate target manifests | `--skip-packs` |
| 4d | Collected copies only an emulator install provides (`bundled_only.py --strict`, gates the run) | - |
| 4e | Games index: what the installer reads for the games on a card (`generate_games_index.py`, fails under its coverage floor) | - (reads the committed install lists) |
| 5 | Consistency check: counts, paths per archive, every emulator pack | if verify or pack skipped |
| 6 | Pack integrity (extract + hash) | `--skip-packs` |
| 7 | Generate README and CONTRIBUTING, then the installer page's tables and run samples (`generate_installer_doc.py`) | `--skip-docs` |
@@ -1359,6 +1360,11 @@ Further options, by script:
| `scraper/coreinfo_scraper.py` | `--compare-db database.json` lists the `.info` firmware the database lacks |
| `scraper/mame_hash_scraper.py` | `--release mameNNNN --profile FILE --source-commit SHA --core NAME --built-by PLATFORM` writes the profile of a MAME build a platform compiles. `--devices --listxml FILE` documents in `mame.yml` every device set with dumped ROMs that the release's `-listxml` names (`isdevice="yes"`), with `--dry-run` to preview |
| `scraper/romset_dat_importer.py` | `--cache-dir` (downloaded listxml and DATs) |
| `scraper/game_dat_importer.py` | `--source no-intro\|redump` (repeatable), `--pack FILE` imports a local ZIP, directory or DAT, `--refresh` asks the servers again, `--cache-dir`, `--output-dir`, `--dry-run` |
| `scraper/arcade_parent_importer.py` | `--profile NAME` (one core version), `--emulators-dir`, `--database` (BIOS roots checked against it by archive name), `--output-dir`, `--cache-dir`, `--offline`, `--dry-run` |
| `scraper/romfolders` | `--sources DIR` (default `platforms/romfolders/_sources`), `--output-dir`, `--dry-run` |
| `game_rules.py` | `--regions`, `--extras`, `--headers` (files checked against the profiles and the header readers), `--emulators-dir`, `--json` |
| `generate_games_index.py` | `--output-dir` (default `install/games`), `--check`, `--quiet` |
## Installation tools
@@ -1603,6 +1609,40 @@ on the database entry and renders a verified dump badge on the system pages.
It never overrides the emulator source code: see
[architecture](architecture.md#dump-catalog-provenance).
## Games on the card
The installer's games-on-card mode reads the rom folders of a card, identifies
each game and installs only the BIOS those games need. Its data is generated
from sources and published under `install/games/` by pipeline step 4e.
```bash
python -m scripts.scraper.romfolders --list # setups and the format each is read in
python -m scripts.scraper.romfolders --all # platforms/romfolders/<setup>.yml at the pins of _sources/
python -m scripts.scraper.romfolders --setup batocera --offline
python -m scripts.scraper.game_dat_importer --fetch --floor # games/catalogs/: No-Intro and Redump game DATs
python -m scripts.scraper.game_dat_importer --floor # same from .cache/game-dats/ only
python -m scripts.scraper.arcade_parent_importer --all # games/arcade/: game set to BIOS parent, per core
python scripts/game_rules.py # games/regions.yml and games/extras.yml against the profiles
python scripts/generate_games_index.py # install/games/ (pipeline step 4e)
python scripts/generate_games_index.py --check # exit 1 when a published file would change
```
| Source | Written by | Holds |
|--------|-----------|-------|
| `platforms/romfolders/_sources/<setup>.yml` | hand, each fact with its `source_ref` | pins, roms root, case rule, folders never scanned |
| `platforms/romfolders/<setup>.yml` | `scraper/romfolders` | folders, extensions and default core as the OS writes them |
| `games/catalogs/<source>/<system>.json` | `scraper/game_dat_importer.py` | crc32, size, sha1, region and title per game, disc serials |
| `games/headers.yml` | hand, cited | header fields each core reads and the region each value means |
| `games/regions.yml`, `games/extras.yml` | hand, cited | region routes, the cases that keep every region, per-game files |
| `games/arcade/<profile>.json` | `scraper/arcade_parent_importer.py` | BIOS parent and device sets of each game set |
Folder names resolve to systems through `SYSTEM_ALIASES` and `_norm_system_id`
in `common.py`, the one alias table, and core names to profiles through the
profiles' `cores:`. The fallback set of a folder, what an unrecognised game
keeps, is computed from `install/<platform>.json` with the installer's own
core narrowing. Every published file is listed in `install/games/index.json`
with its size, its gzip size and its sha256.
## Target scrapers
Located in `scripts/scraper/targets/`. Each inherits `BaseTargetScraper` and implements `fetch_targets()`.