docs: align profiling and platform pages with code

This commit is contained in:
Abdessamad Derraz committed 2026-11-03 10:08:42 +01:00
1 parent e9ef630b3a
commit 8609810d8f
2 files changed
+106 -67

No files matched your search

+73 -40
View File
@@ -57,6 +57,10 @@ class Scraper(BaseScraper):
- `compare_with_config(config) -> ChangeSet` - diffs scraped requirements against
an existing platform YAML, returning added/removed/modified entries.
- `test_connection() -> bool` - checks if the source URL is reachable.
- `lists_files` - `True` by default, and an empty read then stops the scrape.
A scraper whose platform lists no file of its own sets it to `False`: RetroPie
inherits RetroArch's files and reads only its packages, and MinUI names no
file at all.
Two abstract methods must be implemented:
@@ -193,7 +197,8 @@ resolved. Three strategies exist:
- **Explicit list**: `cores: [beetle_psx, dolphin, ...]` - match by profile key name
or by any name in a profile's `cores:`. When a profile's `built_by` names this
platform, that profile answers for the name. Used by Batocera, Recalbox, RetroBat,
RomM, RetroPie, EmuDeck, RetroDECK, ROCKNIX. A list that names `retroarch` (the last
RomM, KNULLI, AmberELEC, EmuELEC, MinUI, NextUI, MyMinUI, RetroPie, EmuDeck,
RetroDECK, ROCKNIX. A list that names `retroarch` (the last
four) also takes every libretro profile, as `all_libretro` does. An empty list
(BizHawk, MiSTer FPGA) resolves no profile, and the pack holds the platform's own
files.
@@ -291,7 +296,8 @@ that text belongs in a guide field.
```yaml
logo: https://... # SVG or PNG for UI/docs
inherits_from: retroarch # inherit systems/cores from another platform
inherits_from: retroarch # the parent, for readers: the merge reads
# inherits: in the platform YAML
case_insensitive_fs: true # if the platform runs on case-insensitive filesystems
extract_to: /userdata/bios/ # folder the pack extracts into, shown in the README,
# the site and the release notes
@@ -379,8 +385,8 @@ AmberELEC and EmuELEC declare `neogeo/` and `ports/` beside `bios/`, and set
Between the file a user drops in the BIOS folder and the emulator, a
platform's own launch code often decides which name is read and where it is
handed over. Batocera's DuckStation generator tries 22 names per region, first
found wins, and writes the hit into `settings.ini`. That behaviour belongs to
handed over. Batocera's DuckStation generator tries a list of names for each
region, first found wins, and writes the hit into `settings.ini`. That behaviour belongs to
the platform, not to the emulator, so the scraper records it from the
platform's code under `launchers:`, never as an alias in an emulator profile.
@@ -494,10 +500,10 @@ RetroPie, EmuDeck and RetroDECK spell it, and a path from a root
### Inheritance
If the new platform inherits from an existing one (e.g. Lakka inherits RetroArch),
set `inherits_from` in the registry AND add `inherits: retroarch` in the platform
YAML itself. `load_platform_config()` reads the `inherits:` field from the YAML to
merge parent systems and shared groups into the child. The child YAML only needs to
declare overrides.
add `inherits: retroarch` in the platform YAML. `load_platform_config()` reads that
field to merge the parent's systems and shared groups into the child, and the child
YAML only declares what differs. The registry's `inherits_from` names the parent
for readers and is read by no tool.
## Step 3: Generate the platform YAML
@@ -507,9 +513,9 @@ Run the scraper with `--output` to produce the initial platform configuration:
python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
```
If a file already exists at the output path, the CLI preserves fields that the
scraper does not generate (e.g. `data_directories`, manually added metadata).
Only the `systems` section is replaced.
If a file already exists at the output path, every top-level key the scraper
writes is replaced, and the keys it does not write are kept. A system's
`data_directories` is kept as well when the new scrape does not carry one.
The YAML is the scraper's output and is never edited by hand: the next scrape
would undo the edit, and the file would stop describing the platform. Anything
@@ -546,37 +552,59 @@ to `verify.py` in the platform-specific verification path.
Read the platform's source code to understand its exact verification behavior before writing any logic. Batocera's `checkInsideZip` uses `casefold()` for case-insensitive matching. Recalbox supports comma-separated MD5 lists. RomM checks file size before hashing. These details matter: the project replicates native behavior, not an approximation of it.
## Step 5: Create an exporter (optional)
## Step 5: Create an exporter
Exporters convert truth data back to the platform's native format. They live in
`scripts/exporter/` and follow the same auto-discovery pattern (`*_exporter.py`).
An exporter writes the platform's own BIOS file back, corrected by what the
emulator code shows. Every registered platform has one, unless its registry
entry says `native_list: none`: `tests/test_exporters.py` fails otherwise.
Exporters live in `scripts/exporter/` and are found the same way as scrapers
(`*_exporter.py`).
### Module contract
The module must export an `Exporter` class inheriting `BaseExporter`:
The module exports an `Exporter` class that extends `BaseExporter`
(`scripts/exporter/base_exporter.py`):
```python
from scripts.exporter.base_exporter import BaseExporter
from .base_exporter import BaseExporter
from .baseline import NativeSystem, Report
class Exporter(BaseExporter):
@staticmethod
def platform_name() -> str:
return "myplatform"
def export(self, truth_data: dict, output_path: str, scraped_data: dict | None = None) -> None:
# Write truth_data in the platform's native format to output_path
@staticmethod
def native_filename() -> str:
return "bios_list.json" # the name the platform gives its file
@staticmethod
def native_sources() -> dict[str, str]:
return {"bios_list.json": "https://..."} # the original it patches
def render(self, systems: dict[str, NativeSystem], report: Report,
originals: dict[str, str], scraped: dict | None = None) -> dict[str, str]:
# Return {relative output path: file content}
...
def validate(self, truth_data: dict, output_path: str) -> list[str]:
def validate(self, systems: dict[str, NativeSystem], produced: dict[str, str]) -> list[str]:
# Return a list of issues (empty = valid)
...
```
`BaseExporter` provides helper methods:
`platform_name`, `native_filename`, `render` and `validate` are abstract. The
other methods say what the format can hold, and their defaults fit a plain list
of files:
- `_is_pattern(name)` - True if the filename contains wildcards or placeholders.
- `_dest(fe)` - resolve destination path from a file entry dict.
- `_display_name(sys_id, scraped_sys)` - convert a system slug to a display name.
- `carries()` - the fields the format has a place for (`md5` by default). A
correction to a field it cannot hold is not counted as made.
- `requires()` - the hash a new entry cannot be written without.
- `can_add()` - `False` when an entry is code rather than a row of data
(BizHawk), so nothing is added.
- `keys_by_system()` - `True` when the platform finds a file by the name of its
own system, so an entry filed under one of our system ids is never written.
- `needs_original()` - `True` as soon as `native_sources()` names a file.
### Round-trip validation
@@ -584,11 +612,10 @@ The scraper and the exporter are checked against each other as
[round-trip testing](adding-a-scraper.md#round-trip-testing) describes. The
export must not lose anything the platform already declares. If your
format carries code as well as data (a script, a C# database, a manifest
whose BIOS list sits beside launch configuration), say so with
`needs_original()` and patch the platform's original file rather than writing
a new one. An
exporter that regenerates such a file from BIOS data alone hands the
maintainer something that no longer runs.
whose BIOS list sits beside launch configuration), name the file in
`native_sources()` and patch it rather than writing a new one. An exporter
that regenerates such a file from BIOS data alone hands the maintainer
something that no longer runs.
## Step 6: Create a target scraper (optional)
@@ -643,19 +670,25 @@ cores the pack ships.
## Step 7: Add install detection (optional)
The `install` section in `_registry.yml` tells `install.py` how to detect
the platform on the user's machine and locate its BIOS directory.
The `install.detect` rules in `_registry.yml` say how `install.py` finds the
platform on the user's machine and where its BIOS folder is. The installer
runs before it fetches any manifest, so it carries the same rules in its own
tables, and `tests/test_install.py` holds the two equal.
Three detection methods are available:
| Method | Finds the platform when | Fields |
|--------|-------------------------|--------|
| `config_file` | a key of a config file names the folder | `config`, `parse_key`, optionally `bios_subdir` or `bios_path` |
| `path_exists` | a folder exists | `path`, optionally `bios_path` |
| `file_exists` | a file exists | `file`, `bios_path`, optionally `name`, `config`, `home` |
| `os_release` | `/etc/os-release` holds one of the `match` pairs | `match`, `name`, `bios_path`, optionally `config`, `home` |
| `file_matches` | a file's text matches `pattern` | `file`, `pattern`, `name`, `bios_path` |
| `executable` | a portable install's program is found | `file` (glob patterns), `bios_subdir` |
| `registry` | a Windows registry value names the install folder | `key`, `value`, `bios_subdir` |
| `card_marker` | a file on the SD card starts with a prefix | `file`, `first_line_prefix` |
| Method | Description | Fields |
|--------|-------------|--------|
| `config_file` | Parse a key from a config file | `config`, `parse_key`, optionally `bios_subdir` |
| `path_exists` | Check if a directory exists | `path`, optionally `bios_path` |
| `file_exists` | Check if a file exists | `file`, optionally `bios_path` |
Each entry is scoped to an OS (`linux`, `darwin`, `windows`). Multiple entries
per OS are tried in order.
Each rule names its `os`: `linux`, `darwin`, `windows`, or `any` for a card
read from another computer. Rules for one OS are tried in order.
`derives_from` names the system a rule's system is built on.
## Step 8: Validate the full pipeline
@@ -708,6 +741,6 @@ Verification is not optional. A platform that passes `pipeline.py` today may bre
- [ ] Guide added to `platforms/_guides.yml`
- [ ] Platform YAML generated and verified
- [ ] `python scripts/pipeline.py --offline` passes
- [ ] Exporter in `scripts/exporter/<name>_exporter.py` (if applicable)
- [ ] Exporter in `scripts/exporter/<name>_exporter.py`, unless the registry says `native_list: none`
- [ ] Target scraper in `scripts/scraper/targets/<name>_targets_scraper.py` (if applicable)
- [ ] Install detection entries in `_registry.yml` (if applicable)
+33 -27
View File
@@ -163,9 +163,9 @@ and consistent across all profiles.
When a core has HLE (high-level emulation), code of its own that stands
in for the file, mark the entry `hle_fallback: true`. The flag records the
stand-in. It does not decide `required`, which still comes from what the
code does without the file: most of the 778 entries that carry the flag
are optional, and 13 are required, among them Beetle PSX's three region
BIOS (`emulators/beetle_psx.yml`). The file still ships in packs, and
code does without the file: most entries that carry the flag are optional,
and a few are required, among them Yabause's `saturn_bios.bin`
(`emulators/yabause.yml`). The file still ships in packs, and
verification reports its absence at INFO whatever `required` says.
MAME is read the same way. A missing ROM is fatal only when it is neither
@@ -275,30 +275,33 @@ what the user actually needs to provide.
emulator: Dolphin
type: standalone + libretro
core_classification: community_fork
source: https://github.com/libretro/dolphin
upstream: https://github.com/dolphin-emu/dolphin
profiled_date: 2026-03-25
source: "https://github.com/libretro/dolphin"
upstream: "https://github.com/dolphin-emu/dolphin"
profiled_date: "2026-03-23"
source_commit: "3aac3f052f733f8df2f1616843a4d2e14002ed11"
upstream_commit: "0d93fc035539f25790deca638d06ee3b3b45ee64"
core_version: "2609-437-g3aac3f0"
version_basis:
kind: tag
tag: "2609"
distance: 437
systems:
- nintendo-gamecube
- nintendo-wii
systems: [nintendo-gamecube, nintendo-wii, sega-triforce]
files:
- name: GC/USA/IPL.bin
- name: "IPL.bin"
path: "dolphin-emu/Sys/GC/USA/IPL.bin"
standalone_path: "GC/USA/IPL.bin"
system: nintendo-gamecube
region: [north-america, latin-america]
size: 2097152
required: false
hle_fallback: true
size: 2097152
validation: [size, adler32]
known_hash_adler32: 0x4f1f6f5c
region: [north-america]
source_ref: Source/Core/Core/Boot/Boot_BS2Emu.cpp:42
source_ref: "upstream Source/Core/Common/CommonPaths.h:139, source Source/Core/Core/HW/EXI/EXI_DeviceIPL.cpp:108"
```
This excerpt of `emulators/dolphin.yml` leaves out the notes and the other
entries.
Record `source_commit` when profiling: upstream code moves, and a
`source_ref` line number only stays checkable against the commit it was
read at. Profiles written before this field existed carry `profiled_date`
@@ -517,6 +520,15 @@ reports the profile as uncheckable, and the CI job that runs
| `version_basis` | every profile (`tests/test_version_basis.py`) | where `core_version` is read: `file` (`path`, `pattern`, optional `format`), `tag` (the nearest `tag` in the pin's history and its `distance`, `core_version` in git describe's form), `alias` (`of`), `package` (`artifact`, the APK or binary read) or `none`. `repo` names `upstream` when the version is the upstream tree's. `profile_sync --verify-version` reads it again, and a pin advance rewrites `core_version` from it |
| `source_commit` | see above | commit SHA of `source` the code was read at. Anchors every `source_ref` line number |
| `upstream_commit` | no | same, for `upstream` |
| `pin_frozen` | no | why `source_commit` is held on purpose rather than followed. The profile is then judged against its own revision, like one pinned to a superseded tag |
| `data_root` | no | per operating system (`android`, `ios`, `windows`, `macos`, `linux`), the folder the app reads its user data from, under which `standalone_path` destinations sit: `path` as the user finds it (`Android/data/<package>/files`, `%APPDATA%/...`), with the `source_ref` that builds it. Only the systems the app runs on are keyed |
| `upstream_frozen` | no | why `upstream_commit` is held: the port vendors that upstream tree, synced to that release, or merged that commit. The upstream view is read at its own pin and a bump leaves it. Without it, a port that is a GitHub fork of its upstream follows the merge base of the two |
| `prose_written_at` | no | the pin a reader confirmed the prose citations against. When the profile's history shows a text living under several pins, profile_sync takes this one as the revision the text describes, so a bump is no longer refused for a citation already reread. It never overrides a history with a single candidate |
| `version_source` | no | for a project published outside any forge, `{url, pattern}`: the page that announces releases and a regular expression whose first group is the latest version. `profile_sync --check-version` compares it with `core_version`, and `check_freshness --profiles` calls the profile stale when they differ. `no_genuine_copy` names an announced version found only rebuilt or re-signed: the profile stays on the newest unmodified build and is not stale until the page announces a later one |
| `upstream_gone` | no | why the declared upstream can no longer be reached, when that is settled rather than transient. Records the death instead of leaving the profile failing every pass. profile_sync reports it as declared, and reports the contradiction if the forge answers again |
| `source_mirror` | no | a repository carrying the same tree, consulted after `source` and `upstream`. What keeps a profile checkable once its own forge stops answering |
| `source_ref` | no | the code a profile with no file entry rests on: the load routine that opens nothing from outside (`emulators/craft.yml`) |
| `found_by` | no | the default `found_by` of every file entry (see the file entry fields) |
| `source_branch` | no | branch `source` was read from when it is not the default one (a port living on a pull request branch). profile_sync follows it |
| `built_by` | no | platforms that ship this build under its core name, compiled by them or installed from the project's own release (Recalbox's `mame0278`, the GSplus legacy build). On those platforms the name resolves to this profile, never through `all_libretro`: the buildbot does not publish it |
| `bios_mode` | no | `agnostic` when the core takes any file of the right shape in its BIOS folder instead of a fixed name (PCSX2). Packs then carry every matching dump, and a `<bios>.bin` placeholder stands for them |
@@ -576,10 +588,11 @@ system that runs the core there, so libretro Dolphin's `User/GBA/gba_bios.bin`
lands in both `saves/gamecube/User/GBA/` and `saves/wii/User/GBA/` on Batocera.
RetroBat sorts Dolphin's saves by core too, which adds `dolphin-emu/`.
RetroPie, ROCKNIX, AmberELEC and EmuELEC keep saves beside each game, and RomM
runs cores in the browser: no single place exists, so those entries are listed
as omitted (`no_save_directory`), never placed in the BIOS tree where the core
does not look.
A platform whose registry entry has no `save_directory` keeps no single save
place: RetroPie, ROCKNIX, AmberELEC and EmuELEC keep saves beside each game,
and RomM runs cores in the browser. Those entries are listed as omitted
(`no_save_directory`), never placed in the BIOS tree where the core does not
look.
### File entry fields
@@ -593,15 +606,8 @@ does not look.
| `size` | expected size in bytes, or a list when the code accepts several exact sizes |
| `min_size`, `max_size` | size range when the code accepts a range |
| `md5`, `sha1`, `crc32`, `sha256` | expected hashes from source code |
| `known_hash_adler32` | expected Adler-32 hash (Dolphin's DSP ROMs). Pair with `adler32_byteswap` when the code hashes 16-bit byte-swapped data |
| `known_hash_adler32`, `adler32_byteswap` | expected Adler-32 hash (Dolphin's DSP ROMs). Pair with `adler32_byteswap` when the code hashes 16-bit byte-swapped data |
| `adler32_pairs_with`, `adler32_pairs` | the file hashed together with this one and the combinations the code accepts, this file's value first, in table order: Dolphin warns on a DSP ROM pair outside its `known_roms` even when each file passes alone |
| `pin_frozen` | why `source_commit` is held on purpose rather than followed. The profile is then judged against its own revision, like one pinned to a superseded tag |
| `data_root` | per operating system (`android`, `ios`, `windows`, `macos`, `linux`), the folder the app reads its user data from, under which `standalone_path` destinations sit: `path` as the user finds it (`Android/data/<package>/files`, `%APPDATA%/...`), with the `source_ref` that builds it. Only the systems the app runs on are keyed |
| `upstream_frozen` | why `upstream_commit` is held: the port vendors that upstream tree, synced to that release, or merged that commit. The upstream view is read at its own pin and a bump leaves it. Without it, a port that is a GitHub fork of its upstream follows the merge base of the two |
| `prose_written_at` | the pin a reader confirmed the prose citations against. When the profile's history shows a text living under several pins, profile_sync takes this one as the revision the text describes, so a bump is no longer refused for a citation already reread. It never overrides a history with a single candidate |
| `version_source` | for a project published outside any forge, `{url, pattern}`: the page that announces releases and a regular expression whose first group is the latest version. `profile_sync --check-version` compares it with `core_version`, and `check_freshness --profiles` calls the profile stale when they differ. `no_genuine_copy` names an announced version found only rebuilt or re-signed: the profile stays on the newest unmodified build and is not stale until the page announces a later one |
| `upstream_gone` | why the declared upstream can no longer be reached, when that is settled rather than transient. Records the death instead of leaving the profile failing every pass. profile_sync reports it as declared, and reports the contradiction if the forge answers again |
| `source_mirror` | a repository carrying the same tree, consulted after `source` and `upstream`. What keeps a profile checkable once its own forge stops answering |
| `validation` | checks the code performs: `size`, `crc32`, `md5`, `sha1`, `adler32`, `signature`, `crypto`. Can be a list or dict `{core: [...], upstream: [...]}` for divergent checks |
| `aliases` | alternate filenames for the same file |
| `mode` | `libretro`, `standalone`, or `both` |