mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-11-06 01:53:56 -06:00
docs: align profiling and platform pages with code
This commit is contained in:
1 parent
e9ef630b3a
commit
8609810d8f
2 files changed
+106
-67
No files matched your search
+73
-40
@@ -57,6 +57,10 @@ class Scraper(BaseScraper):
|
||||
- `compare_with_config(config) -> ChangeSet` - diffs scraped requirements against
|
||||
an existing platform YAML, returning added/removed/modified entries.
|
||||
- `test_connection() -> bool` - checks if the source URL is reachable.
|
||||
- `lists_files` - `True` by default, and an empty read then stops the scrape.
|
||||
A scraper whose platform lists no file of its own sets it to `False`: RetroPie
|
||||
inherits RetroArch's files and reads only its packages, and MinUI names no
|
||||
file at all.
|
||||
|
||||
Two abstract methods must be implemented:
|
||||
|
||||
@@ -193,7 +197,8 @@ resolved. Three strategies exist:
|
||||
- **Explicit list**: `cores: [beetle_psx, dolphin, ...]` - match by profile key name
|
||||
or by any name in a profile's `cores:`. When a profile's `built_by` names this
|
||||
platform, that profile answers for the name. Used by Batocera, Recalbox, RetroBat,
|
||||
RomM, RetroPie, EmuDeck, RetroDECK, ROCKNIX. A list that names `retroarch` (the last
|
||||
RomM, KNULLI, AmberELEC, EmuELEC, MinUI, NextUI, MyMinUI, RetroPie, EmuDeck,
|
||||
RetroDECK, ROCKNIX. A list that names `retroarch` (the last
|
||||
four) also takes every libretro profile, as `all_libretro` does. An empty list
|
||||
(BizHawk, MiSTer FPGA) resolves no profile, and the pack holds the platform's own
|
||||
files.
|
||||
@@ -291,7 +296,8 @@ that text belongs in a guide field.
|
||||
|
||||
```yaml
|
||||
logo: https://... # SVG or PNG for UI/docs
|
||||
inherits_from: retroarch # inherit systems/cores from another platform
|
||||
inherits_from: retroarch # the parent, for readers: the merge reads
|
||||
# inherits: in the platform YAML
|
||||
case_insensitive_fs: true # if the platform runs on case-insensitive filesystems
|
||||
extract_to: /userdata/bios/ # folder the pack extracts into, shown in the README,
|
||||
# the site and the release notes
|
||||
@@ -379,8 +385,8 @@ AmberELEC and EmuELEC declare `neogeo/` and `ports/` beside `bios/`, and set
|
||||
|
||||
Between the file a user drops in the BIOS folder and the emulator, a
|
||||
platform's own launch code often decides which name is read and where it is
|
||||
handed over. Batocera's DuckStation generator tries 22 names per region, first
|
||||
found wins, and writes the hit into `settings.ini`. That behaviour belongs to
|
||||
handed over. Batocera's DuckStation generator tries a list of names for each
|
||||
region, first found wins, and writes the hit into `settings.ini`. That behaviour belongs to
|
||||
the platform, not to the emulator, so the scraper records it from the
|
||||
platform's code under `launchers:`, never as an alias in an emulator profile.
|
||||
|
||||
@@ -494,10 +500,10 @@ RetroPie, EmuDeck and RetroDECK spell it, and a path from a root
|
||||
### Inheritance
|
||||
|
||||
If the new platform inherits from an existing one (e.g. Lakka inherits RetroArch),
|
||||
set `inherits_from` in the registry AND add `inherits: retroarch` in the platform
|
||||
YAML itself. `load_platform_config()` reads the `inherits:` field from the YAML to
|
||||
merge parent systems and shared groups into the child. The child YAML only needs to
|
||||
declare overrides.
|
||||
add `inherits: retroarch` in the platform YAML. `load_platform_config()` reads that
|
||||
field to merge the parent's systems and shared groups into the child, and the child
|
||||
YAML only declares what differs. The registry's `inherits_from` names the parent
|
||||
for readers and is read by no tool.
|
||||
|
||||
## Step 3: Generate the platform YAML
|
||||
|
||||
@@ -507,9 +513,9 @@ Run the scraper with `--output` to produce the initial platform configuration:
|
||||
python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
|
||||
```
|
||||
|
||||
If a file already exists at the output path, the CLI preserves fields that the
|
||||
scraper does not generate (e.g. `data_directories`, manually added metadata).
|
||||
Only the `systems` section is replaced.
|
||||
If a file already exists at the output path, every top-level key the scraper
|
||||
writes is replaced, and the keys it does not write are kept. A system's
|
||||
`data_directories` is kept as well when the new scrape does not carry one.
|
||||
|
||||
The YAML is the scraper's output and is never edited by hand: the next scrape
|
||||
would undo the edit, and the file would stop describing the platform. Anything
|
||||
@@ -546,37 +552,59 @@ to `verify.py` in the platform-specific verification path.
|
||||
|
||||
Read the platform's source code to understand its exact verification behavior before writing any logic. Batocera's `checkInsideZip` uses `casefold()` for case-insensitive matching. Recalbox supports comma-separated MD5 lists. RomM checks file size before hashing. These details matter: the project replicates native behavior, not an approximation of it.
|
||||
|
||||
## Step 5: Create an exporter (optional)
|
||||
## Step 5: Create an exporter
|
||||
|
||||
Exporters convert truth data back to the platform's native format. They live in
|
||||
`scripts/exporter/` and follow the same auto-discovery pattern (`*_exporter.py`).
|
||||
An exporter writes the platform's own BIOS file back, corrected by what the
|
||||
emulator code shows. Every registered platform has one, unless its registry
|
||||
entry says `native_list: none`: `tests/test_exporters.py` fails otherwise.
|
||||
Exporters live in `scripts/exporter/` and are found the same way as scrapers
|
||||
(`*_exporter.py`).
|
||||
|
||||
### Module contract
|
||||
|
||||
The module must export an `Exporter` class inheriting `BaseExporter`:
|
||||
The module exports an `Exporter` class that extends `BaseExporter`
|
||||
(`scripts/exporter/base_exporter.py`):
|
||||
|
||||
```python
|
||||
from scripts.exporter.base_exporter import BaseExporter
|
||||
from .base_exporter import BaseExporter
|
||||
from .baseline import NativeSystem, Report
|
||||
|
||||
|
||||
class Exporter(BaseExporter):
|
||||
@staticmethod
|
||||
def platform_name() -> str:
|
||||
return "myplatform"
|
||||
|
||||
def export(self, truth_data: dict, output_path: str, scraped_data: dict | None = None) -> None:
|
||||
# Write truth_data in the platform's native format to output_path
|
||||
@staticmethod
|
||||
def native_filename() -> str:
|
||||
return "bios_list.json" # the name the platform gives its file
|
||||
|
||||
@staticmethod
|
||||
def native_sources() -> dict[str, str]:
|
||||
return {"bios_list.json": "https://..."} # the original it patches
|
||||
|
||||
def render(self, systems: dict[str, NativeSystem], report: Report,
|
||||
originals: dict[str, str], scraped: dict | None = None) -> dict[str, str]:
|
||||
# Return {relative output path: file content}
|
||||
...
|
||||
|
||||
def validate(self, truth_data: dict, output_path: str) -> list[str]:
|
||||
def validate(self, systems: dict[str, NativeSystem], produced: dict[str, str]) -> list[str]:
|
||||
# Return a list of issues (empty = valid)
|
||||
...
|
||||
```
|
||||
|
||||
`BaseExporter` provides helper methods:
|
||||
`platform_name`, `native_filename`, `render` and `validate` are abstract. The
|
||||
other methods say what the format can hold, and their defaults fit a plain list
|
||||
of files:
|
||||
|
||||
- `_is_pattern(name)` - True if the filename contains wildcards or placeholders.
|
||||
- `_dest(fe)` - resolve destination path from a file entry dict.
|
||||
- `_display_name(sys_id, scraped_sys)` - convert a system slug to a display name.
|
||||
- `carries()` - the fields the format has a place for (`md5` by default). A
|
||||
correction to a field it cannot hold is not counted as made.
|
||||
- `requires()` - the hash a new entry cannot be written without.
|
||||
- `can_add()` - `False` when an entry is code rather than a row of data
|
||||
(BizHawk), so nothing is added.
|
||||
- `keys_by_system()` - `True` when the platform finds a file by the name of its
|
||||
own system, so an entry filed under one of our system ids is never written.
|
||||
- `needs_original()` - `True` as soon as `native_sources()` names a file.
|
||||
|
||||
### Round-trip validation
|
||||
|
||||
@@ -584,11 +612,10 @@ The scraper and the exporter are checked against each other as
|
||||
[round-trip testing](adding-a-scraper.md#round-trip-testing) describes. The
|
||||
export must not lose anything the platform already declares. If your
|
||||
format carries code as well as data (a script, a C# database, a manifest
|
||||
whose BIOS list sits beside launch configuration), say so with
|
||||
`needs_original()` and patch the platform's original file rather than writing
|
||||
a new one. An
|
||||
exporter that regenerates such a file from BIOS data alone hands the
|
||||
maintainer something that no longer runs.
|
||||
whose BIOS list sits beside launch configuration), name the file in
|
||||
`native_sources()` and patch it rather than writing a new one. An exporter
|
||||
that regenerates such a file from BIOS data alone hands the maintainer
|
||||
something that no longer runs.
|
||||
|
||||
## Step 6: Create a target scraper (optional)
|
||||
|
||||
@@ -643,19 +670,25 @@ cores the pack ships.
|
||||
|
||||
## Step 7: Add install detection (optional)
|
||||
|
||||
The `install` section in `_registry.yml` tells `install.py` how to detect
|
||||
the platform on the user's machine and locate its BIOS directory.
|
||||
The `install.detect` rules in `_registry.yml` say how `install.py` finds the
|
||||
platform on the user's machine and where its BIOS folder is. The installer
|
||||
runs before it fetches any manifest, so it carries the same rules in its own
|
||||
tables, and `tests/test_install.py` holds the two equal.
|
||||
|
||||
Three detection methods are available:
|
||||
| Method | Finds the platform when | Fields |
|
||||
|--------|-------------------------|--------|
|
||||
| `config_file` | a key of a config file names the folder | `config`, `parse_key`, optionally `bios_subdir` or `bios_path` |
|
||||
| `path_exists` | a folder exists | `path`, optionally `bios_path` |
|
||||
| `file_exists` | a file exists | `file`, `bios_path`, optionally `name`, `config`, `home` |
|
||||
| `os_release` | `/etc/os-release` holds one of the `match` pairs | `match`, `name`, `bios_path`, optionally `config`, `home` |
|
||||
| `file_matches` | a file's text matches `pattern` | `file`, `pattern`, `name`, `bios_path` |
|
||||
| `executable` | a portable install's program is found | `file` (glob patterns), `bios_subdir` |
|
||||
| `registry` | a Windows registry value names the install folder | `key`, `value`, `bios_subdir` |
|
||||
| `card_marker` | a file on the SD card starts with a prefix | `file`, `first_line_prefix` |
|
||||
|
||||
| Method | Description | Fields |
|
||||
|--------|-------------|--------|
|
||||
| `config_file` | Parse a key from a config file | `config`, `parse_key`, optionally `bios_subdir` |
|
||||
| `path_exists` | Check if a directory exists | `path`, optionally `bios_path` |
|
||||
| `file_exists` | Check if a file exists | `file`, optionally `bios_path` |
|
||||
|
||||
Each entry is scoped to an OS (`linux`, `darwin`, `windows`). Multiple entries
|
||||
per OS are tried in order.
|
||||
Each rule names its `os`: `linux`, `darwin`, `windows`, or `any` for a card
|
||||
read from another computer. Rules for one OS are tried in order.
|
||||
`derives_from` names the system a rule's system is built on.
|
||||
|
||||
## Step 8: Validate the full pipeline
|
||||
|
||||
@@ -708,6 +741,6 @@ Verification is not optional. A platform that passes `pipeline.py` today may bre
|
||||
- [ ] Guide added to `platforms/_guides.yml`
|
||||
- [ ] Platform YAML generated and verified
|
||||
- [ ] `python scripts/pipeline.py --offline` passes
|
||||
- [ ] Exporter in `scripts/exporter/<name>_exporter.py` (if applicable)
|
||||
- [ ] Exporter in `scripts/exporter/<name>_exporter.py`, unless the registry says `native_list: none`
|
||||
- [ ] Target scraper in `scripts/scraper/targets/<name>_targets_scraper.py` (if applicable)
|
||||
- [ ] Install detection entries in `_registry.yml` (if applicable)
|
||||
+33
-27
@@ -163,9 +163,9 @@ and consistent across all profiles.
|
||||
When a core has HLE (high-level emulation), code of its own that stands
|
||||
in for the file, mark the entry `hle_fallback: true`. The flag records the
|
||||
stand-in. It does not decide `required`, which still comes from what the
|
||||
code does without the file: most of the 778 entries that carry the flag
|
||||
are optional, and 13 are required, among them Beetle PSX's three region
|
||||
BIOS (`emulators/beetle_psx.yml`). The file still ships in packs, and
|
||||
code does without the file: most entries that carry the flag are optional,
|
||||
and a few are required, among them Yabause's `saturn_bios.bin`
|
||||
(`emulators/yabause.yml`). The file still ships in packs, and
|
||||
verification reports its absence at INFO whatever `required` says.
|
||||
|
||||
MAME is read the same way. A missing ROM is fatal only when it is neither
|
||||
@@ -275,30 +275,33 @@ what the user actually needs to provide.
|
||||
emulator: Dolphin
|
||||
type: standalone + libretro
|
||||
core_classification: community_fork
|
||||
source: https://github.com/libretro/dolphin
|
||||
upstream: https://github.com/dolphin-emu/dolphin
|
||||
profiled_date: 2026-03-25
|
||||
source: "https://github.com/libretro/dolphin"
|
||||
upstream: "https://github.com/dolphin-emu/dolphin"
|
||||
profiled_date: "2026-03-23"
|
||||
source_commit: "3aac3f052f733f8df2f1616843a4d2e14002ed11"
|
||||
upstream_commit: "0d93fc035539f25790deca638d06ee3b3b45ee64"
|
||||
core_version: "2609-437-g3aac3f0"
|
||||
version_basis:
|
||||
kind: tag
|
||||
tag: "2609"
|
||||
distance: 437
|
||||
systems:
|
||||
- nintendo-gamecube
|
||||
- nintendo-wii
|
||||
systems: [nintendo-gamecube, nintendo-wii, sega-triforce]
|
||||
|
||||
files:
|
||||
- name: GC/USA/IPL.bin
|
||||
- name: "IPL.bin"
|
||||
path: "dolphin-emu/Sys/GC/USA/IPL.bin"
|
||||
standalone_path: "GC/USA/IPL.bin"
|
||||
system: nintendo-gamecube
|
||||
region: [north-america, latin-america]
|
||||
size: 2097152
|
||||
required: false
|
||||
hle_fallback: true
|
||||
size: 2097152
|
||||
validation: [size, adler32]
|
||||
known_hash_adler32: 0x4f1f6f5c
|
||||
region: [north-america]
|
||||
source_ref: Source/Core/Core/Boot/Boot_BS2Emu.cpp:42
|
||||
source_ref: "upstream Source/Core/Common/CommonPaths.h:139, source Source/Core/Core/HW/EXI/EXI_DeviceIPL.cpp:108"
|
||||
```
|
||||
|
||||
This excerpt of `emulators/dolphin.yml` leaves out the notes and the other
|
||||
entries.
|
||||
|
||||
Record `source_commit` when profiling: upstream code moves, and a
|
||||
`source_ref` line number only stays checkable against the commit it was
|
||||
read at. Profiles written before this field existed carry `profiled_date`
|
||||
@@ -517,6 +520,15 @@ reports the profile as uncheckable, and the CI job that runs
|
||||
| `version_basis` | every profile (`tests/test_version_basis.py`) | where `core_version` is read: `file` (`path`, `pattern`, optional `format`), `tag` (the nearest `tag` in the pin's history and its `distance`, `core_version` in git describe's form), `alias` (`of`), `package` (`artifact`, the APK or binary read) or `none`. `repo` names `upstream` when the version is the upstream tree's. `profile_sync --verify-version` reads it again, and a pin advance rewrites `core_version` from it |
|
||||
| `source_commit` | see above | commit SHA of `source` the code was read at. Anchors every `source_ref` line number |
|
||||
| `upstream_commit` | no | same, for `upstream` |
|
||||
| `pin_frozen` | no | why `source_commit` is held on purpose rather than followed. The profile is then judged against its own revision, like one pinned to a superseded tag |
|
||||
| `data_root` | no | per operating system (`android`, `ios`, `windows`, `macos`, `linux`), the folder the app reads its user data from, under which `standalone_path` destinations sit: `path` as the user finds it (`Android/data/<package>/files`, `%APPDATA%/...`), with the `source_ref` that builds it. Only the systems the app runs on are keyed |
|
||||
| `upstream_frozen` | no | why `upstream_commit` is held: the port vendors that upstream tree, synced to that release, or merged that commit. The upstream view is read at its own pin and a bump leaves it. Without it, a port that is a GitHub fork of its upstream follows the merge base of the two |
|
||||
| `prose_written_at` | no | the pin a reader confirmed the prose citations against. When the profile's history shows a text living under several pins, profile_sync takes this one as the revision the text describes, so a bump is no longer refused for a citation already reread. It never overrides a history with a single candidate |
|
||||
| `version_source` | no | for a project published outside any forge, `{url, pattern}`: the page that announces releases and a regular expression whose first group is the latest version. `profile_sync --check-version` compares it with `core_version`, and `check_freshness --profiles` calls the profile stale when they differ. `no_genuine_copy` names an announced version found only rebuilt or re-signed: the profile stays on the newest unmodified build and is not stale until the page announces a later one |
|
||||
| `upstream_gone` | no | why the declared upstream can no longer be reached, when that is settled rather than transient. Records the death instead of leaving the profile failing every pass. profile_sync reports it as declared, and reports the contradiction if the forge answers again |
|
||||
| `source_mirror` | no | a repository carrying the same tree, consulted after `source` and `upstream`. What keeps a profile checkable once its own forge stops answering |
|
||||
| `source_ref` | no | the code a profile with no file entry rests on: the load routine that opens nothing from outside (`emulators/craft.yml`) |
|
||||
| `found_by` | no | the default `found_by` of every file entry (see the file entry fields) |
|
||||
| `source_branch` | no | branch `source` was read from when it is not the default one (a port living on a pull request branch). profile_sync follows it |
|
||||
| `built_by` | no | platforms that ship this build under its core name, compiled by them or installed from the project's own release (Recalbox's `mame0278`, the GSplus legacy build). On those platforms the name resolves to this profile, never through `all_libretro`: the buildbot does not publish it |
|
||||
| `bios_mode` | no | `agnostic` when the core takes any file of the right shape in its BIOS folder instead of a fixed name (PCSX2). Packs then carry every matching dump, and a `<bios>.bin` placeholder stands for them |
|
||||
@@ -576,10 +588,11 @@ system that runs the core there, so libretro Dolphin's `User/GBA/gba_bios.bin`
|
||||
lands in both `saves/gamecube/User/GBA/` and `saves/wii/User/GBA/` on Batocera.
|
||||
RetroBat sorts Dolphin's saves by core too, which adds `dolphin-emu/`.
|
||||
|
||||
RetroPie, ROCKNIX, AmberELEC and EmuELEC keep saves beside each game, and RomM
|
||||
runs cores in the browser: no single place exists, so those entries are listed
|
||||
as omitted (`no_save_directory`), never placed in the BIOS tree where the core
|
||||
does not look.
|
||||
A platform whose registry entry has no `save_directory` keeps no single save
|
||||
place: RetroPie, ROCKNIX, AmberELEC and EmuELEC keep saves beside each game,
|
||||
and RomM runs cores in the browser. Those entries are listed as omitted
|
||||
(`no_save_directory`), never placed in the BIOS tree where the core does not
|
||||
look.
|
||||
|
||||
### File entry fields
|
||||
|
||||
@@ -593,15 +606,8 @@ does not look.
|
||||
| `size` | expected size in bytes, or a list when the code accepts several exact sizes |
|
||||
| `min_size`, `max_size` | size range when the code accepts a range |
|
||||
| `md5`, `sha1`, `crc32`, `sha256` | expected hashes from source code |
|
||||
| `known_hash_adler32` | expected Adler-32 hash (Dolphin's DSP ROMs). Pair with `adler32_byteswap` when the code hashes 16-bit byte-swapped data |
|
||||
| `known_hash_adler32`, `adler32_byteswap` | expected Adler-32 hash (Dolphin's DSP ROMs). Pair with `adler32_byteswap` when the code hashes 16-bit byte-swapped data |
|
||||
| `adler32_pairs_with`, `adler32_pairs` | the file hashed together with this one and the combinations the code accepts, this file's value first, in table order: Dolphin warns on a DSP ROM pair outside its `known_roms` even when each file passes alone |
|
||||
| `pin_frozen` | why `source_commit` is held on purpose rather than followed. The profile is then judged against its own revision, like one pinned to a superseded tag |
|
||||
| `data_root` | per operating system (`android`, `ios`, `windows`, `macos`, `linux`), the folder the app reads its user data from, under which `standalone_path` destinations sit: `path` as the user finds it (`Android/data/<package>/files`, `%APPDATA%/...`), with the `source_ref` that builds it. Only the systems the app runs on are keyed |
|
||||
| `upstream_frozen` | why `upstream_commit` is held: the port vendors that upstream tree, synced to that release, or merged that commit. The upstream view is read at its own pin and a bump leaves it. Without it, a port that is a GitHub fork of its upstream follows the merge base of the two |
|
||||
| `prose_written_at` | the pin a reader confirmed the prose citations against. When the profile's history shows a text living under several pins, profile_sync takes this one as the revision the text describes, so a bump is no longer refused for a citation already reread. It never overrides a history with a single candidate |
|
||||
| `version_source` | for a project published outside any forge, `{url, pattern}`: the page that announces releases and a regular expression whose first group is the latest version. `profile_sync --check-version` compares it with `core_version`, and `check_freshness --profiles` calls the profile stale when they differ. `no_genuine_copy` names an announced version found only rebuilt or re-signed: the profile stays on the newest unmodified build and is not stale until the page announces a later one |
|
||||
| `upstream_gone` | why the declared upstream can no longer be reached, when that is settled rather than transient. Records the death instead of leaving the profile failing every pass. profile_sync reports it as declared, and reports the contradiction if the forge answers again |
|
||||
| `source_mirror` | a repository carrying the same tree, consulted after `source` and `upstream`. What keeps a profile checkable once its own forge stops answering |
|
||||
| `validation` | checks the code performs: `size`, `crc32`, `md5`, `sha1`, `adler32`, `signature`, `crypto`. Can be a list or dict `{core: [...], upstream: [...]}` for divergent checks |
|
||||
| `aliases` | alternate filenames for the same file |
|
||||
| `mode` | `libretro`, `standalone`, or `both` |
|
||||
|
||||
Reference in new issue
Block a user