From 8609810d8fb2da59dd69a1e11471a56df7117420 Mon Sep 17 00:00:00 2001 From: Abdessamad Derraz <3028866+Abdess@users.noreply.github.com> Date: Tue, 3 Nov 2026 10:08:42 +0100 Subject: [PATCH] docs: align profiling and platform pages with code --- wiki/adding-a-platform.md | 113 ++++++++++++++++++++++++-------------- wiki/profiling.md | 60 +++++++++++--------- 2 files changed, 106 insertions(+), 67 deletions(-) diff --git a/wiki/adding-a-platform.md b/wiki/adding-a-platform.md index bed3b080a..ab5aabc6c 100644 --- a/wiki/adding-a-platform.md +++ b/wiki/adding-a-platform.md @@ -57,6 +57,10 @@ class Scraper(BaseScraper): - `compare_with_config(config) -> ChangeSet` - diffs scraped requirements against an existing platform YAML, returning added/removed/modified entries. - `test_connection() -> bool` - checks if the source URL is reachable. +- `lists_files` - `True` by default, and an empty read then stops the scrape. + A scraper whose platform lists no file of its own sets it to `False`: RetroPie + inherits RetroArch's files and reads only its packages, and MinUI names no + file at all. Two abstract methods must be implemented: @@ -193,7 +197,8 @@ resolved. Three strategies exist: - **Explicit list**: `cores: [beetle_psx, dolphin, ...]` - match by profile key name or by any name in a profile's `cores:`. When a profile's `built_by` names this platform, that profile answers for the name. Used by Batocera, Recalbox, RetroBat, - RomM, RetroPie, EmuDeck, RetroDECK, ROCKNIX. A list that names `retroarch` (the last + RomM, KNULLI, AmberELEC, EmuELEC, MinUI, NextUI, MyMinUI, RetroPie, EmuDeck, + RetroDECK, ROCKNIX. A list that names `retroarch` (the last four) also takes every libretro profile, as `all_libretro` does. An empty list (BizHawk, MiSTer FPGA) resolves no profile, and the pack holds the platform's own files. @@ -291,7 +296,8 @@ that text belongs in a guide field. ```yaml logo: https://... # SVG or PNG for UI/docs - inherits_from: retroarch # inherit systems/cores from another platform + inherits_from: retroarch # the parent, for readers: the merge reads + # inherits: in the platform YAML case_insensitive_fs: true # if the platform runs on case-insensitive filesystems extract_to: /userdata/bios/ # folder the pack extracts into, shown in the README, # the site and the release notes @@ -379,8 +385,8 @@ AmberELEC and EmuELEC declare `neogeo/` and `ports/` beside `bios/`, and set Between the file a user drops in the BIOS folder and the emulator, a platform's own launch code often decides which name is read and where it is -handed over. Batocera's DuckStation generator tries 22 names per region, first -found wins, and writes the hit into `settings.ini`. That behaviour belongs to +handed over. Batocera's DuckStation generator tries a list of names for each +region, first found wins, and writes the hit into `settings.ini`. That behaviour belongs to the platform, not to the emulator, so the scraper records it from the platform's code under `launchers:`, never as an alias in an emulator profile. @@ -494,10 +500,10 @@ RetroPie, EmuDeck and RetroDECK spell it, and a path from a root ### Inheritance If the new platform inherits from an existing one (e.g. Lakka inherits RetroArch), -set `inherits_from` in the registry AND add `inherits: retroarch` in the platform -YAML itself. `load_platform_config()` reads the `inherits:` field from the YAML to -merge parent systems and shared groups into the child. The child YAML only needs to -declare overrides. +add `inherits: retroarch` in the platform YAML. `load_platform_config()` reads that +field to merge the parent's systems and shared groups into the child, and the child +YAML only declares what differs. The registry's `inherits_from` names the parent +for readers and is read by no tool. ## Step 3: Generate the platform YAML @@ -507,9 +513,9 @@ Run the scraper with `--output` to produce the initial platform configuration: python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml ``` -If a file already exists at the output path, the CLI preserves fields that the -scraper does not generate (e.g. `data_directories`, manually added metadata). -Only the `systems` section is replaced. +If a file already exists at the output path, every top-level key the scraper +writes is replaced, and the keys it does not write are kept. A system's +`data_directories` is kept as well when the new scrape does not carry one. The YAML is the scraper's output and is never edited by hand: the next scrape would undo the edit, and the file would stop describing the platform. Anything @@ -546,37 +552,59 @@ to `verify.py` in the platform-specific verification path. Read the platform's source code to understand its exact verification behavior before writing any logic. Batocera's `checkInsideZip` uses `casefold()` for case-insensitive matching. Recalbox supports comma-separated MD5 lists. RomM checks file size before hashing. These details matter: the project replicates native behavior, not an approximation of it. -## Step 5: Create an exporter (optional) +## Step 5: Create an exporter -Exporters convert truth data back to the platform's native format. They live in -`scripts/exporter/` and follow the same auto-discovery pattern (`*_exporter.py`). +An exporter writes the platform's own BIOS file back, corrected by what the +emulator code shows. Every registered platform has one, unless its registry +entry says `native_list: none`: `tests/test_exporters.py` fails otherwise. +Exporters live in `scripts/exporter/` and are found the same way as scrapers +(`*_exporter.py`). ### Module contract -The module must export an `Exporter` class inheriting `BaseExporter`: +The module exports an `Exporter` class that extends `BaseExporter` +(`scripts/exporter/base_exporter.py`): ```python -from scripts.exporter.base_exporter import BaseExporter +from .base_exporter import BaseExporter +from .baseline import NativeSystem, Report + class Exporter(BaseExporter): @staticmethod def platform_name() -> str: return "myplatform" - def export(self, truth_data: dict, output_path: str, scraped_data: dict | None = None) -> None: - # Write truth_data in the platform's native format to output_path + @staticmethod + def native_filename() -> str: + return "bios_list.json" # the name the platform gives its file + + @staticmethod + def native_sources() -> dict[str, str]: + return {"bios_list.json": "https://..."} # the original it patches + + def render(self, systems: dict[str, NativeSystem], report: Report, + originals: dict[str, str], scraped: dict | None = None) -> dict[str, str]: + # Return {relative output path: file content} ... - def validate(self, truth_data: dict, output_path: str) -> list[str]: + def validate(self, systems: dict[str, NativeSystem], produced: dict[str, str]) -> list[str]: # Return a list of issues (empty = valid) ... ``` -`BaseExporter` provides helper methods: +`platform_name`, `native_filename`, `render` and `validate` are abstract. The +other methods say what the format can hold, and their defaults fit a plain list +of files: -- `_is_pattern(name)` - True if the filename contains wildcards or placeholders. -- `_dest(fe)` - resolve destination path from a file entry dict. -- `_display_name(sys_id, scraped_sys)` - convert a system slug to a display name. +- `carries()` - the fields the format has a place for (`md5` by default). A + correction to a field it cannot hold is not counted as made. +- `requires()` - the hash a new entry cannot be written without. +- `can_add()` - `False` when an entry is code rather than a row of data + (BizHawk), so nothing is added. +- `keys_by_system()` - `True` when the platform finds a file by the name of its + own system, so an entry filed under one of our system ids is never written. +- `needs_original()` - `True` as soon as `native_sources()` names a file. ### Round-trip validation @@ -584,11 +612,10 @@ The scraper and the exporter are checked against each other as [round-trip testing](adding-a-scraper.md#round-trip-testing) describes. The export must not lose anything the platform already declares. If your format carries code as well as data (a script, a C# database, a manifest -whose BIOS list sits beside launch configuration), say so with -`needs_original()` and patch the platform's original file rather than writing -a new one. An -exporter that regenerates such a file from BIOS data alone hands the -maintainer something that no longer runs. +whose BIOS list sits beside launch configuration), name the file in +`native_sources()` and patch it rather than writing a new one. An exporter +that regenerates such a file from BIOS data alone hands the maintainer +something that no longer runs. ## Step 6: Create a target scraper (optional) @@ -643,19 +670,25 @@ cores the pack ships. ## Step 7: Add install detection (optional) -The `install` section in `_registry.yml` tells `install.py` how to detect -the platform on the user's machine and locate its BIOS directory. +The `install.detect` rules in `_registry.yml` say how `install.py` finds the +platform on the user's machine and where its BIOS folder is. The installer +runs before it fetches any manifest, so it carries the same rules in its own +tables, and `tests/test_install.py` holds the two equal. -Three detection methods are available: +| Method | Finds the platform when | Fields | +|--------|-------------------------|--------| +| `config_file` | a key of a config file names the folder | `config`, `parse_key`, optionally `bios_subdir` or `bios_path` | +| `path_exists` | a folder exists | `path`, optionally `bios_path` | +| `file_exists` | a file exists | `file`, `bios_path`, optionally `name`, `config`, `home` | +| `os_release` | `/etc/os-release` holds one of the `match` pairs | `match`, `name`, `bios_path`, optionally `config`, `home` | +| `file_matches` | a file's text matches `pattern` | `file`, `pattern`, `name`, `bios_path` | +| `executable` | a portable install's program is found | `file` (glob patterns), `bios_subdir` | +| `registry` | a Windows registry value names the install folder | `key`, `value`, `bios_subdir` | +| `card_marker` | a file on the SD card starts with a prefix | `file`, `first_line_prefix` | -| Method | Description | Fields | -|--------|-------------|--------| -| `config_file` | Parse a key from a config file | `config`, `parse_key`, optionally `bios_subdir` | -| `path_exists` | Check if a directory exists | `path`, optionally `bios_path` | -| `file_exists` | Check if a file exists | `file`, optionally `bios_path` | - -Each entry is scoped to an OS (`linux`, `darwin`, `windows`). Multiple entries -per OS are tried in order. +Each rule names its `os`: `linux`, `darwin`, `windows`, or `any` for a card +read from another computer. Rules for one OS are tried in order. +`derives_from` names the system a rule's system is built on. ## Step 8: Validate the full pipeline @@ -708,6 +741,6 @@ Verification is not optional. A platform that passes `pipeline.py` today may bre - [ ] Guide added to `platforms/_guides.yml` - [ ] Platform YAML generated and verified - [ ] `python scripts/pipeline.py --offline` passes -- [ ] Exporter in `scripts/exporter/_exporter.py` (if applicable) +- [ ] Exporter in `scripts/exporter/_exporter.py`, unless the registry says `native_list: none` - [ ] Target scraper in `scripts/scraper/targets/_targets_scraper.py` (if applicable) - [ ] Install detection entries in `_registry.yml` (if applicable) diff --git a/wiki/profiling.md b/wiki/profiling.md index 2bc88aab1..0b69e0423 100644 --- a/wiki/profiling.md +++ b/wiki/profiling.md @@ -163,9 +163,9 @@ and consistent across all profiles. When a core has HLE (high-level emulation), code of its own that stands in for the file, mark the entry `hle_fallback: true`. The flag records the stand-in. It does not decide `required`, which still comes from what the -code does without the file: most of the 778 entries that carry the flag -are optional, and 13 are required, among them Beetle PSX's three region -BIOS (`emulators/beetle_psx.yml`). The file still ships in packs, and +code does without the file: most entries that carry the flag are optional, +and a few are required, among them Yabause's `saturn_bios.bin` +(`emulators/yabause.yml`). The file still ships in packs, and verification reports its absence at INFO whatever `required` says. MAME is read the same way. A missing ROM is fatal only when it is neither @@ -275,30 +275,33 @@ what the user actually needs to provide. emulator: Dolphin type: standalone + libretro core_classification: community_fork -source: https://github.com/libretro/dolphin -upstream: https://github.com/dolphin-emu/dolphin -profiled_date: 2026-03-25 +source: "https://github.com/libretro/dolphin" +upstream: "https://github.com/dolphin-emu/dolphin" +profiled_date: "2026-03-23" +source_commit: "3aac3f052f733f8df2f1616843a4d2e14002ed11" +upstream_commit: "0d93fc035539f25790deca638d06ee3b3b45ee64" core_version: "2609-437-g3aac3f0" version_basis: kind: tag tag: "2609" distance: 437 -systems: - - nintendo-gamecube - - nintendo-wii +systems: [nintendo-gamecube, nintendo-wii, sega-triforce] files: - - name: GC/USA/IPL.bin + - name: "IPL.bin" + path: "dolphin-emu/Sys/GC/USA/IPL.bin" + standalone_path: "GC/USA/IPL.bin" system: nintendo-gamecube + region: [north-america, latin-america] + size: 2097152 required: false hle_fallback: true - size: 2097152 - validation: [size, adler32] - known_hash_adler32: 0x4f1f6f5c - region: [north-america] - source_ref: Source/Core/Core/Boot/Boot_BS2Emu.cpp:42 + source_ref: "upstream Source/Core/Common/CommonPaths.h:139, source Source/Core/Core/HW/EXI/EXI_DeviceIPL.cpp:108" ``` +This excerpt of `emulators/dolphin.yml` leaves out the notes and the other +entries. + Record `source_commit` when profiling: upstream code moves, and a `source_ref` line number only stays checkable against the commit it was read at. Profiles written before this field existed carry `profiled_date` @@ -517,6 +520,15 @@ reports the profile as uncheckable, and the CI job that runs | `version_basis` | every profile (`tests/test_version_basis.py`) | where `core_version` is read: `file` (`path`, `pattern`, optional `format`), `tag` (the nearest `tag` in the pin's history and its `distance`, `core_version` in git describe's form), `alias` (`of`), `package` (`artifact`, the APK or binary read) or `none`. `repo` names `upstream` when the version is the upstream tree's. `profile_sync --verify-version` reads it again, and a pin advance rewrites `core_version` from it | | `source_commit` | see above | commit SHA of `source` the code was read at. Anchors every `source_ref` line number | | `upstream_commit` | no | same, for `upstream` | +| `pin_frozen` | no | why `source_commit` is held on purpose rather than followed. The profile is then judged against its own revision, like one pinned to a superseded tag | +| `data_root` | no | per operating system (`android`, `ios`, `windows`, `macos`, `linux`), the folder the app reads its user data from, under which `standalone_path` destinations sit: `path` as the user finds it (`Android/data//files`, `%APPDATA%/...`), with the `source_ref` that builds it. Only the systems the app runs on are keyed | +| `upstream_frozen` | no | why `upstream_commit` is held: the port vendors that upstream tree, synced to that release, or merged that commit. The upstream view is read at its own pin and a bump leaves it. Without it, a port that is a GitHub fork of its upstream follows the merge base of the two | +| `prose_written_at` | no | the pin a reader confirmed the prose citations against. When the profile's history shows a text living under several pins, profile_sync takes this one as the revision the text describes, so a bump is no longer refused for a citation already reread. It never overrides a history with a single candidate | +| `version_source` | no | for a project published outside any forge, `{url, pattern}`: the page that announces releases and a regular expression whose first group is the latest version. `profile_sync --check-version` compares it with `core_version`, and `check_freshness --profiles` calls the profile stale when they differ. `no_genuine_copy` names an announced version found only rebuilt or re-signed: the profile stays on the newest unmodified build and is not stale until the page announces a later one | +| `upstream_gone` | no | why the declared upstream can no longer be reached, when that is settled rather than transient. Records the death instead of leaving the profile failing every pass. profile_sync reports it as declared, and reports the contradiction if the forge answers again | +| `source_mirror` | no | a repository carrying the same tree, consulted after `source` and `upstream`. What keeps a profile checkable once its own forge stops answering | +| `source_ref` | no | the code a profile with no file entry rests on: the load routine that opens nothing from outside (`emulators/craft.yml`) | +| `found_by` | no | the default `found_by` of every file entry (see the file entry fields) | | `source_branch` | no | branch `source` was read from when it is not the default one (a port living on a pull request branch). profile_sync follows it | | `built_by` | no | platforms that ship this build under its core name, compiled by them or installed from the project's own release (Recalbox's `mame0278`, the GSplus legacy build). On those platforms the name resolves to this profile, never through `all_libretro`: the buildbot does not publish it | | `bios_mode` | no | `agnostic` when the core takes any file of the right shape in its BIOS folder instead of a fixed name (PCSX2). Packs then carry every matching dump, and a `.bin` placeholder stands for them | @@ -576,10 +588,11 @@ system that runs the core there, so libretro Dolphin's `User/GBA/gba_bios.bin` lands in both `saves/gamecube/User/GBA/` and `saves/wii/User/GBA/` on Batocera. RetroBat sorts Dolphin's saves by core too, which adds `dolphin-emu/`. -RetroPie, ROCKNIX, AmberELEC and EmuELEC keep saves beside each game, and RomM -runs cores in the browser: no single place exists, so those entries are listed -as omitted (`no_save_directory`), never placed in the BIOS tree where the core -does not look. +A platform whose registry entry has no `save_directory` keeps no single save +place: RetroPie, ROCKNIX, AmberELEC and EmuELEC keep saves beside each game, +and RomM runs cores in the browser. Those entries are listed as omitted +(`no_save_directory`), never placed in the BIOS tree where the core does not +look. ### File entry fields @@ -593,15 +606,8 @@ does not look. | `size` | expected size in bytes, or a list when the code accepts several exact sizes | | `min_size`, `max_size` | size range when the code accepts a range | | `md5`, `sha1`, `crc32`, `sha256` | expected hashes from source code | -| `known_hash_adler32` | expected Adler-32 hash (Dolphin's DSP ROMs). Pair with `adler32_byteswap` when the code hashes 16-bit byte-swapped data | +| `known_hash_adler32`, `adler32_byteswap` | expected Adler-32 hash (Dolphin's DSP ROMs). Pair with `adler32_byteswap` when the code hashes 16-bit byte-swapped data | | `adler32_pairs_with`, `adler32_pairs` | the file hashed together with this one and the combinations the code accepts, this file's value first, in table order: Dolphin warns on a DSP ROM pair outside its `known_roms` even when each file passes alone | -| `pin_frozen` | why `source_commit` is held on purpose rather than followed. The profile is then judged against its own revision, like one pinned to a superseded tag | -| `data_root` | per operating system (`android`, `ios`, `windows`, `macos`, `linux`), the folder the app reads its user data from, under which `standalone_path` destinations sit: `path` as the user finds it (`Android/data//files`, `%APPDATA%/...`), with the `source_ref` that builds it. Only the systems the app runs on are keyed | -| `upstream_frozen` | why `upstream_commit` is held: the port vendors that upstream tree, synced to that release, or merged that commit. The upstream view is read at its own pin and a bump leaves it. Without it, a port that is a GitHub fork of its upstream follows the merge base of the two | -| `prose_written_at` | the pin a reader confirmed the prose citations against. When the profile's history shows a text living under several pins, profile_sync takes this one as the revision the text describes, so a bump is no longer refused for a citation already reread. It never overrides a history with a single candidate | -| `version_source` | for a project published outside any forge, `{url, pattern}`: the page that announces releases and a regular expression whose first group is the latest version. `profile_sync --check-version` compares it with `core_version`, and `check_freshness --profiles` calls the profile stale when they differ. `no_genuine_copy` names an announced version found only rebuilt or re-signed: the profile stays on the newest unmodified build and is not stale until the page announces a later one | -| `upstream_gone` | why the declared upstream can no longer be reached, when that is settled rather than transient. Records the death instead of leaving the profile failing every pass. profile_sync reports it as declared, and reports the contradiction if the forge answers again | -| `source_mirror` | a repository carrying the same tree, consulted after `source` and `upstream`. What keeps a profile checkable once its own forge stops answering | | `validation` | checks the code performs: `size`, `crc32`, `md5`, `sha1`, `adler32`, `signature`, `crypto`. Can be a list or dict `{core: [...], upstream: [...]}` for divergent checks | | `aliases` | alternate filenames for the same file | | `mode` | `libretro`, `standalone`, or `both` |