diff --git a/wiki/measurements.md b/wiki/measurements.md index 113b26b25..4c48a41cc 100644 --- a/wiki/measurements.md +++ b/wiki/measurements.md @@ -2,13 +2,15 @@ level: reference order: 70 group: Data -terms: measurement, benchmark, tested, core probe, real conditions, method +terms: measurement, proof, tested, core probe, real conditions, method --- # How emulators are tested -Some pages say what an emulator does with a file. Where we ran it to check, -the emulator's page shows a **Tested** block. Every result is on the +We run an emulator only when a question needs a real answer. That is an +issue, a fix we send to a project, or a check before a release. + +A page that relies on a run says so in a **Proof** line. Every run is on the [Measurements](../measurements.md) page. ## What a test does @@ -16,11 +18,7 @@ the emulator's page shows a **Tested** block. Every result is on the - A script runs the emulator with no window and no sound. - It starts a game and runs it for a set time, at least twice. - It compares the screens with ones we checked by eye. - -## Your BIOS folder { #packs } - -We also test the files as you install them, for RetroArch, Batocera and -RetroBat, then start a game. The setup's page shows the result under **Tested**. +- It reads the emulator's log to see which BIOS it ran. ## What a result says @@ -33,7 +31,13 @@ RetroBat, then start a game. The setup's page shows the result under **Tested**. | hang | the screen stayed blank, or the run never ended | | unknown | none of these could be shown | | no verdict | an older test that only compared screens | -| ms per frame | a speed test | + +## What a result does not prove + +- It shows the first minutes of one game, not the whole game. +- It sees the files the emulator opened on this path, not every path. +- Some emulators run their own BIOS when yours is refused. The game still works. +- The app you play in may change settings the test left as they were. ## How a file is proven @@ -44,27 +48,12 @@ file under the BIOS name: zeros, half the file, or another region's BIOS. - **Built-in stand-in**: no BIOS still works. - **Checked**: a wrong file is refused. -## When tests run - -A maintainer starts a test run. A clear result updates the emulator's page. -Anything else waits on the [Measurements](../measurements.md) page for someone -to read the code. - -## What a result keeps - -Each result names its method, the computer it ran on and the exact build of -the emulator. Its inputs are kept by fingerprint. A result holds for that -build and those inputs only. - -A speed figure depends on the computer and how busy it was. It is never -shown without them. - ## Where results are kept Each result is a full record in the [`measurements/`](https://github.com/Abdess/retrobios/tree/main/measurements) -folder of this project. A record is never changed once added. A new run is -added beside it, and a correction names the record it replaces. +folder of this project. It names its method, the computer and the exact build +of the emulator. A record is never changed once added. The results so far ran on these computers: diff --git a/wiki/release-process.md b/wiki/release-process.md index f3c73dd90..7ccf74e0c 100644 --- a/wiki/release-process.md +++ b/wiki/release-process.md @@ -287,6 +287,9 @@ gh release list --json tagName,createdAt \ | tail -n +2 | while read tag; do gh release delete "$tag" --yes --cleanup-tag; done ``` +Before a release, since 2026-10-28, a maintainer runs `python scripts/pack_probe.py run +--platform ` on the platforms that get a pack, and reads its findings before cutting. + ## Which platforms get a pack { #which-platforms-get-a-pack } `release_pack: true` in `platforms/_registry.yml` is the list. `generate_pack.py diff --git a/wiki/testing-guide.md b/wiki/testing-guide.md index 290932592..0bfe1dec7 100644 --- a/wiki/testing-guide.md +++ b/wiki/testing-guide.md @@ -57,11 +57,11 @@ Every module in `tests/`, by the area it guards. | Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_app`, `test_install_from`, `test_install_offline`, `test_install_selection`, `test_install_tui` | | Handheld launchers | a pty as the console, fake installer and firmware tools, stubbed PortMaster dialogs | `test_handheld_launchers`, `test_port` | | Running cores | parsers on fixed inputs, a libretro core built from C in the test when a compiler exists | `test_core_probe` | -| Measurement records | synthetic record files, the committed `measurements/` against HEAD | `test_measurements` | +| Measurement records | synthetic record files, the committed `measurements/` against HEAD, the **Proof** line naming the BIOS a core ran, and that no page shows a Tested block, a region outcome or a time per frame | `test_measurements` | | Real-condition campaign | synthetic profiles, records and content folders, a fake host for load, disk, clock and commands, and whole passes against a libretro core built from C when a compiler and strace exist | `test_campaign` | -| Region checks | the variants a region game runs under, the outcome read from synthetic records, the system page sentence and the coverage table, and that every system whose BIOS a profile ties to Japan, the US and Europe has a game per region in the content list or a reason | `test_region_check` | +| Region checks | the variants a region game runs under, the outcome read from synthetic records, the outcome sentence that never calls a built-in BIOS boot another region's, and that every system whose BIOS a profile ties to Japan, the US and Europe has a game per region in the content list or a reason | `test_region_check` | | Standalone emulators | the release asset EmuDeck's rule picks, the AppImage offset, the Xvfb framebuffer reader, the settings written and the oracle the verdicts use, with no emulator run | `test_standalone_probe` | -| Pack probe | the cores a platform runs for a system, staging grown to a fixed point, runs mapped to pack and manifest entries, the line and its schema, the probe lock per run, inert run output on the site. One class builds a real pack narrowed to PlayStation and lays it out both ways, from the archive and with `install.py --dest`, and skips without the collection | `test_pack_probe` | +| Pack probe | the cores a platform runs for a system, staging grown to a fixed point, runs mapped to pack and manifest entries, the line and its schema, the probe lock per run, that no page shows its lines. One class builds a real pack narrowed to PlayStation and lays it out both ways, from the archive and with `install.py --dest`, and skips without the collection | `test_pack_probe` | | Automation runner, disk cleanup, asset sync and bit-flip repair | fake jobs, git repositories with a bare origin and session worktrees, a committed file with one bit flipped, canned CI runs in place of `gh` | `test_autorun` | | Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` | | Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_availability`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_emulator_page`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_picker`, `test_site_platform_hashes`, `test_site_r2_components`, `test_site_r2_css`, `test_site_r2_emulators`, `test_site_r2_entry`, `test_site_r2_gate`, `test_site_r2_nav`, `test_site_r2_p0`, `test_site_r2_reference`, `test_site_r2_systems`, `test_site_r2_wiki`, `test_site_r3_collection`, `test_site_r3_components`, `test_site_r3_css`, `test_site_r3_emulators`, `test_site_r3_entry`, `test_site_r3_nav`, `test_site_r3_p0`, `test_site_r3_reference`, `test_site_r3_systems`, `test_site_r3_wiki`, `test_site_r4_components`, `test_site_r4_css`, `test_site_r4_emulators`, `test_site_r4_entry`, `test_site_r4_merge`, `test_site_r4_p0`, `test_site_r4_reference`, `test_site_r4_search`, `test_site_r4_systems`, `test_site_r4_wiki`, `test_site_r5_components`, `test_site_r5_css`, `test_site_r5_emulators`, `test_site_r5_entry`, `test_site_r5_reference`, `test_site_r5_search`, `test_site_r5_systems`, `test_site_r5_wiki`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_version_basis`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` | diff --git a/wiki/tools.md b/wiki/tools.md index feaab591a..41d76846e 100644 --- a/wiki/tools.md +++ b/wiki/tools.md @@ -846,7 +846,10 @@ python scripts/sync_release_assets.py --apply Runs a libretro core with no window, no sound and no input device, and reports what it did. It is the test in real conditions behind a claim about -a core: which files it opens, what it shows, how fast it runs. +a core: which files it opens, what it shows, which BIOS it ran. + +These probes run on demand only, for three cases: to settle a dispute or an issue, to back a correction sent to a platform, and to check the packs before a release. They are never run as a campaign over every emulator, and the site keeps no showcase of their runs: an emulator page shows a run only when a profile's `evidence:` names it or it answers an issue (`issue-` subjects), as one **Proof** line naming the BIOS the core ran. Every record stays in `measurements/` and on the Measurements page. Benchmarks (`bench`) stay in the records and the SQLite files and +only choose between variants: no page shows a time per frame. ```bash # one run, printed as JSON @@ -1025,9 +1028,11 @@ sorted and nothing reads the clock, so two builds give the same bytes. ### campaign.py -Runs the real-condition campaign with nobody watching: for each emulator -profile in scope it gets the core and the test content, runs `core_probe.py`, -applies what the runs prove and queues the rest. It is a thin layer over +Runs a batch of `core_probe.py` checks for the profiles a question names +(`--only`, `--system`): it gets the core and the test content, runs the +checks, applies what the runs prove and queues the rest. A maintainer runs it +on demand, for one of the three cases of `core_probe.py`, never over every +profile as a standing campaign. It is a thin layer over `core_probe.py`, the libretro buildbot and romfetch. RetroArch's `--max-frames` and libretro.py were the other candidates: neither traces the files a core opens nor writes a record. @@ -1161,10 +1166,12 @@ claim: `confirmed` when its `region:` holds the game's region, `contradicted` when it does not, `ambiguous` when the core opened its own region's file and the others too (PCSX-ReARMed opens all three), `unread` when it opened none of them. Each variant's result is the oracle verdict of its record. While the oracle has no reference frame for a game, a variant counts as booting when it samples the same frames -as the `full` run, and the outcome says it was judged that way. Other frames give no verdict: the US disc reaches its game on the Japanese and European BIOS through another licence screen. The system -pages show each outcome as a sentence ("the Japanese BIOS boots the Japanese -game, the other regions' BIOS do not") with links to the four records. A -maintainer starts each run. Each core_probe run takes the host probe lock for itself and frees it before the next, so do not wrap the command in `flock`. Nothing runs by itself. +as the `full` run, and the outcome says it was judged that way. Other frames give no verdict: the US disc reaches its game on the Japanese and European BIOS through another licence screen. A boot +counts as another region's BIOS only when `bios_used` names that file: Beetle +PSX and SwanStation boot their built-in OpenBIOS with only the other regions' +files, which they refused. `report` prints the outcomes as JSON, and no page +shows them: a profile's `region:` cites a record in its `evidence:` when one +settles it. A maintainer starts each run, on demand. Each core_probe run takes the host probe lock for itself and frees it before the next, so do not wrap the command in `flock`. Nothing runs by itself. ### standalone_probe.py @@ -1173,7 +1180,9 @@ Runs a platform's standalone emulator on a game and records it like record schema (`frontend.script` names this script). No existing harness traces the files a standalone emulator opens or writes a record, so this is a thin layer over the emulator's own batch mode, Xvfb and core_probe's -sandbox, oracle and record functions, imported and not copied. +sandbox, oracle and record functions, imported and not copied. Like +`core_probe.py`, it runs on demand only: for an issue or a dispute, to back a +correction sent to a platform, or to check the packs before a release. ```bash python scripts/standalone_probe.py resolve duckstation # the asset EmuDeck's installer picks @@ -1218,8 +1227,8 @@ by itself. ### pack_probe.py Tests what the project promises: a platform's pack, installed the way a user -installs it, starts that platform's games. A maintainer starts it, nothing -runs it on a schedule. It takes the host's probe lock +installs it, starts that platform's games. A maintainer starts it before a +release, or to answer an issue about a pack, nothing runs it on a schedule. It takes the host's probe lock (`~/.local/state/retrobios/probe.lock`, `RETROBIOS_PROBE_LOCK`) around each `core_probe` run and releases it before the next, so other probes run between its runs, and tells `core_probe` it holds it (`RETROBIOS_PROBE_LOCK_HELD=1`). @@ -1265,9 +1274,9 @@ is no finding: BlastEm tries `tmss.md`, then reads `bios_MD.bin`. The run cannot see two things the platform does: core options a frontend sets (Batocera's configgen writes some), and the files the installer puts in -the save directory, counted in `source.save_dir_files`. The Measurements page -lists the latest probe of each platform, system, install path, core and -game, and each platform page shows its own in a **Tested** block. +the save directory, counted in `source.save_dir_files`. A maintainer runs it +before a release (wiki/release-process.md). Its lines stay in +`measurements/pack-probes.jsonl`, and no page shows a table of them. ### Other tools