docs: say probes run on demand only

wiki/measurements.md

@@ -2,13 +2,15 @@
 level: reference
 order: 70
 group: Data
-terms: measurement, benchmark, tested, core probe, real conditions, method
+terms: measurement, proof, tested, core probe, real conditions, method
 ---
 
 # How emulators are tested
 
-Some pages say what an emulator does with a file. Where we ran it to check,
-the emulator's page shows a **Tested** block. Every result is on the
+We run an emulator only when a question needs a real answer. That is an
+issue, a fix we send to a project, or a check before a release.
+
+A page that relies on a run says so in a **Proof** line. Every run is on the
 [Measurements](../measurements.md) page.
 
 ## What a test does
@@ -16,11 +18,7 @@ the emulator's page shows a **Tested** block. Every result is on the
 - A script runs the emulator with no window and no sound.
 - It starts a game and runs it for a set time, at least twice.
 - It compares the screens with ones we checked by eye.
-
-## Your BIOS folder { #packs }
-
-We also test the files as you install them, for RetroArch, Batocera and
-RetroBat, then start a game. The setup's page shows the result under **Tested**.
+- It reads the emulator's log to see which BIOS it ran.
 
 ## What a result says
 
@@ -33,7 +31,13 @@ RetroBat, then start a game. The setup's page shows the result under **Tested**.
 | hang | the screen stayed blank, or the run never ended |
 | unknown | none of these could be shown |
 | no verdict | an older test that only compared screens |
-| ms per frame | a speed test |
+
+## What a result does not prove
+
+- It shows the first minutes of one game, not the whole game.
+- It sees the files the emulator opened on this path, not every path.
+- Some emulators run their own BIOS when yours is refused. The game still works.
+- The app you play in may change settings the test left as they were.
 
 ## How a file is proven
 
@@ -44,27 +48,12 @@ file under the BIOS name: zeros, half the file, or another region's BIOS.
 - **Built-in stand-in**: no BIOS still works.
 - **Checked**: a wrong file is refused.
 
-## When tests run
-
-A maintainer starts a test run. A clear result updates the emulator's page.
-Anything else waits on the [Measurements](../measurements.md) page for someone
-to read the code.
-
-## What a result keeps
-
-Each result names its method, the computer it ran on and the exact build of
-the emulator. Its inputs are kept by fingerprint. A result holds for that
-build and those inputs only.
-
-A speed figure depends on the computer and how busy it was. It is never
-shown without them.
-
 ## Where results are kept
 
 Each result is a full record in the
 [`measurements/`](https://github.com/Abdess/retrobios/tree/main/measurements)
-folder of this project. A record is never changed once added. A new run is
-added beside it, and a correction names the record it replaces.
+folder of this project. It names its method, the computer and the exact build
+of the emulator. A record is never changed once added.
 
 The results so far ran on these computers:
 

wiki/release-process.md

@@ -287,6 +287,9 @@ gh release list --json tagName,createdAt \
   | tail -n +2 | while read tag; do gh release delete "$tag" --yes --cleanup-tag; done
 ```
 
+Before a release, since 2026-10-28, a maintainer runs `python scripts/pack_probe.py run
+--platform <name>` on the platforms that get a pack, and reads its findings before cutting.
+
 ## Which platforms get a pack { #which-platforms-get-a-pack }
 
 `release_pack: true` in `platforms/_registry.yml` is the list. `generate_pack.py

wiki/testing-guide.md

@@ -57,11 +57,11 @@ Every module in `tests/`, by the area it guards.
 | Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_app`, `test_install_from`, `test_install_offline`, `test_install_selection`, `test_install_tui` |
 | Handheld launchers | a pty as the console, fake installer and firmware tools, stubbed PortMaster dialogs | `test_handheld_launchers`, `test_port` |
 | Running cores | parsers on fixed inputs, a libretro core built from C in the test when a compiler exists | `test_core_probe` |
-| Measurement records | synthetic record files, the committed `measurements/` against HEAD | `test_measurements` |
+| Measurement records | synthetic record files, the committed `measurements/` against HEAD, the **Proof** line naming the BIOS a core ran, and that no page shows a Tested block, a region outcome or a time per frame | `test_measurements` |
 | Real-condition campaign | synthetic profiles, records and content folders, a fake host for load, disk, clock and commands, and whole passes against a libretro core built from C when a compiler and strace exist | `test_campaign` |
-| Region checks | the variants a region game runs under, the outcome read from synthetic records, the system page sentence and the coverage table, and that every system whose BIOS a profile ties to Japan, the US and Europe has a game per region in the content list or a reason | `test_region_check` |
+| Region checks | the variants a region game runs under, the outcome read from synthetic records, the outcome sentence that never calls a built-in BIOS boot another region's, and that every system whose BIOS a profile ties to Japan, the US and Europe has a game per region in the content list or a reason | `test_region_check` |
 | Standalone emulators | the release asset EmuDeck's rule picks, the AppImage offset, the Xvfb framebuffer reader, the settings written and the oracle the verdicts use, with no emulator run | `test_standalone_probe` |
-| Pack probe | the cores a platform runs for a system, staging grown to a fixed point, runs mapped to pack and manifest entries, the line and its schema, the probe lock per run, inert run output on the site. One class builds a real pack narrowed to PlayStation and lays it out both ways, from the archive and with `install.py --dest`, and skips without the collection | `test_pack_probe` |
+| Pack probe | the cores a platform runs for a system, staging grown to a fixed point, runs mapped to pack and manifest entries, the line and its schema, the probe lock per run, that no page shows its lines. One class builds a real pack narrowed to PlayStation and lays it out both ways, from the archive and with `install.py --dest`, and skips without the collection | `test_pack_probe` |
 | Automation runner, disk cleanup, asset sync and bit-flip repair | fake jobs, git repositories with a bare origin and session worktrees, a committed file with one bit flipped, canned CI runs in place of `gh` | `test_autorun` |
 | Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` |
 | Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_availability`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_emulator_page`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_picker`, `test_site_platform_hashes`, `test_site_r2_components`, `test_site_r2_css`, `test_site_r2_emulators`, `test_site_r2_entry`, `test_site_r2_gate`, `test_site_r2_nav`, `test_site_r2_p0`, `test_site_r2_reference`, `test_site_r2_systems`, `test_site_r2_wiki`, `test_site_r3_collection`, `test_site_r3_components`, `test_site_r3_css`, `test_site_r3_emulators`, `test_site_r3_entry`, `test_site_r3_nav`, `test_site_r3_p0`, `test_site_r3_reference`, `test_site_r3_systems`, `test_site_r3_wiki`, `test_site_r4_components`, `test_site_r4_css`, `test_site_r4_emulators`, `test_site_r4_entry`, `test_site_r4_merge`, `test_site_r4_p0`, `test_site_r4_reference`, `test_site_r4_search`, `test_site_r4_systems`, `test_site_r4_wiki`, `test_site_r5_components`, `test_site_r5_css`, `test_site_r5_emulators`, `test_site_r5_entry`, `test_site_r5_reference`, `test_site_r5_search`, `test_site_r5_systems`, `test_site_r5_wiki`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_version_basis`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` |

wiki/tools.md

@@ -846,7 +846,10 @@ python scripts/sync_release_assets.py --apply
 
 Runs a libretro core with no window, no sound and no input device, and
 reports what it did. It is the test in real conditions behind a claim about
-a core: which files it opens, what it shows, how fast it runs.
+a core: which files it opens, what it shows, which BIOS it ran.
+
+These probes run on demand only, for three cases: to settle a dispute or an issue, to back a correction sent to a platform, and to check the packs before a release. They are never run as a campaign over every emulator, and the site keeps no showcase of their runs: an emulator page shows a run only when a profile's `evidence:` names it or it answers an issue (`issue<N>-` subjects), as one **Proof** line naming the BIOS the core ran. Every record stays in `measurements/` and on the Measurements page. Benchmarks (`bench`) stay in the records and the SQLite files and
+only choose between variants: no page shows a time per frame.
 
 ```bash
 # one run, printed as JSON
@@ -1025,9 +1028,11 @@ sorted and nothing reads the clock, so two builds give the same bytes.
 
 ### campaign.py
 
-Runs the real-condition campaign with nobody watching: for each emulator
-profile in scope it gets the core and the test content, runs `core_probe.py`,
-applies what the runs prove and queues the rest. It is a thin layer over
+Runs a batch of `core_probe.py` checks for the profiles a question names
+(`--only`, `--system`): it gets the core and the test content, runs the
+checks, applies what the runs prove and queues the rest. A maintainer runs it
+on demand, for one of the three cases of `core_probe.py`, never over every
+profile as a standing campaign. It is a thin layer over
 `core_probe.py`, the libretro buildbot and romfetch. RetroArch's
 `--max-frames` and libretro.py were the other candidates: neither traces the
 files a core opens nor writes a record.
@@ -1161,10 +1166,12 @@ claim: `confirmed` when its `region:` holds the game's region, `contradicted`
 when it does not, `ambiguous` when the core opened its own region's file and the others too (PCSX-ReARMed opens all three), `unread` when it opened none of them. Each variant's
 result is the oracle verdict of its record. While the oracle has no reference
 frame for a game, a variant counts as booting when it samples the same frames
-as the `full` run, and the outcome says it was judged that way. Other frames give no verdict: the US disc reaches its game on the Japanese and European BIOS through another licence screen. The system
-pages show each outcome as a sentence ("the Japanese BIOS boots the Japanese
-game, the other regions' BIOS do not") with links to the four records. A
-maintainer starts each run. Each core_probe run takes the host probe lock for itself and frees it before the next, so do not wrap the command in `flock`. Nothing runs by itself.
+as the `full` run, and the outcome says it was judged that way. Other frames give no verdict: the US disc reaches its game on the Japanese and European BIOS through another licence screen. A boot
+counts as another region's BIOS only when `bios_used` names that file: Beetle
+PSX and SwanStation boot their built-in OpenBIOS with only the other regions'
+files, which they refused. `report` prints the outcomes as JSON, and no page
+shows them: a profile's `region:` cites a record in its `evidence:` when one
+settles it. A maintainer starts each run, on demand. Each core_probe run takes the host probe lock for itself and frees it before the next, so do not wrap the command in `flock`. Nothing runs by itself.
 
 ### standalone_probe.py
 
@@ -1173,7 +1180,9 @@ Runs a platform's standalone emulator on a game and records it like
 record schema (`frontend.script` names this script). No existing harness
 traces the files a standalone emulator opens or writes a record, so this is
 a thin layer over the emulator's own batch mode, Xvfb and core_probe's
-sandbox, oracle and record functions, imported and not copied.
+sandbox, oracle and record functions, imported and not copied. Like
+`core_probe.py`, it runs on demand only: for an issue or a dispute, to back a
+correction sent to a platform, or to check the packs before a release.
 
 ```bash
 python scripts/standalone_probe.py resolve duckstation   # the asset EmuDeck's installer picks
@@ -1218,8 +1227,8 @@ by itself.
 ### pack_probe.py
 
 Tests what the project promises: a platform's pack, installed the way a user
-installs it, starts that platform's games. A maintainer starts it, nothing
-runs it on a schedule. It takes the host's probe lock
+installs it, starts that platform's games. A maintainer starts it before a
+release, or to answer an issue about a pack, nothing runs it on a schedule. It takes the host's probe lock
 (`~/.local/state/retrobios/probe.lock`, `RETROBIOS_PROBE_LOCK`) around each
 `core_probe` run and releases it before the next, so other probes run between
 its runs, and tells `core_probe` it holds it (`RETROBIOS_PROBE_LOCK_HELD=1`).
@@ -1265,9 +1274,9 @@ is no finding: BlastEm tries `tmss.md`, then reads `bios_MD.bin`.
 
 The run cannot see two things the platform does: core options a frontend
 sets (Batocera's configgen writes some), and the files the installer puts in
-the save directory, counted in `source.save_dir_files`. The Measurements page
-lists the latest probe of each platform, system, install path, core and
-game, and each platform page shows its own in a **Tested** block.
+the save directory, counted in `source.save_dir_files`. A maintainer runs it
+before a release (wiki/release-process.md). Its lines stay in
+`measurements/pack-probes.jsonl`, and no page shows a table of them.
 
 ### Other tools