diff --git a/wiki/testing-guide.md b/wiki/testing-guide.md index 71ec43c26..7e8989966 100644 --- a/wiki/testing-guide.md +++ b/wiki/testing-guide.md @@ -57,6 +57,7 @@ Every module in `tests/`, by the area it guards. | Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_app`, `test_install_from`, `test_install_offline`, `test_install_selection`, `test_install_tui` | | Handheld launchers | a pty as the console, fake installer and firmware tools, stubbed PortMaster dialogs | `test_handheld_launchers`, `test_port` | | Running cores | parsers on fixed inputs, a libretro core built from C in the test when a compiler exists | `test_core_probe` | +| Measurement records | a synthetic measurements repository built with git | `test_measurements` | | Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` | | Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_availability`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_emulator_page`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_picker`, `test_site_platform_hashes`, `test_site_r2_components`, `test_site_r2_css`, `test_site_r2_emulators`, `test_site_r2_entry`, `test_site_r2_gate`, `test_site_r2_nav`, `test_site_r2_p0`, `test_site_r2_reference`, `test_site_r2_systems`, `test_site_r2_wiki`, `test_site_r3_collection`, `test_site_r3_components`, `test_site_r3_css`, `test_site_r3_emulators`, `test_site_r3_entry`, `test_site_r3_nav`, `test_site_r3_p0`, `test_site_r3_reference`, `test_site_r3_systems`, `test_site_r3_wiki`, `test_site_r4_components`, `test_site_r4_css`, `test_site_r4_emulators`, `test_site_r4_entry`, `test_site_r4_merge`, `test_site_r4_p0`, `test_site_r4_reference`, `test_site_r4_search`, `test_site_r4_systems`, `test_site_r4_wiki`, `test_site_r5_components`, `test_site_r5_css`, `test_site_r5_emulators`, `test_site_r5_entry`, `test_site_r5_reference`, `test_site_r5_search`, `test_site_r5_systems`, `test_site_r5_wiki`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` | diff --git a/wiki/tools.md b/wiki/tools.md index 492e0c734..09fe577c3 100644 --- a/wiki/tools.md +++ b/wiki/tools.md @@ -641,8 +641,9 @@ profile, `--record-coreinfo ` records its blob as reviewed. Data directories are checked against their upstream, dump catalogs and romset snapshots against the catalog they were read from (Redump DAT versions, the No-Intro daily rebuild, the newest TOSEC pack, the newest MAME release, the -git blobs of the FBNeo DATs), and the CI toolchain against PyPI and the -actions' latest releases. +git blobs of the FBNeo DATs), the measurements pin against the head of the +measurements repository, and the CI toolchain against PyPI and the actions' +latest releases. The `native` area needs no network. It reads the cache `export_native.py` patches from and reports an original that is absent, that came from another @@ -684,6 +685,7 @@ committed. | `targets/` | the target scraper | | `data/` | `refresh_data_dirs.py --key --force` | | `catalogs/mame recipes`, `catalogs/fbneo recipes` | `scraper/romset_dat_importer.py --source --fetch` | +| `measurements/records pin` | `sync_measurements.py --fetch --bump` | Rows that need a decision are listed and never run: a buildbot core with no profile, a new `.info` file, the Redump, No-Intro and TOSEC packs, the CI @@ -718,12 +720,12 @@ python scripts/core_probe.py run --core mednafen_saturn_libretro.so \ # timed repetitions after warm-up, written as a record python scripts/core_probe.py bench --core core.so --content game.cue \ - --warmup 300 --frames 3000 --repeat 5 --record measurements/ + --warmup 300 --frames 3000 --repeat 5 --record .cache/measurements # one system file swapped between runs, compared on one observable python scripts/core_probe.py check --core core.so --content disc.cue \ --vary mpeg_card.bin=a.bin --vary mpeg_card.bin=b.bin --vary mpeg_card.bin= \ - --observable video_sha1 --claim "..." --method "..." --record measurements/ + --observable video_sha1 --claim "..." --method "..." --record .cache/measurements ``` Each repetition runs in a fresh process under `strace -f -e trace=openat,open` @@ -745,7 +747,10 @@ strace the record says that tracing was unavailable. | `--state-var SECTION.VAR` | a variable of a Mednafen save state, as an observable | | `--state-dir` | keep the save state taken at the end of the first repetition | | `--core-repo`, `--core-commit`, `--build-command` | where the core build comes from, copied into the record | -| `--record DIR` | write `DIR//-.json` (schema `core-probe-record.schema.json`) | +| `--record DIR` | append the record to the measurements clone `DIR`: one line of `records/.jsonl` and its sidecar `data//.json.gz`. A directory inside this repository's tracked tree is refused | +| `--emulator` | the profile the record is filed under. By default the profile whose `cores:` names the `.so`, else the core's library name | +| `--subject` | a slug naming the test, shared by its reruns. By default derived from the claim and method, or from the inputs of a run or bench | +| `--supersedes ID` | the id of an earlier record of the same emulator this one corrects | | `--max-load` | refuse to record when the 1-minute load average is above it (default: a quarter of the CPU count) | | `--no-trace` | run without strace | | `@FILE` | read arguments from FILE, one per line | @@ -763,6 +768,50 @@ host (CPU, kernel, Python, load before and after) and each input by hash. Core log lines arrive through the libretro log interface. Floating-point arguments of a log line cannot be read from Python and print as ``. +A record is stored in two parts by `measurement_records.py`. The line keeps +the claim, method, result, timing statistics, host, core build, frontend +commit, inputs by hash and, per variant, the first repetition's observables +and files opened, under 20 KB. The sidecar keeps every repetition in full +(per-frame hashes, core logs, declared options), the raw timing values and +any compared value too large for a line, and its sha256 is in the line. The +id is the date and the first 12 hex digits of the sha256 of the line without +its id. Schemas: `core-probe-record.schema.json` for a line, +`core-probe-data.schema.json` for a sidecar. `join` rebuilds the full record +the run printed, `check_pair` checks a line against its sidecar. + +### sync_measurements.py + +The records live in their own repository, +[Abdess/retrobios-measurements](https://github.com/Abdess/retrobios-measurements): +`records/.jsonl`, one record per line, append-only, and +`data//.json.gz`. This repository keeps only the pin, +`measurements-pin.json`, and reads the clone in `.cache/measurements/` at +that commit with git, whatever its working tree holds. + +```bash +python scripts/sync_measurements.py --fetch # clone or update, check the remote +python scripts/sync_measurements.py --fetch --bump # then move the pin to its head +python scripts/sync_measurements.py --check-only # check every line and sidecar at the pin +python scripts/sync_measurements.py # write the site's measurement files +``` + +`--fetch` is the only step on the network. It refuses a remote whose history +changed or removed a line the pin holds, or rewrote a data file: a correction +is a new line naming the record it `supersedes`. Every line must hash to its +id and join its sidecar. Two branches that each add a line merge cleanly, +the measurements repository marking `records/*.jsonl` `merge=union`. + +`generate_site.py` compiles the records at the pin into files it never +commits: `public/data/measurements/index.json` (one item per emulator and +subject, its latest record not superseded, with method, host and the +permalink of the line at the pin), `public/downloads/measurements.sqlite` +(normalised tables, indexed on emulator, system and claim, `meta` naming the +commit) and one `public/downloads/measurements/.sqlite` per +emulator. Rows are sorted and nothing reads the clock, so two builds give +the same bytes. Without the pinned commit the site lists no measurements and +says so. `--require-measurements` makes that an error, and the deploy passes +it after a shallow fetch of the pinned commit. + ### Other tools | Script | Purpose |