diff --git a/wiki/measurements.md b/wiki/measurements.md index 88d5daf85..3b42676e1 100644 --- a/wiki/measurements.md +++ b/wiki/measurements.md @@ -38,9 +38,9 @@ shown without them. ## Where results are kept Each result is a full record in the -[retrobios-measurements](https://github.com/Abdess/retrobios-measurements) -project. A record is never changed once added. A new run is added beside it, -and a correction names the record it replaces. +[`measurements/`](https://github.com/Abdess/retrobios/tree/main/measurements) +folder of this project. A record is never changed once added. A new run is +added beside it, and a correction names the record it replaces. The results so far ran on these computers: diff --git a/wiki/testing-guide.md b/wiki/testing-guide.md index 7e8989966..9615008aa 100644 --- a/wiki/testing-guide.md +++ b/wiki/testing-guide.md @@ -57,7 +57,7 @@ Every module in `tests/`, by the area it guards. | Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_app`, `test_install_from`, `test_install_offline`, `test_install_selection`, `test_install_tui` | | Handheld launchers | a pty as the console, fake installer and firmware tools, stubbed PortMaster dialogs | `test_handheld_launchers`, `test_port` | | Running cores | parsers on fixed inputs, a libretro core built from C in the test when a compiler exists | `test_core_probe` | -| Measurement records | a synthetic measurements repository built with git | `test_measurements` | +| Measurement records | synthetic record trees, the committed `measurements/` against HEAD | `test_measurements` | | Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` | | Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_availability`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_emulator_page`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_picker`, `test_site_platform_hashes`, `test_site_r2_components`, `test_site_r2_css`, `test_site_r2_emulators`, `test_site_r2_entry`, `test_site_r2_gate`, `test_site_r2_nav`, `test_site_r2_p0`, `test_site_r2_reference`, `test_site_r2_systems`, `test_site_r2_wiki`, `test_site_r3_collection`, `test_site_r3_components`, `test_site_r3_css`, `test_site_r3_emulators`, `test_site_r3_entry`, `test_site_r3_nav`, `test_site_r3_p0`, `test_site_r3_reference`, `test_site_r3_systems`, `test_site_r3_wiki`, `test_site_r4_components`, `test_site_r4_css`, `test_site_r4_emulators`, `test_site_r4_entry`, `test_site_r4_merge`, `test_site_r4_p0`, `test_site_r4_reference`, `test_site_r4_search`, `test_site_r4_systems`, `test_site_r4_wiki`, `test_site_r5_components`, `test_site_r5_css`, `test_site_r5_emulators`, `test_site_r5_entry`, `test_site_r5_reference`, `test_site_r5_search`, `test_site_r5_systems`, `test_site_r5_wiki`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` | diff --git a/wiki/tools.md b/wiki/tools.md index 09fe577c3..f51071e9f 100644 --- a/wiki/tools.md +++ b/wiki/tools.md @@ -641,9 +641,8 @@ profile, `--record-coreinfo ` records its blob as reviewed. Data directories are checked against their upstream, dump catalogs and romset snapshots against the catalog they were read from (Redump DAT versions, the No-Intro daily rebuild, the newest TOSEC pack, the newest MAME release, the -git blobs of the FBNeo DATs), the measurements pin against the head of the -measurements repository, and the CI toolchain against PyPI and the actions' -latest releases. +git blobs of the FBNeo DATs), and the CI toolchain against PyPI and the +actions' latest releases. The `native` area needs no network. It reads the cache `export_native.py` patches from and reports an original that is absent, that came from another @@ -685,7 +684,6 @@ committed. | `targets/` | the target scraper | | `data/` | `refresh_data_dirs.py --key --force` | | `catalogs/mame recipes`, `catalogs/fbneo recipes` | `scraper/romset_dat_importer.py --source --fetch` | -| `measurements/records pin` | `sync_measurements.py --fetch --bump` | Rows that need a decision are listed and never run: a buildbot core with no profile, a new `.info` file, the Redump, No-Intro and TOSEC packs, the CI @@ -720,12 +718,12 @@ python scripts/core_probe.py run --core mednafen_saturn_libretro.so \ # timed repetitions after warm-up, written as a record python scripts/core_probe.py bench --core core.so --content game.cue \ - --warmup 300 --frames 3000 --repeat 5 --record .cache/measurements + --warmup 300 --frames 3000 --repeat 5 --record # one system file swapped between runs, compared on one observable python scripts/core_probe.py check --core core.so --content disc.cue \ --vary mpeg_card.bin=a.bin --vary mpeg_card.bin=b.bin --vary mpeg_card.bin= \ - --observable video_sha1 --claim "..." --method "..." --record .cache/measurements + --observable video_sha1 --claim "..." --method "..." --record ``` Each repetition runs in a fresh process under `strace -f -e trace=openat,open` @@ -747,7 +745,7 @@ strace the record says that tracing was unavailable. | `--state-var SECTION.VAR` | a variable of a Mednafen save state, as an observable | | `--state-dir` | keep the save state taken at the end of the first repetition | | `--core-repo`, `--core-commit`, `--build-command` | where the core build comes from, copied into the record | -| `--record DIR` | append the record to the measurements clone `DIR`: one line of `records/.jsonl` and its sidecar `data//.json.gz`. A directory inside this repository's tracked tree is refused | +| `--record [DIR]` | append the record to `measurements/` of the checkout `DIR`, by default this one: one line of `measurements/.jsonl` and its sidecar `measurements/data//.json.gz`. Commit both | | `--emulator` | the profile the record is filed under. By default the profile whose `cores:` names the `.so`, else the core's library name | | `--subject` | a slug naming the test, shared by its reruns. By default derived from the claim and method, or from the inputs of a run or bench | | `--supersedes ID` | the id of an earlier record of the same emulator this one corrects | @@ -779,38 +777,27 @@ its id. Schemas: `core-probe-record.schema.json` for a line, `core-probe-data.schema.json` for a sidecar. `join` rebuilds the full record the run printed, `check_pair` checks a line against its sidecar. -### sync_measurements.py +### Measurement records -The records live in their own repository, -[Abdess/retrobios-measurements](https://github.com/Abdess/retrobios-measurements): -`records/.jsonl`, one record per line, append-only, and -`data//.json.gz`. This repository keeps only the pin, -`measurements-pin.json`, and reads the clone in `.cache/measurements/` at -that commit with git, whatever its working tree holds. +The records live in `measurements/` of this repository: +`measurements/.jsonl`, one record per line in the order added, and +`measurements/data//.json.gz`, the sidecar of each. A line is +never rewritten or removed: a new run is a new line at the end of its file, +and a correction is a new line naming the record it `supersedes`. Every line +must hash to its id and join its sidecar, which `validate_schemas.py` checks +with both schemas. `tests/test_measurements.py` fails when a line or data +file that HEAD holds has changed or gone. Two branches that each add a line +merge cleanly: `.gitattributes` marks `measurements/*.jsonl` `merge=union`. -```bash -python scripts/sync_measurements.py --fetch # clone or update, check the remote -python scripts/sync_measurements.py --fetch --bump # then move the pin to its head -python scripts/sync_measurements.py --check-only # check every line and sidecar at the pin -python scripts/sync_measurements.py # write the site's measurement files -``` - -`--fetch` is the only step on the network. It refuses a remote whose history -changed or removed a line the pin holds, or rewrote a data file: a correction -is a new line naming the record it `supersedes`. Every line must hash to its -id and join its sidecar. Two branches that each add a line merge cleanly, -the measurements repository marking `records/*.jsonl` `merge=union`. - -`generate_site.py` compiles the records at the pin into files it never -commits: `public/data/measurements/index.json` (one item per emulator and -subject, its latest record not superseded, with method, host and the -permalink of the line at the pin), `public/downloads/measurements.sqlite` +`generate_site.py` compiles the records into files it never commits +(`sitemeasurements.py`): `public/data/measurements/index.json` (one item per +emulator and subject, its latest record not superseded, with method, host and +the permalink of its line), `public/downloads/measurements.sqlite` (normalised tables, indexed on emulator, system and claim, `meta` naming the commit) and one `public/downloads/measurements/.sqlite` per -emulator. Rows are sorted and nothing reads the clock, so two builds give -the same bytes. Without the pinned commit the site lists no measurements and -says so. `--require-measurements` makes that an error, and the deploy passes -it after a shallow fetch of the pinned commit. +emulator. A permalink names the last commit that touched `measurements/`, and +the build warns when the records on disk differ from that commit. Rows are +sorted and nothing reads the clock, so two builds give the same bytes. ### Other tools