kmoody/libretro
1
0 Fork 0
mirror of https://github.com/Abdess/retroarch_system.git synced 2026-11-01 00:27:20 -05:00
Code Issues Packages Projects Releases Wiki Activity

chore: merge the automatic test campaign

Browse Source
This commit is contained in:
Abdessamad Derraz committed 2026-10-27 14:05:18 +01:00
2 parents 9c63327da3 332c217155
commit 9c8dbcd99a
31 files changed
+4132 -49
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

wiki/measurements.md
+8
View File
@@ -19,6 +19,14 @@ the emulator's page shows a **Tested** block. Every result is on the
- It compares what the emulator showed, played or saved, by fingerprint (a hash).
- Each version runs at least twice, each time from a fresh start.
## When tests run
Tests run by themselves each night on an idle computer. A clear result updates
the emulator's page and names the test as proof.
Anything else waits on the [Measurements](../measurements.md) page for someone
to read the code.
## What a result says
- **same**: the file changed nothing that was compared.
wiki/profiling.md
+1
View File
@@ -601,6 +601,7 @@ does not look.
| `aliases` | alternate filenames for the same file |
| `mode` | `libretro`, `standalone`, or `both` |
| `hle_fallback` | true if a high-level emulation path exists |
| `evidence` | by field (`required`, `hle_fallback`, `path`, `aliases`), the ids of the records in `measurements/<emulator>.jsonl` whose runs settled it. Written by `campaign.py` from its rule table, never by hand |
| `category` | `bios` (default), `game_data`, `bios_zip` |
| `region` | always a list of territory slugs from the schema enum (`[north-america]`, `[japan, asia-ntsc]`). The region the code selects the file **for**, never the region of the dump. See step 3b |
| `region_check` | true if the core refuses to boot a game whose region does not match the BIOS |
wiki/testing-guide.md
+1
View File
@@ -58,6 +58,7 @@ Every module in `tests/`, by the area it guards.
| Handheld launchers | a pty as the console, fake installer and firmware tools, stubbed PortMaster dialogs | `test_handheld_launchers`, `test_port` |
| Running cores | parsers on fixed inputs, a libretro core built from C in the test when a compiler exists | `test_core_probe` |
| Measurement records | synthetic record files, the committed `measurements/` against HEAD | `test_measurements` |
| Real-condition campaign | synthetic profiles, records and content folders, a fake host for load, disk, clock and commands, and whole passes against a libretro core built from C when a compiler and strace exist | `test_campaign` |
| Automation runner, disk cleanup, asset sync and bit-flip repair | fake jobs, git repositories with a bare origin and session worktrees, a committed file with one bit flipped, canned CI runs in place of `gh` | `test_autorun` |
| Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` |
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_availability`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_emulator_page`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_picker`, `test_site_platform_hashes`, `test_site_r2_components`, `test_site_r2_css`, `test_site_r2_emulators`, `test_site_r2_entry`, `test_site_r2_gate`, `test_site_r2_nav`, `test_site_r2_p0`, `test_site_r2_reference`, `test_site_r2_systems`, `test_site_r2_wiki`, `test_site_r3_collection`, `test_site_r3_components`, `test_site_r3_css`, `test_site_r3_emulators`, `test_site_r3_entry`, `test_site_r3_nav`, `test_site_r3_p0`, `test_site_r3_reference`, `test_site_r3_systems`, `test_site_r3_wiki`, `test_site_r4_components`, `test_site_r4_css`, `test_site_r4_emulators`, `test_site_r4_entry`, `test_site_r4_merge`, `test_site_r4_p0`, `test_site_r4_reference`, `test_site_r4_search`, `test_site_r4_systems`, `test_site_r4_wiki`, `test_site_r5_components`, `test_site_r5_css`, `test_site_r5_emulators`, `test_site_r5_entry`, `test_site_r5_reference`, `test_site_r5_search`, `test_site_r5_systems`, `test_site_r5_wiki`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` |
wiki/tools.md
+97 -4
View File
@@ -717,6 +717,7 @@ them is a reviewed commit.
| Load | Waits while the 1-minute load is above a quarter of the CPUs, up to `max_load_wait` seconds, then `deferred`. |
| Disk | Below the job's `min_free_gb` the job is `refused`. Under 100 GB free the `disk` alert is raised. |
| Retry | Exit 139 (or signal 11), `SystemError` or "unknown opcode" is run again once on an idle host. A second failure is a failure. |
| Environment | A step, gate or recomputation inherits `PATH`, `HOME`, `LANG`, `LC_*`, `TZ` and `TMPDIR`, the runner's own markers and the variables its `env:` key names. A name with `TOKEN`, `SECRET` or `PASS` in it, `GH_TOKEN` and the RomStation account are dropped even when named, and a step naming one is refused when the registry loads. `GITHUB_TOKEN` goes to the push, the `gh` calls that read CI, and a step marked `github_token: true`. The registry accepts that key only on `python scripts/check_freshness.py`, `python scripts/refresh_stale.py` and `python -m scripts.scraper.*` tracked in this repository, and never in the `campaign` job. |
| Double computation | Each `twice` entry recomputes an artifact the gate wrote and compares both, timestamps aside. A difference is recomputed once on an idle host, then the job fails with the `host-integrity` alert. |
Jobs that commit run in `.cache/autorun/worktree`, on the branch `autorun`,
@@ -739,13 +740,15 @@ until a green run on `main` follows.
State is `.cache/autorun/state.json`, logs are
`.cache/autorun/logs/<job>/<date>.log`, kept 30 days. A job's `report` key
names a JSON file one of its steps writes, and the runner keeps it in the
job's status. Exit codes: 0 for `ok`,
job's status. A report holding an `alert` raises the `job:<name>` alert
until a report without one. Exit codes: 0 for `ok`,
75 for `deferred`, 1 otherwise.
| Job | Schedule | What it does |
|-----|----------|--------------|
| `nightly` | 02:00 | `check_freshness.py --json` saved to a file, `refresh_stale.py --findings` on its `data` and `catalogs` rows, the MAME and FBNeo hash scrapers, then `pipeline.py --offline` as the gate. `database.json` and `install/*.json` are computed a second time and compared. Commits the hash profiles, the recipes and the regenerated records |
| `assets` | 04:30 | `sync_release_assets.py`, as a dry run |
| `campaign` | 06:00 | `campaign.py run --budget 150m` once the host is idle (up to two hours of waiting), gated by `campaign.py gate`. Commits the records, findings and pass line, then the profile fields the runs proved |
| `bios-integrity` | Sunday 08:30 | `bios_integrity.py --apply --json` in the main checkout, once the host is idle (up to four hours of waiting). Its report is kept in the job's status |
| `cleanup` | 05:30 | `cleanup_disk.py --apply --hours 72` in the main checkout |
| `disk-check` | hourly | `cleanup_disk.py --check`, without the lock |
@@ -845,7 +848,18 @@ python scripts/core_probe.py check --core core.so --content disc.cue \
```
Each repetition runs in a fresh process under `strace -f -e trace=openat,open`
when strace is installed. The files the core opened under the system
when strace is installed. The worker starts with an empty environment and
clears its own before it loads the core, keeping only the `--env` values.
When bubblewrap works on the host, the worker runs in it, built as an
allowlist: `/usr` and the `/lib*`, `/bin`, `/sbin` links, the files of `/etc`
the loader, fonts, TLS, the clock and user lookups read, all read-only. Home,
`/tmp` and `/mnt` are empty, the core, the content folder and the staged
system directory are read-only, and only the repetition's scratch folder is
writable. Every namespace is unshared, the network included, so nothing of
the user (`~/.config`, `~/.ssh`, `~/.git-credentials`) or of the checkout is
there to read. Each repetition names what it ran under in `isolation`
(`bwrap`, or `environment only` with the reason), and `--require-sandbox`
refuses to run without it. A timeout kills the worker's whole process group. The files the core opened under the system
directory come from the kernel, each hashed again after the run. Without
strace the record says that tracing was unavailable.
@@ -863,7 +877,8 @@ strace the record says that tracing was unavailable.
| `--state-var SECTION.VAR` | a variable of a Mednafen save state, as an observable |
| `--state-dir` | keep the save state taken at the end of the first repetition |
| `--core-repo`, `--core-commit`, `--build-command` | where the core build comes from, copied into the record |
| `--record [DIR]` | append the record as one line of `measurements/<emulator>.jsonl` in the checkout `DIR`, by default this one. Commit it |
| `--core-artifact URL` | where a downloaded `.so` came from, such as a buildbot nightly, copied into the record |
| `--record [DIR]` | append the record as one line of `measurements/<emulator>.jsonl` in the checkout `DIR`, by default this one. `DIR` must be the root of a git checkout, never its `measurements/` folder. Commit it |
| `--emulator` | the profile the record is filed under. By default the profile whose `cores:` names the `.so`, else the core's library name |
| `--subject` | a slug naming the test, shared by its reruns. By default derived from the claim and method, or from the inputs of a run or bench |
| `--supersedes ID` | the id of an earlier record of the same emulator this one corrects |
@@ -873,7 +888,8 @@ strace the record says that tracing was unavailable.
Observables are `frame_sha1` (last frame), `video_sha1` (every frame, chained),
`audio_sha1`, `state_sha1` (save state at the end), plus `state:SECTION.VAR`
and `frame_hashes` when asked for. A repetition that disagrees with the first
and `frame_hashes` when asked for, with `uniform_frames` (the sampled frames
of a single colour) beside it. A repetition that disagrees with the first
one is flagged. A `check` reports `inconclusive` instead of a comparison when,
in any variant, the compared observable differs between repetitions, a run did
not load, or a file changed during the run. A disagreement on another
@@ -905,6 +921,8 @@ and a gzip data file, then merged back into one line. Each keeps its first
id, which hashes the first file (`migrated_from_version: 1`), so the line
still proves it rebuilds that file. Two branches that each add a line
merge cleanly: `.gitattributes` marks `measurements/*.jsonl` `merge=union`.
`findings.jsonl` and `campaign-runs.jsonl` sit beside the records under the
same rules, written by `campaign.py`, and no profile takes those names.
`generate_site.py` compiles the records into files it never commits
(`sitemeasurements.py`): `public/data/measurements/index.json` (one item per
@@ -916,6 +934,81 @@ emulator. A permalink names the last commit that touched `measurements/`, and
the build warns when the records on disk differ from that commit. Rows are
sorted and nothing reads the clock, so two builds give the same bytes.
### campaign.py
Runs the real-condition campaign with nobody watching: for each emulator
profile in scope it gets the core and the test content, runs `core_probe.py`,
applies what the runs prove and queues the rest. It is a thin layer over
`core_probe.py`, the libretro buildbot and romfetch. RetroArch's
`--max-frames` and libretro.py were the other candidates: neither traces the
files a core opens nor writes a record.
```bash
python scripts/campaign.py run --system sony-playstation --budget 6h
python scripts/campaign.py run --only beetle_psx,swanstation --dry-run
python scripts/campaign.py run --commit --commit-every 10
python scripts/campaign.py status # done, failed, pending, and why
python scripts/campaign.py findings # what waits for review
python scripts/campaign.py gate # the checks behind a commit
python scripts/campaign.py install-timer # the autorun units, the campaign's among them
```
| Step | What happens |
|------|--------------|
| Core | The x86_64 Linux nightly from `buildbot.libretro.com`, for each name of the profile's `cores:` the buildbot publishes, downloaded again only when its `ETag` or `Last-Modified` changes. The record keeps the artifact URL, the `.so` sha256 and the `retro_get_system_info` version. With no nightly, the profile's `source_commit` is cloned and built once with `make` (`Makefile.libretro` first), the failure kept so it is not tried again for that commit. Host git only fetches into bare mirrors (`.cache/campaign/mirrors/`), one per repository, submodules included, that it alone touches: no working tree, never bound into a sandbox, and earlier working-tree clones are deleted. It runs with no user or system config, https as the only transport, and `-c` overrides emptying hooks, `core.fsmonitor`, `core.sshCommand`, `core.gitProxy`, credential helpers and every filter driver the mirror's config defines. The commit and each submodule at the commit its tree records are exported with `git archive` into a fresh folder, and `make` runs there in the same bubblewrap sandbox as a core, that folder the one writable path. The core copied out is the newest `*_libretro.so` that is a regular file inside that folder, opened without following links. The record's `build_command` says which isolation was used. A standalone emulator is `pending: standalone` |
| Content | Each system's games come from `tests/campaign/content.yml`, by catalog name and with the reason each was chosen. They are read from `.cache/test-content/<romfetch system>/<name>/`, each file checked against its `identity.json`. Missing content is fetched with `romfetch get <system> <name> --json` when romfetch is on `PATH`, in `RETROBIOS_ROMFETCH` or given with `--romfetch`. Otherwise the checks are `pending: content`, with romfetch's exit code and error code when it failed. Content never enters git, and a record keeps only its identity |
| Checks | `files` stages every declared file the collection holds and reads which ones the core opens (strace). `boot` runs the game with no file and with the declared one (`boot_file:`, else the entry of the game's region). An entry the `files` run left unread is staged `alone`, then where the core looked for it (`path`) and under the names it looked for (`alias`). `variants` runs each `.variants/` copy of a file the core opened. `bench` times the game, warm-up excluded, 5 repetitions |
| Rules | `campaign_rules.py`, a fixed table of pure functions (below). A field the table settles is written into the profile, with `evidence:` naming the record ids. Anything else is a line of `measurements/findings.jsonl` |
| Content reuse | A game stays on disk until the last emulator that needs it in the pass is done, then the campaign deletes it if it fetched it. Past `--cache-cap-gb` (default 20) the least recently used goes first. An entry marked `pinned:`, or one the campaign did not fetch, is never deleted |
| Observable | Field | Written |
|------------|-------|---------|
| Without the file, at least 2 screens the run with the file also shows (single-colour frames left out), in every repetition | `hle_fallback` | `true` |
| Without the file, no repetition loads the game or shows more than one colour, while the run with it reaches the game | `required`, `hle_fallback` | `true`, `false` |
| The core opened the file staged at another place it looked for | `path` | that path |
| The core opened the file staged under a name no entry declares | `aliases` | the name added |
| A staged file the game's region uses, never opened, staged alone either | nothing | flagged |
| Anything else (runs that disagree, a profile that says `required: true` for a file the core boots without) | nothing | open finding |
A run that agrees with the profile adds its record id to `evidence:` and
resolves an open finding of the same field. The profile is edited as text and
the edit is refused unless the result parses to the same profile with only
those fields changed.
State lives in `.cache/campaign/state.json` of the main checkout: each check
is `done`, `failed` (with the reason and its log) or `pending` (with the
reason), with the core sha256 and the profile digest it ran against. A pass
skips a check that is done or failed against the same core and profile, so it
takes new nightlies, changed profiles, newly fetched content and pending
checks. The campaign's own `evidence:` writes do not count as a profile change.
| Guard | Rule |
|-------|------|
| Lock | The host-wide autorun lock. Under the `campaign` job, autorun holds it and says so with `RETROBIOS_AUTO_LOCK_HELD` |
| Environment | `core_probe.py`, git and `make` get autorun's allowlist, romfetch the same plus its `ROMFETCH_*` settings. None sees a token or a password |
| Sandbox | Fails closed: without a working bubblewrap the pass is `refused` (exit 3) and nothing third-party runs, every probe is started with `--require-sandbox`, and a build is refused. `--allow-unsandboxed` overrides. Under autorun the refusal raises the `job:campaign` alert |
| Load | Before each run, waits while the 1-minute load is above a quarter of the CPUs, up to `--max-load-wait` seconds, then the pass stops as `paused` |
| Disk | Below `--min-free-gb` (default 30) the pass stops as `paused` |
| Budget | No check starts after `--budget` |
| Timeout | A run is killed after 900 seconds per repetition and variant, recorded `failed: timeout`, and the pass moves on |
| Crash | A run that dies is tried 3 times, then `failed` with its log |
| Network | Buildbot and romfetch source errors are retried after 10, 60 and 300 seconds |
Each pass appends one line to `measurements/campaign-runs.jsonl`: when it ran,
how it ended, the checks done, failed and pending by reason, the records and
the findings it added. Logs are `.cache/campaign/logs/`, kept 30 days.
`--commit` commits the records, findings and pass line in one commit, and
each changed profile in its own, behind `campaign.py gate`: the measurement
and campaign tests, `validate_schemas.py --source-only`, then `verify.py
--emulator` and `generate_pack.py --emulator` for each changed profile. A file
that had uncommitted changes before the pass is left alone. The nightly
`campaign` autorun job (06:00, after `nightly`) runs a 150-minute pass in the
automation worktree, gates it with `campaign.py gate` and commits with its
own rules, pushing only with `--push`.
Schemas: `campaign-finding.schema.json` and `campaign-run.schema.json`.
### Other tools
| Script | Purpose |