kmoody/libretro
1
0 Fork 0
mirror of https://github.com/Abdess/retroarch_system.git synced 2026-11-01 00:27:20 -05:00
Code Issues Packages Projects Releases Wiki Activity

chore: merge the boot oracle work

Browse Source
This commit is contained in:
Abdessamad Derraz committed 2026-10-27 17:20:35 +01:00
2 parents f9f688ae6e a17406edae
commit 417ed86405
9 files changed
+357 -53
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

wiki/tools.md
+73 -16
View File
@@ -706,10 +706,12 @@ python scripts/refresh_stale.py --only platforms,targets --jobs 6
### autorun.py
Runs the unattended jobs declared in `automation/jobs.yml`, one at a time.
The registry holds each job's commands, its gate, the artifacts computed
twice, the paths each commit may carry and the schedule. Changing any of
them is a reviewed commit.
Runs the jobs declared in `automation/jobs.yml`, one at a time. Nothing is
scheduled by default: a maintainer runs a job by hand, and a contributor who
wants a schedule on their own machine installs the timers below. The registry
holds each job's commands, its gate, the artifacts computed twice, the paths
each commit may carry and the schedule. Changing any of them is a reviewed
commit.
| Guard | Rule |
|-------|------|
@@ -845,6 +847,12 @@ python scripts/core_probe.py bench --core core.so --content game.cue \
python scripts/core_probe.py check --core core.so --content disc.cue \
--vary mpeg_card.bin=a.bin --vary mpeg_card.bin=b.bin --vary mpeg_card.bin= \
--observable video_sha1 --claim "..." --method "..." --record
# negative variants: the file, zeros of its size, its first half
python scripts/core_probe.py check --core core.so --content disc.cue \
--frame-hash-every 10 --vary scph5501.bin=bios/scph5501.bin \
--vary scph5501.bin=@zeros:524288 --vary scph5501.bin=@head:262144:bios/scph5501.bin \
--observable frame_hashes --claim "..." --method "..." --repeat 2 --record
```
Each repetition runs in a fresh process under `strace -f -e trace=openat,open`
@@ -859,15 +867,41 @@ writable. Every namespace is unshared, the network included, so nothing of
the user (`~/.config`, `~/.ssh`, `~/.git-credentials`) or of the checkout is
there to read. Each repetition names what it ran under in `isolation`
(`bwrap`, or `environment only` with the reason), and `--require-sandbox`
refuses to run without it. A timeout kills the worker's whole process group. The files the core opened under the system
refuses to run without it. The files the core opened under the system
directory come from the kernel, each hashed again after the run. Without
strace the record says that tracing was unavailable.
**Boot oracle.** Every repetition and every variant carries a `verdict` and
the `verdict_evidence` that decided it (record version 4), judged against
`tests/campaign/oracle.yml` (`--oracle`), in this order:
| Verdict | Evidence |
|---------|----------|
| `hang` | the worker was killed after `--timeout` (`timeout`) |
| `crash` | the worker died, by exit status or signal (`exit_status`). Its logs and sampled frames are rebuilt from the events it streamed |
| `boots` | a sampled frame or the last frame is a `game` reference frame of this content on this core (`reference_frame`), with the `stages` reached: `bios` for a screen only the console's BIOS draws, `game` for one of the content. The BIOS screens alone are never a boot: the stages go into the evidence of the verdict below |
| `bios_error` | a log line matches an error pattern of the core (`error_pattern`), or a frame is one of its error screens (`error_screen`) |
| `hang` | every sampled frame of the second half is one colour (`uniform_tail`) |
| `unknown` | none of these (`no_evidence`) |
A variant's verdict is the one all its repetitions agree on, else `unknown`
(`repetitions_disagree`). A check lists them in `check.result.verdicts`, and
`inputs.oracle` keeps what they were judged against. The oracle finds a
content by the sha1 of the file handed to the core and a core by its library
name. A reference frame is recorded once with the full BIOS set and reviewed
by looking at its PNG. An error pattern is a log line after which the core
stops, read from its source: a line a core logs before it falls back to a
built-in BIOS is not one. Frames that differ from another run never make a
boot.
| Option | Effect |
|--------|--------|
| `--system-file DEST=SRC` | copy SRC into a staged system directory as DEST |
| `--content-label` | identity of the content, such as its Redump name, copied into the record |
| `--work-dir`, `--timeout` | scratch directory for the staged runs, and seconds allowed per repetition (default 1800) |
| `--work-dir`, `--timeout` | scratch directory for the staged runs, and seconds allowed per repetition (default 1800), after which the run is killed and judged a hang |
| `--oracle FILE` | reference frames, error patterns and error screens the verdicts are judged against (default `tests/campaign/oracle.yml`) |
| `--repeat N` | runs of each variant, each in a fresh process (`run` defaults to 1, `check` to 2, `bench` to 5) |
| `--vary DEST=SRC` | (`check`) one variant per occurrence. An empty SRC leaves DEST absent, `@zeros:SIZE` stages SIZE zero bytes and `@head:SIZE:PATH` the first SIZE bytes of PATH |
| `--option KEY=VALUE` | core option. Unset options take the first value the core declares |
| `--frames`, `--warmup` | frames timed, and frames run before timing starts |
| `--press FRAME:BUTTON[:LEN]` | hold a libretro joypad button on port 0 (`a`, `b`, `start`, `up`...). A menu may need 15 frames or more to see it |
@@ -947,6 +981,9 @@ files a core opens nor writes a record.
python scripts/campaign.py run --system sony-playstation --budget 6h
python scripts/campaign.py run --only beetle_psx,swanstation --dry-run
python scripts/campaign.py run --commit --commit-every 10
python scripts/campaign.py run --only pcsx_rearmed --bench # with benchmarks
python scripts/campaign.py references --only beetle_psx --system sony-playstation
python scripts/campaign.py references --only beetle_saturn --content-file other.yml # another content list
python scripts/campaign.py status # done, failed, pending, and why
python scripts/campaign.py findings # what waits for review
python scripts/campaign.py gate # the checks behind a commit
@@ -957,18 +994,37 @@ python scripts/campaign.py install-timer # opt-in: writes the autorun units,
|------|--------------|
| Core | The x86_64 Linux nightly from `buildbot.libretro.com`, for each name of the profile's `cores:` the buildbot publishes, downloaded again only when its `ETag` or `Last-Modified` changes. The record keeps the artifact URL, the `.so` sha256 and the `retro_get_system_info` version. With no nightly, the profile's `source_commit` is cloned and built once with `make` (`Makefile.libretro` first), the failure kept so it is not tried again for that commit. Host git only fetches into bare mirrors (`.cache/campaign/mirrors/`), one per repository, submodules included, that it alone touches: no working tree, never bound into a sandbox, and earlier working-tree clones are deleted. It runs with no user or system config, https as the only transport, and `-c` overrides emptying hooks, `core.fsmonitor`, `core.sshCommand`, `core.gitProxy`, credential helpers and every filter driver the mirror's config defines. The commit and each submodule at the commit its tree records are exported with `git archive` into a fresh folder, and `make` runs there in the same bubblewrap sandbox as a core, that folder the one writable path. The core copied out is the newest `*_libretro.so` that is a regular file inside that folder, opened without following links. The record's `build_command` says which isolation was used. A standalone emulator is `pending: standalone` |
| Content | Each system's games come from `tests/campaign/content.yml`, by catalog name and with the reason each was chosen. They are read from `.cache/test-content/<romfetch system>/<name>/`, each file checked against its `identity.json`. Missing content is fetched with `romfetch get <system> <name> --json` when romfetch is on `PATH`, in `RETROBIOS_ROMFETCH` or given with `--romfetch`. Otherwise the checks are `pending: content`, with romfetch's exit code and error code when it failed. Content never enters git, and a record keeps only its identity |
| Checks | `files` stages every declared file the collection holds and reads which ones the core opens (strace). `boot` runs the game with no file and with the declared one (`boot_file:`, else the entry of the game's region). An entry the `files` run left unread is staged `alone`, then where the core looked for it (`path`) and under the names it looked for (`alias`). `variants` runs each `.variants/` copy of a file the core opened. `bench` times the game, warm-up excluded, 5 repetitions |
| Checks | `files` stages every declared file the collection holds (the full set) and reads which ones the core opens (strace). `none` runs the game with no BIOS at all. `negative` runs, for each declared file of the game's region the collection holds, the file alone, then zeros of its size, its first half and up to two other regions' dumps staged under its name. An entry neither the full set nor its own run opened is staged where the core looked for it (`path`), else flagged. The `boot_file:` entry is also staged under the names the core looked for with no BIOS (`alias`). `variants` runs each `.variants/` copy of a file the core opened. Each runs twice from a fresh process and is judged by the boot oracle. `bench` (only with `--bench`) times the game once the full set boots, warm-up excluded, 5 repetitions |
| Rules | `campaign_rules.py`, a fixed table of pure functions (below). A field the table settles is written into the profile, with `evidence:` naming the record ids. Anything else is a line of `measurements/findings.jsonl` |
| Content reuse | A game stays on disk until the last emulator that needs it in the pass is done, then the campaign deletes it if it fetched it. Past `--cache-cap-gb` (default 20) the least recently used goes first. An entry marked `pinned:`, or one the campaign did not fetch, is never deleted |
| Observable | Field | Written |
|------------|-------|---------|
| Without the file, at least 2 screens the run with the file also shows (single-colour frames left out), in every repetition | `hle_fallback` | `true` |
| Without the file, no repetition loads the game or shows more than one colour, while the run with it reaches the game | `required`, `hle_fallback` | `true`, `false` |
| Verdicts | Field | Written |
|----------|-------|---------|
| `none` boots | `hle_fallback` | `true` |
| `none` fails (`bios_error`, `crash`, `hang`) and the file alone boots | `required`, `hle_fallback` | `true`, `false` |
| Zeros of the file's size refused while the file boots | `validation` | a content check is proven. Agreeing adds evidence, a profile with no md5, sha1, sha256, crc32 or adler32 is an open finding |
| Zeros used while the file boots | `validation` | no content check. A profile that declares one is an open finding |
| Its first half refused while zeros are used | `validation` | `size`, added when missing |
| Its first half used | `validation` | no size check. A profile that declares one is an open finding |
| The core opened the file staged at another place it looked for | `path` | that path |
| The core opened the file staged under a name no entry declares | `aliases` | the name added |
| A staged file the game's region uses, never opened, staged alone either | nothing | flagged |
| Anything else (runs that disagree, a profile that says `required: true` for a file the core boots without) | nothing | open finding |
| Copies of one file whose verdicts differ | nothing | open finding |
| Anything else (an `unknown`, a profile that says `required: true` for a file the core boots without) | nothing | open finding |
A wrong file is refused when the run reports `bios_error`, or boots without
the `bios` screens the right file shows: the core fell back to a built-in
BIOS. It is used when it turns the run into a `hang` or a `crash` that the
run with no BIOS does not show. Another region's dump is recorded with its
verdict, and the region claims are settled by their own test. Evidence that a
record from before the oracle wrote no longer counts: when a verdict settles a
field again, those ids leave its `evidence:`.
`references` runs the full set once per core and content with every sampled
frame written as PNG under `tmp/campaign/references/<core>/<content>/`, and
prints the record id with the screens that stay on for 3 samples or more in
every repetition. A reviewer opens those PNGs and writes the chosen frames
into `tests/campaign/oracle.yml` with their stage and what they show.
A run that agrees with the profile adds its record id to `evidence:` and
resolves an open finding of the same field. The profile is edited as text and
@@ -977,8 +1033,9 @@ those fields changed.
State lives in `.cache/campaign/state.json` of the main checkout: each check
is `done`, `failed` (with the reason and its log) or `pending` (with the
reason), with the core sha256 and the profile digest it ran against. A pass
skips a check that is done or failed against the same core and profile, so it
reason), with the core sha256, the profile digest, the oracle sha256 and the
record version it ran against. A pass skips a check that is done or failed
against the same four, so it
takes new nightlies, changed profiles, newly fetched content and pending
checks. The campaign's own `evidence:` writes do not count as a profile change.
@@ -990,8 +1047,8 @@ checks. The campaign's own `evidence:` writes do not count as a profile change.
| Load | Before each run, waits while the 1-minute load is above a quarter of the CPUs, up to `--max-load-wait` seconds, then the pass stops as `paused` |
| Disk | Below `--min-free-gb` (default 30) the pass stops as `paused` |
| Budget | No check starts after `--budget` |
| Timeout | A run is killed after 900 seconds per repetition and variant, recorded `failed: timeout`, and the pass moves on |
| Crash | A run that dies is tried 3 times, then `failed` with its log |
| Timeout | A repetition is killed after 900 seconds and judged `hang`. A core_probe call that outlives all its repetitions is `failed: timeout` |
| Crash | A core that dies is a `crash` verdict in its record. A core_probe call that fails for another reason is tried 3 times, then `failed` with its log |
| Network | Buildbot and romfetch source errors are retried after 10, 60 and 300 seconds |
Each pass appends one line to `measurements/campaign-runs.jsonl`: when it ran,