Commit Graph
1904 Commits
Author SHA1 Message Date
Abdessamad Derraz 2b94be9516 fix: count a game only under its emulator systems
One catalog name is a PlayStation disc and a Saturn disc. The coverage table counted a record for every system listing its label, so the Saturn row showed the PlayStation cores. A record now counts only where its emulator's profile names the system, runs on games outside the list are named apart, and two ids of one console share one row.
2026-10-27 16:22:42 +01:00
Abdessamad Derraz cdaaf7529f feat: add region checks and site coverage
region_check.py runs each region game of content.yml with every regional BIOS, its own only, the others only, and the region-filtered pack, and reads the outcome from the records: the file opened settles region:, the verdicts say whether the BIOS is locked. System pages show the outcome with its records, the Measurements page lists which systems are tested and why the others are not. The campaign text says a maintainer starts each pass.
2026-10-27 16:17:56 +01:00
Abdessamad Derraz cfc7a13be2 feat: add the standalone emulator probe
standalone_probe.py runs the release binary a platform installs (EmuDeck's asset rule for DuckStation and PCSX2), unpacked without running the AppImage runtime, in core_probe's allowlist sandbox with a private Xvfb, strace and core_probe's oracle and record functions. Records go to measurements/<emulator>.jsonl.
2026-10-27 16:17:49 +01:00
Abdessamad Derraz d6537d5838 feat: list test games per system and region
The content list now covers the BIOS-required systems in order of the users a missing BIOS stops, one game per region where a profile ties the BIOS to a territory (PS1, Saturn, Mega CD, Master System, N64, GameCube, DS), the same title in each region so the screens compare. Systems no catalog romfetch reads covers are listed under uncovered with the set or game wanted and romfetch's error code.
2026-10-27 16:17:45 +01:00
Abdessamad Derraz 38dd7401fd feat: keep the campaign timer as an opt-in
campaign.py install-timer writes the autorun units and enables nothing unless --enable is passed. The docs say a contributor who wants a schedule on their own machine installs the timer.
2026-10-27 15:46:20 +01:00
Abdessamad Derraz 44ab3febf4 feat: settle profile fields from boot verdicts
The campaign runs the full set, no BIOS, and per file the file alone with zeros, its first half and other regions' dumps under its name. required, hle_fallback and validation are read from the verdicts, never from frames that differ. Benchmarks run only with --bench. The timer command is gone: a maintainer starts each pass. references lists stable screens for the oracle.
2026-10-27 15:45:22 +01:00
Abdessamad Derraz e4c7ded489 feat: judge each probe run with a boot oracle
core_probe writes record version 4. Each run is boots, bios_error, crash, hang or unknown, from reference frames, core error patterns and screens, and the process status. A crash or a timeout is a verdict now, rebuilt from the events the worker streamed. --vary accepts @zeros:SIZE and @head:SIZE:PATH, and run takes --repeat.
2026-10-27 15:45:22 +01:00
Abdessamad Derraz aa6ee6db1c feat: add the boot verdict to the record schema
Record version 4 carries verdict and verdict_evidence on every repetition and variant, verdicts on a check result, and inputs.oracle. Version 3 records stay valid.
2026-10-27 15:25:30 +01:00
Abdessamad Derraz 9c8dbcd99a chore: merge the automatic test campaign 2026-10-27 14:05:18 +01:00
Abdessamad Derraz 332c217155 feat: pass GITHUB_TOKEN to declared API steps
A step marked github_token: true receives the token. autorun accepts the key only on check_freshness.py, refresh_stale.py and python -m scripts.scraper.* tracked in this repository, refuses it on any other step and in the campaign job. The nightly steps declare it.
2026-10-27 13:41:40 +01:00
Abdessamad Derraz 57436839a8 fix: start each open finding with a capital
The statement of a finding follows a full stop on the Measurements page.
2026-10-27 13:07:33 +01:00
Abdessamad Derraz 5f3c836db9 chore: merge origin/main into the campaign work 2026-10-27 13:02:18 +01:00
Abdessamad Derraz 37d2c14e1d chore: keep campaign prose to the style rules
No clause semicolons in the campaign docstring and content list, and a help text for every option.
2026-10-27 13:02:15 +01:00
Abdessamad Derraz 8ab24a97d3 fix: test the new setup labels and skip 3.8 builds
The site tests read the SHARE drive label and the line-mode wording
from the source, and the pack-systems test needs generate_pack.py,
which needs Python 3.9.
2026-10-27 13:02:03 +01:00
Abdessamad Derraz a67dc339bc chore: merge main into the audit 17 answers
# Conflicts:
#	install.ps1
#	install.sh
#	install/launchers/index.json
2026-10-27 12:48:04 +01:00
Abdessamad Derraz e2bc06f216 fix: keep the copy hint whole under a long list
The browser hint goes below the list, which scrolls: three folders at
40x12 cut it. The test browses a folder of its own, not the host's /home.
2026-10-27 12:47:54 +01:00
Abdessamad Derraz c138270edc refactor: keep commit rules in a tracked module
The rules lived only in the local commit-msg hook, which CI and other clones do not have, so autorun could only check a subject's shape there. scripts/commit_rules.py now holds them, the hook calls it, and autorun checks every job subject with it when the job list loads.
2026-10-27 12:41:06 +01:00
Abdessamad Derraz b439371579 fix: take a backup folder on one-owner storage 2026-10-27 12:39:28 +01:00
Abdessamad Derraz 6fd831fc67 fix: keep pack system names out of the terminal
A pack's system titles, names and ids are read only without control
characters, and the systems list and the same-choice line go through
screen_text. The --yes row of the option table says it implies
--no-input.
2026-10-27 12:36:15 +01:00
Abdessamad Derraz 31c574aab0 fix: line mode and screen share one decision
Answers code_audit_17 and tui_audit_10: the folder found after o, SHARE
drive roots, one same-choice command, a counted confirm step in line
mode, counts from the narrowed list, read-only advice without sudo,
systems recorded in packs, Ctrl-Z stopping the whole job, and the
wording, layout and docs items.
2026-10-27 12:24:26 +01:00
Abdessamad Derraz f5e98bfda6 refactor: fetch builds into bare mirrors
Host git only fetches into bare mirrors no sandbox ever sees, https only, with no user or system config and overrides emptying hooks, fsmonitor, ssh and proxy commands and every filter driver the mirror defines. Each build is a git archive of the commit and of each submodule commit from its own mirror. Old working-tree clones are deleted. The core copied out must be a regular file inside the build folder, opened without following links.
2026-10-27 12:19:55 +01:00
Abdessamad Derraz fdcf120a7d refactor: build cores from an exported tree
make runs on a git archive export of the commit and its submodules, the clone and its .git never bound into the sandbox. Host-side git runs with no user or system config, no hooks, no fsmonitor and the file and ext transports refused.
2026-10-27 12:07:16 +01:00
Abdessamad Derraz fd449fdae3 refactor: refuse unsandboxed campaign runs
Without a working bubblewrap the pass is refused and nothing third-party runs, probes require the sandbox and builds are refused, unless --allow-unsandboxed is given. A job report holding an alert raises an autorun alert.
2026-10-27 11:53:35 +01:00
Abdessamad Derraz e440073562 refactor: build the core sandbox as an allowlist
bwrap now binds only /usr, the /lib* and /bin links and the files of /etc the loader, fonts, TLS, the clock and user lookups read. Home, /tmp and /mnt are empty, the core, content and staged system directory read-only, the scratch folder the one writable path, every namespace unshared. --require-sandbox refuses to run without it. The worker imports nothing from the repository.
2026-10-27 11:53:35 +01:00
Abdessamad Derraz 95ca227bce refactor: run cores and builds in a sandbox
core_probe starts its worker with only the --env values, the worker clears its own environment before loading the core, and a working bubblewrap runs it with a read-only tree, no network and the user's credentials masked. The record names the isolation used. A timeout kills the whole process group. campaign.py starts core_probe, git, make and romfetch with the allowlist, make inside the same sandbox. --record refuses a folder that is not a checkout root.
2026-10-27 11:48:22 +01:00
Abdessamad Derraz e5cc1b5255 refactor: keep tokens out of autorun job steps
Every step, gate and recomputation runs with an allowlisted environment (PATH, HOME, LANG, LC_*, TZ, TMPDIR and the variables its env: key names). Names holding TOKEN, SECRET or PASS are dropped and refused in the registry. Only the push and the gh calls that read CI get GITHUB_TOKEN.
2026-10-27 11:48:22 +01:00
Abdessamad Derraz ee7319d838 feat: add the real-condition campaign runner
campaign.py takes buildbot nightlies (or a build of the pinned commit), the test content of tests/campaign/content.yml through romfetch, and runs core_probe for the files, boot, path, alias, variants and bench checks. A fixed rule table settles hle_fallback, required, path and aliases with the record ids as evidence, the rest goes to measurements/findings.jsonl. Resumable state, load, disk and budget guards, retries, content reference counting with an LRU cap, and an autorun job after the nightly.
2026-10-27 11:38:01 +01:00
Abdessamad Derraz faa79c5920 refactor: check job subjects with the repo hook
The runner carried its own copy of the refused vocabulary, which put those words in a tracked file. It now runs the commit-msg hook on each job's subject when the job list loads, and checks only the shape where no hook exists.
2026-10-27 11:08:19 +01:00
Abdessamad Derraz 07c34781e7 chore: merge the autorun runner and jobs 2026-10-27 10:41:53 +01:00
Abdessamad Derraz 343a4e8bd7 feat: add the autorun job runner
Runs the jobs of automation/jobs.yml under one host lock, with load and disk guards, one idle retry on a host misread, artifacts computed twice, explicit-path commits in .cache/autorun/worktree, an opt-in push (--push) and a CI circuit breaker. First jobs: nightly refresh chain, large-files asset dry run, disk cleanup and check, weekly bios integrity. install-timers writes systemd user units without enabling them.
2026-10-27 10:40:10 +01:00
Abdessamad Derraz 79c27b3798 feat: read saved findings in refresh_stale
--findings FILE takes a check_freshness.py --json report written earlier, kept to the --only areas, so one full check can drive a refresh of a few areas.
2026-10-27 10:40:05 +01:00
Abdessamad Derraz 0fbf920e45 chore: merge one-file measurement records 2026-10-27 10:24:41 +01:00
Abdessamad Derraz a9301fbdc0 chore: merge line mode on the screen's decisions
# Conflicts:
#	tests/test_install_tui.py
2026-10-27 10:20:31 +01:00
Abdessamad Derraz 77bdf7c206 fix: line mode answers in the merged tests
The TERM probe test answers the setup question, first-install keeps
its button name on one line and under 400 words, and --page of the
installer doc script is named on the page.
2026-10-27 10:19:37 +01:00
Abdessamad Derraz 11c65f40fc chore: align card depth test and document page 2026-10-27 10:14:56 +01:00
Abdessamad Derraz 102108f849 fix: hold launcher tests off the host's arkos 2026-10-27 10:14:55 +01:00
Abdessamad Derraz da68f19411 refactor: keep one full record per line
Each line of measurements/<emulator>.jsonl now holds the whole run:
every repetition, core logs, per-frame hashes and raw timings. The gzip
data files, the data schema and the split and join code are gone, and
core_probe --record appends one line. record_version moves to 3.

The seven records are rewritten once: each new line equals its old line
joined with its data file, and rebuilds the first-layout file its id
hashes.
2026-10-27 09:57:39 +01:00
Abdessamad Derraz 8c1887fd0c chore: merge main into the audit answers 2026-10-27 09:57:16 +01:00
Abdessamad Derraz 559494ae43 fix: line mode reads the screen's decisions
Answers code_audit_16 and tui_audit_9: share drives give their bios
folder, card markers stop at mount points, a model finds its family,
the folder question quits on q, pages move by the rows drawn, and a
folder the installer proposed is created without the guard.
2026-10-27 09:55:50 +01:00
Abdessamad Derraz 9cf0f9177d chore: merge installer tables from code
# Conflicts:
#	install.ps1
#	install.sh
#	install/launchers/index.json
#	scripts/measurement_records.py
#	tests/test_install_tui.py
#	wiki/installer.md
2026-10-27 09:48:44 +01:00
Abdessamad Derraz 47ed945b4a feat: write installer tables from install.py
scripts/generate_installer_doc.py rewrites the marked tables of
wiki/installer.md: options from build_parser(), events from EVENT_KEYS,
exit codes from the sys.exit calls, RETROBIOS_PAD values and the keys of
each route from TUI_ROUTES, the one table the bar, the help and the stop
line now read. Prose cells are kept, a row the code gains stops the
script until its text is written, and --check runs in the tests. The
site drops the markers.

The full screen falls back to the line questions where the terminal's
terminfo moves no cursor the ANSI way (vt52, glass terminals), and --tui
names why. The line mode draws no escape code with TERM unset, dumb or
such a terminal. Ctrl-Z typed on the screen stops the run as SIGTSTP
does, the terminal handed back and redrawn on fg. A window too small
names 40x12 and its own size.

Pty tests check the modes, the cursor and the main screen after Ctrl-C,
Ctrl-D, SIGTERM, SIGHUP and Ctrl-Z then fg. Tests from a mutmut run over
the device filter, the card bounds and the WSL paths raise its score
from 62/71 to 71/71. The parser lists its options in the page's order,
with NAME, PATH, NAMES and N as arguments.
2026-10-27 09:47:44 +01:00
Abdessamad Derraz 9c34e8891f chore: merge origin/main 2026-10-27 09:11:43 +01:00
Abdessamad Derraz 571e50f2a8 refactor: read measurements from the local tree
The site compiles the index and the SQLite files from measurements/
instead of a pinned clone. The pin, the sync script, the CI clone steps,
the freshness area and its refresh handler are gone. core_probe --record
writes into measurements/ of the checkout by default, and a test fails
when a line or data file HEAD holds changed or vanished.
2026-10-27 09:11:12 +01:00
Abdessamad Derraz 11a4698e06 chore: merge line and screen setup parity 2026-10-27 08:43:49 +01:00
Abdessamad Derraz 4fbbd2c760 fix: one setup decision for line mode and screen
Line mode finds mounted cards, shows its menu and names the folder for a
pack, asks the folder instead of taking a default, and lists setups in the
screen's order. Devices have one spelling each, matched without case,
spaces or punctuation, and a device several systems run lists them all.
--tui --dest reads a card as --plain does. Card boundaries read every
parent up to a mount point from mountinfo. --offline with --yes counts the
files it leaves out as pending, never as failed. The P3 points of
code_audit_15 and tui_audit_8 follow.
2026-10-27 08:43:08 +01:00
Abdessamad Derraz 087e5b4d73 feat: show measurements on the site
Each emulator with records gets a Tested block: claim, result, method, computer, core build and the permalink of its record at the pin. A Measurements page filters every test in the browser from the compact index, and the Data page lists measurements.sqlite with its tables. One SQLite file per emulator is linked from its page. wiki/measurements.md says how a test runs, held to the Start here budgets.

Both workflows fetch the pinned commit alone with git before the build, and the deploy fails without it.
2026-10-27 08:30:16 +01:00
Abdessamad Derraz 6109d7abd5 feat: split probe records into line and sidecar
A record is now one line of records/<emulator>.jsonl in the measurements repository, under 20 KB, and a gzip sidecar holding every repetition, the core logs and the raw timings, its sha256 in the line. record_version 2, core-probe-data.schema.json for the sidecar. core_probe --record appends to a measurements clone and refuses a directory inside this repository's tracked tree.

The seven records of issue 85 moved to Abdess/retrobios-measurements: each line plus its sidecar rebuilds the first file, and keeps its first id.
2026-10-27 08:30:07 +01:00
Abdessamad Derraz e627f0cf49 chore: merge the core probe and issue 85 runs 2026-10-27 07:02:05 +01:00
Abdessamad Derraz fecc6d9d32 chore: merge round 5 search fixes 2026-10-27 06:54:51 +01:00
Abdessamad Derraz 57c5d3b7ce chore: check the PlayStation opening names a BIOS
The opening line follows the profiles' required and hle_fallback data: it reads "No file needed with ... Only ... require scph5501.bin" since most PlayStation cores run without the BIOS. The test asserted the words "Start with"; it now asserts what it meant, a linked scph BIOS before the first card.
2026-10-27 06:54:13 +01:00