Commit Graph
159 Commits
Author SHA1 Message Date
Abdessamad Derraz e6f7ea8f07 chore: merge the region reference frames 2026-10-27 23:07:23 +01:00
Abdessamad Derraz 6d3045cbb3 fix: no region verdict from frames that differ
The first runs showed it: beetle_psx reaches the game of the US disc on the Japanese and European BIOS, through a different licence screen, so its frames differ from the full run while the oracle says boots. The interim rule read those frames as a failure. A variant now boots on the same frames as the full run or by its oracle verdict, and otherwise has no answer.
2026-10-27 22:28:43 +01:00
Abdessamad Derraz dd64434122 chore: merge main 2026-10-27 22:05:47 +01:00
Abdessamad Derraz 1c613dc4b4 fix: take the probe lock per standalone run
standalone_probe now holds the host probe lock for one run, its load check included, as core_probe does, and the docs no longer wrap region_check or standalone_probe in flock.
2026-10-27 21:08:47 +01:00
Abdessamad Derraz f8b54cc53a chore: merge the per-run probe lock 2026-10-27 21:07:57 +01:00
Abdessamad Derraz 9327d3f97f docs: two campaign passes share one state 2026-10-27 21:02:20 +01:00
Abdessamad Derraz 5fd691a7db docs: say which lock a campaign pass keeps 2026-10-27 21:02:15 +01:00
Abdessamad Derraz fabd431adb feat: take the host probe lock per run
core_probe takes probe.lock itself around one execution, load check included, and releases it when the run ends. A campaign pass never holds it between runs, so two passes interleave. RETROBIOS_PROBE_LOCK names another file, RETROBIOS_PROBE_LOCK_HELD says the caller holds it. Tests use their own lock file.
2026-10-27 21:02:03 +01:00
Abdessamad Derraz 77a1a7285a feat: write the romset table from the profiles
The FAQ table of arcade romsets was typed by hand and told readers to
collect MAME 0.160 and FinalBurn Neo 1.0.0.2 sets for cores whose source
states 0.159 and 1.0.0.03. generate_site now writes it between generated
markers from the profiles, and a test holds the page to the generator.
An emulator page names where its version is read (the file, the tag and
the commits after it, the package), and says when the code states none
or the version could not be confirmed. HBMAME no longer claims a MAME
romset. tools.md and profiling.md document version_basis and the two
modes.
2026-10-27 18:59:30 +01:00
Abdessamad Derraz 21e6520fdf fix: let region runs wait for a busy host
core_probe refuses to record above its load threshold. region_check gave up on the first refusal, so twelve runs ended at once on a host still busy from the previous run. It now tries a refused run again every 15 seconds for up to --max-load-wait seconds, the threshold unchanged.
2026-10-27 18:17:24 +01:00
Abdessamad Derraz 70c3bbd853 feat: settle a file only when the run uses it
hle_fallback needs the file to change the run: Beetle PSX hashes every file of its folder, so opened is not used. PCSX-ReARMed hides the BIOS logo by default: its BIOS load line stands for the bios stage. A check runs again only when the oracle entries of its content and core change, and the rules run on every pass from the kept records.
2026-10-27 17:58:48 +01:00
Abdessamad Derraz 417ed86405 chore: merge the boot oracle work 2026-10-27 17:20:35 +01:00
Abdessamad Derraz f9f688ae6e fix: hold the probes to the repository checks
The new probes now refuse abbreviated options, read the GitHub token through github_headers, type every signature and document every option. The site build watches region_check.py and the content list, and validate.yml's tests/** counts as watching the list. The testing guide names the two new test modules.
2026-10-27 17:05:10 +01:00
Abdessamad Derraz 4cc0a34d3c docs: describe the region and standalone probes 2026-10-27 16:17:56 +01:00
Abdessamad Derraz a17406edae feat: let the campaign read another content list
--content-file names the test content list for run and references.
2026-10-27 15:53:25 +01:00
Abdessamad Derraz 873b645c0c chore: merge main 2026-10-27 15:52:46 +01:00
Abdessamad Derraz 3f93da6394 docs: show boot verdicts on the measurement pages
The Tested blocks and the explorer show each variant's verdict. A frame comparison from before the oracle shows no verdict. The SQLite variants table gains verdict columns. The method page says what each verdict means and how a file is proven.
2026-10-27 15:52:29 +01:00
Abdessamad Derraz 38dd7401fd feat: keep the campaign timer as an opt-in
campaign.py install-timer writes the autorun units and enables nothing unless --enable is passed. The docs say a contributor who wants a schedule on their own machine installs the timer.
2026-10-27 15:46:20 +01:00
Abdessamad Derraz 50e84866cf docs: autorun schedules nothing by default 2026-10-27 15:25:06 +01:00
Abdessamad Derraz 332c217155 feat: pass GITHUB_TOKEN to declared API steps
A step marked github_token: true receives the token. autorun accepts the key only on check_freshness.py, refresh_stale.py and python -m scripts.scraper.* tracked in this repository, refuses it on any other step and in the campaign job. The nightly steps declare it.
2026-10-27 13:41:40 +01:00
Abdessamad Derraz f5e98bfda6 refactor: fetch builds into bare mirrors
Host git only fetches into bare mirrors no sandbox ever sees, https only, with no user or system config and overrides emptying hooks, fsmonitor, ssh and proxy commands and every filter driver the mirror defines. Each build is a git archive of the commit and of each submodule commit from its own mirror. Old working-tree clones are deleted. The core copied out must be a regular file inside the build folder, opened without following links.
2026-10-27 12:19:55 +01:00
Abdessamad Derraz fdcf120a7d refactor: build cores from an exported tree
make runs on a git archive export of the commit and its submodules, the clone and its .git never bound into the sandbox. Host-side git runs with no user or system config, no hooks, no fsmonitor and the file and ext transports refused.
2026-10-27 12:07:16 +01:00
Abdessamad Derraz 34963a732f docs: describe the allowlist sandbox 2026-10-27 11:53:35 +01:00
Abdessamad Derraz a76129f0df docs: describe the campaign and its sandbox 2026-10-27 11:48:25 +01:00
Abdessamad Derraz 07c34781e7 chore: merge the autorun runner and jobs 2026-10-27 10:41:53 +01:00
Abdessamad Derraz 343a4e8bd7 feat: add the autorun job runner
Runs the jobs of automation/jobs.yml under one host lock, with load and disk guards, one idle retry on a host misread, artifacts computed twice, explicit-path commits in .cache/autorun/worktree, an opt-in push (--push) and a CI circuit breaker. First jobs: nightly refresh chain, large-files asset dry run, disk cleanup and check, weekly bios integrity. install-timers writes systemd user units without enabling them.
2026-10-27 10:40:10 +01:00
Abdessamad Derraz 0fbf920e45 chore: merge one-file measurement records 2026-10-27 10:24:41 +01:00
Abdessamad Derraz 11c65f40fc chore: align card depth test and document page 2026-10-27 10:14:56 +01:00
Abdessamad Derraz da68f19411 refactor: keep one full record per line
Each line of measurements/<emulator>.jsonl now holds the whole run:
every repetition, core logs, per-frame hashes and raw timings. The gzip
data files, the data schema and the split and join code are gone, and
core_probe --record appends one line. record_version moves to 3.

The seven records are rewritten once: each new line equals its old line
joined with its data file, and rebuilds the first-layout file its id
hashes.
2026-10-27 09:57:39 +01:00
Abdessamad Derraz 9cf0f9177d chore: merge installer tables from code
# Conflicts:
#	install.ps1
#	install.sh
#	install/launchers/index.json
#	scripts/measurement_records.py
#	tests/test_install_tui.py
#	wiki/installer.md
2026-10-27 09:48:44 +01:00
Abdessamad Derraz fb15ca68bb docs: name the sync_measurements path options
The option audit test failed on --clone and --pin, and two docstrings in
measurement_records.py joined clauses with a semicolon.
2026-10-27 09:47:36 +01:00
Abdessamad Derraz ec2b7081ad docs: describe measurements/ in the repository 2026-10-27 09:11:12 +01:00
Abdessamad Derraz 7fa8d2d019 docs: document measurement sync clone and pin 2026-10-27 09:02:28 +01:00
Abdessamad Derraz b02dd77d37 docs: describe the measurements repository
core_probe options --emulator, --subject and --supersedes, the record split into line and sidecar, sync_measurements.py and what the site build compiles from the pin.
2026-10-27 08:30:20 +01:00
Abdessamad Derraz cb5bb738cb fix: core probe through run_cli, documented
The probe skipped run_cli, accepted option prefixes in its shared options, left three options undocumented and was missing from the published schemas. The worker runs under -I, so the script directory is put back on sys.path before importing common.
2026-10-27 07:11:45 +01:00
Abdessamad Derraz 8b12685c16 feat: settle a probe check per observable
A check is inconclusive when the compared observable differs between repetitions, a run fails to load or a file changes during it. A disagreement on another observable stays flagged without voiding the comparison. Core log lines name the staged directories instead of the host paths.
2026-10-27 05:53:52 +01:00
Abdessamad Derraz 9690579f34 feat: record env files by name and sha256
A path given to --env is a detail of the host that ran the probe. The record keeps the file name and its sha256, which is what the run depended on.
2026-10-27 05:34:31 +01:00
Abdessamad Derraz 00b034612f feat: add headless libretro core probe
core_probe.py loads a libretro core with no window, runs it for a
number of frames and reports the files it opened (strace), its log
lines, frame, audio and save-state hashes and its speed. bench repeats
a run after warm-up and gives median, mean, deviation and range; check
swaps one system file between runs and compares one observable.

--record writes a JSON record with the core build, frontend commit,
host and every input by hash, validated against
core-probe-record.schema.json. A busy host is refused, and a
repetition that disagrees with the first is flagged.
2026-10-27 05:18:10 +01:00
Abdessamad Derraz 4b11a43a86 feat: version check reads tags and numbers
A project that tags without releases (eightyone, higan) was never compared,
and a core_version naming no tag (parallel-launcher 9.0.5 beside v9.0-6)
silenced the passed-release warning. The newest tag stands in when it is a
version number, and an untagged core_version is compared by number within
one scheme. --emulator is repeatable: a stored value kept only the last.
2026-10-27 04:16:31 +01:00
Abdessamad Derraz 1fc85ce601 docs: name --fresh in the tools page 2026-10-26 21:28:36 +01:00
Abdessamad Derraz 2ee777824a feat: name doubled destinations in exports
A platform list that declares one path for two files fails one of the
two checks on every install. The export report lists each declaration
with its hash and the emulators whose profile reads that file.
2026-10-24 02:42:04 +02:00
Abdessamad Derraz de55360b56 feat: flag a stale source-paths record
check_freshness reports source-paths.json STALE when it records a tree no profile pins any more, and tools.md names sitesources.py --fetch where pins move.
2026-10-23 20:04:12 +02:00
Abdessamad Derraz 35e7300465 feat: add --declared-only and the events schema
--declared-only keeps the entries the platform's own list declares
(cores: null), each slot settled to the platform's copy with no core
alternative left behind. Install, --check, --dry-run, --events,
--list-systems, --system, --region, --emulator, --target,
--installed-cores, --manifest and --from-pack apply it. --core,
--list-cores, --list-emulators, --list-platforms, --list-targets,
--standalone-copies, --restore and --tui refuse it, and the interface
does not open by itself with it. On Batocera it gives 353 files of
14531, the list batocera-systems checks.

Event decision: no schema bump. A `declared` key on `file` would add a
key to an existing kind, which schema 1 forbids. A new kind, `declared`,
follows `check` with the same counts over the platform's own files and
their dests in file-event order. Readers already skip a kind they do
not know (rbevents.py and the interface reader were checked), so
EVENT_SCHEMA stays 1.

An unknown --system is answered with the systems the list files its
key file under: --platform retroarch --system snk-neogeo now suggests
arcade, which holds neogeo.zip, instead of snk-neogeo-cd. neogeo, mvs
and aes follow the same route. Never taken, only suggested. A system
title the manifest carries (SNK Neo Geo CD) is taken for its id when
no other system shares it and no platform name reads the same.

schemas/install-events.schema.json is written from EVENT_KEYS by
scripts/generate_event_schema.py. validate_schemas.py reports a stale
file, and tests compare it with EVENT_KEYS and validate a real run.
Bootstrap pins recomputed.
2026-10-23 19:07:18 +02:00
Abdessamad Derraz b657cff327 feat: export md5 lists the checker reads 2026-10-23 08:27:27 +02:00
Abdessamad Derraz 5b083c45b5 feat: export md5 lists the pinned checker reads
The script exporters (Batocera, ROCKNIX, KNULLI, AmberELEC, EmuELEC)
read the checker of the pinned original: when checkBios hands an
item's md5 to a function that tests it with isinstance(..., list),
as md5Matches does since batocera.linux 6b08b873, an entry whose
emulator code accepts several dumps is written with all of them, and
an addition that needed several becomes writable. Items with altmd5
keep a string, since checkBios wraps them in a list of its own.

No cached original reads lists today, so every export is unchanged.
The report records a list as a list, counts the lists written, and
records a list the truth extends without contradicting it.
2026-10-23 08:06:58 +02:00
Abdessamad Derraz 08f174bf32 docs: name the part marker in the release steps 2026-10-23 07:22:46 +02:00
Abdessamad Derraz 2927c49734 docs: split project reports, pin checker links
Reading this project's own reports gets its own page. Checker code is linked at the revision each platform list was read from. Installer moves to Your setup, legal to Start here, build internals to Contribute, and Reference pages carry a group.
2026-10-23 02:42:19 +02:00
Abdessamad Derraz 123d698e43 feat: build the site with Astro Starlight
generate_site.py writes the Starlight project in website/ and nothing
else: mkdocs.yml, the MkDocs hooks, theme overrides and stylesheet go,
and the wiki is written in Starlight's syntax. Both workflows set up
Node with the official actions/setup-node pinned by commit, run npm ci
from the committed lockfile, build with the strict link validator and
check the result with validate_site.py.

The search weighs a page by what it answers: Start here pages first,
a system page's cards and file names before a guide citing them, deep
guides, launchers, aliases and frozen copies after. Status badges,
hash blocks and the export change lists leave the index. The ranking
sits at the centre of a range where the 13 queries of search_gate.py
all answer, in the search box itself.

839 pages, every URL and 45,611 ids match the MkDocs build. The tests
read the Starlight content and syntax.
2026-10-22 18:47:22 +02:00
Abdessamad Derraz f18cfa1f5c chore: merge the archive recipe follow-ups 2026-10-22 15:10:14 +02:00
Abdessamad Derraz 06e37b36f7 feat: collect the tyrian files emuelec declares
EmuELEC declares 19 files of the Tyrian 2.1 distribution (setup.ini,
netterm.int, the net*.pcx screens...) that bios/ did not hold. They
resolved only at the data directory step, against data/opentyrian, the
cache refresh_data_dirs.py fills from the opentyrian-data.zip asset. verify
gave 544/572 in a checkout holding that cache and 525/572 in one without
it, for the same commit.

The 19 files are taken from the freeware tyrian21.zip (camanis.net, sha1
29827de9), each equal to the cached copy. netset.pcx and tyrset.pcx are
the same bytes under two names, so the folder joins NODEDUP_DIRS. A test
resolves every platform entry with the data directory caches present and
fails on any that only a cache answers.
2026-10-22 15:08:15 +02:00