Commit Graph
127 Commits
Author SHA1 Message Date
Abdessamad Derraz 448989665a chore: merge main into the findings review 2026-10-28 09:28:29 +01:00
Abdessamad Derraz 48f5dbc813 feat: close campaign findings by code review
campaign.py review applies or rejects an open finding with the reason read in the code. Applied fields go through the profile editor with the finding's records as evidence. The rule table reads on_mismatch: warn and counts a signature as a content check.
2026-10-28 08:16:27 +01:00
Abdessamad Derraz 5c8da8a424 feat: name the BIOS each run used
Record version 5: every repetition and variant carries bios_used, staged:<dest> when a load line of the core names the file, builtin when a fallback line says it ran its own BIOS, unknown otherwise. An opened file is no proof. The oracle gains fallback_patterns and load_patterns per core, and a record judged again carries judged_at and judged_by.
2026-10-28 05:18:38 +01:00
Abdessamad Derraz d7e17491f0 chore: merge the per-run probe lock 2026-10-27 22:53:16 +01:00
Abdessamad Derraz dd64434122 chore: merge main 2026-10-27 22:05:47 +01:00
Abdessamad Derraz cd92fe5b4d feat: let validation carry campaign evidence
The emulator schema allows evidence under validation, and a test holds the settled fields of the rule table to the schema's evidence keys.
2026-10-27 21:27:16 +01:00
Abdessamad Derraz 0d510eb4bd chore: merge core versions read from code 2026-10-27 19:42:04 +01:00
Abdessamad Derraz 5184b55b82 chore: merge the oracle pages 2026-10-27 19:24:19 +01:00
Abdessamad Derraz 91b7bd94d0 feat: refuse placeholder versions in the schema
core_version is a string the code states, null or keyed by build mode,
and the schema refuses Git, SVN, unknown, master, main, latest, dev and
HEAD, alone or annotated. version_basis is defined per kind with the
fields each one needs, and kind none requires a null version.

tests/test_version_basis.py reads a recorded fixture of forge answers
offline and fails on drift, holds every profile to a basis (tag versions
in describe form, aliases equal to their profile, none as null), checks
that a pin advance reads the version again or is refused, and that the
schema refuses placeholders.
2026-10-27 19:23:17 +01:00
Abdessamad Derraz c8eb3a79e3 feat: say when the system runs another version
The installer reads the version the system states in its own file,
without running anything: batocera.version, recalbox.version (a release
only), RetroBat's system/version.info, VERSION_ID in os-release on Lakka
and EmuELEC. When it differs from the version the list was built for, a
line says so before the install and again with the result, and the new
version event carries it for the screen. An unknown version says
nothing, and the install never stops on it.
2026-10-27 19:10:44 +01:00
Abdessamad Derraz 25c356669b fix: record each platform's own list version
install/<platform>.json wrote the constant "1.0" as the version of the
list it was built from. The manifest now carries the version the platform
YAML records, and generate_pack refuses a YAML without one.

inherits: no longer passes a parent's version to a child: Lakka was
published as RetroArch v1.22.2 in its manifest, pack name and site page.
lakka.yml records v6.1, the Lakka-LibreELEC tag its target list is read
at. The MiSTer scraper refuses a BiosDB without a timestamp instead of
writing an empty version.

A pack's manifest.json records list_version, so an install from the pack
alone knows it too. Manifests regenerated twice and compared: only
version and generated changed.
2026-10-27 19:10:39 +01:00
Abdessamad Derraz 70c3bbd853 feat: settle a file only when the run uses it
hle_fallback needs the file to change the run: Beetle PSX hashes every file of its folder, so opened is not used. PCSX-ReARMed hides the BIOS logo by default: its BIOS load line stands for the bios stage. A check runs again only when the oracle entries of its content and core change, and the rules run on every pass from the kept records.
2026-10-27 17:58:48 +01:00
Abdessamad Derraz f0320a2bd9 feat: test packs the way a user installs them
pack_probe.py lays a platform's BIOS folder out from its pack, or with install.py --dest, and starts each test game through core_probe with that folder, mapping every file the core opened to the pack entry that put it there.
2026-10-27 15:37:40 +01:00
Abdessamad Derraz 0fa4552dae fix: refuse only the axis a list lacks
The MinUI family's lists record cores and regions and no system: every
narrowing was refused with a remedy that fetched the same list. Each
option is now refused by the axis missing, with what works, and a pack
names its platform's list only when that list records the axis. Line
mode names what c chooses and skips a question with one answer, and
Options leaves out parallel downloads when nothing is downloaded.
2026-10-27 15:37:24 +01:00
Abdessamad Derraz c43872cdb6 feat: accept the pack and standalone probes
frontend.script names core_probe.py, pack_probe.py or standalone_probe.py.
2026-10-27 15:30:52 +01:00
Abdessamad Derraz aa6ee6db1c feat: add the boot verdict to the record schema
Record version 4 carries verdict and verdict_evidence on every repetition and variant, verdicts on a check result, and inputs.oracle. Version 3 records stay valid.
2026-10-27 15:25:30 +01:00
Abdessamad Derraz 9c8dbcd99a chore: merge the automatic test campaign 2026-10-27 14:05:18 +01:00
Abdessamad Derraz 31c574aab0 fix: line mode and screen share one decision
Answers code_audit_17 and tui_audit_10: the folder found after o, SHARE
drive roots, one same-choice command, a counted confirm step in line
mode, counts from the narrowed list, read-only advice without sudo,
systems recorded in packs, Ctrl-Z stopping the whole job, and the
wording, layout and docs items.
2026-10-27 12:24:26 +01:00
Abdessamad Derraz fd449fdae3 refactor: refuse unsandboxed campaign runs
Without a working bubblewrap the pass is refused and nothing third-party runs, probes require the sandbox and builds are refused, unless --allow-unsandboxed is given. A job report holding an alert raises an autorun alert.
2026-10-27 11:53:35 +01:00
Abdessamad Derraz 95ca227bce refactor: run cores and builds in a sandbox
core_probe starts its worker with only the --env values, the worker clears its own environment before loading the core, and a working bubblewrap runs it with a read-only tree, no network and the user's credentials masked. The record names the isolation used. A timeout kills the whole process group. campaign.py starts core_probe, git, make and romfetch with the allowlist, make inside the same sandbox. --record refuses a folder that is not a checkout root.
2026-10-27 11:48:22 +01:00
Abdessamad Derraz ee7319d838 feat: add the real-condition campaign runner
campaign.py takes buildbot nightlies (or a build of the pinned commit), the test content of tests/campaign/content.yml through romfetch, and runs core_probe for the files, boot, path, alias, variants and bench checks. A fixed rule table settles hle_fallback, required, path and aliases with the record ids as evidence, the rest goes to measurements/findings.jsonl. Resumable state, load, disk and budget guards, retries, content reference counting with an LRU cap, and an autorun job after the nightly.
2026-10-27 11:38:01 +01:00
Abdessamad Derraz da68f19411 refactor: keep one full record per line
Each line of measurements/<emulator>.jsonl now holds the whole run:
every repetition, core logs, per-frame hashes and raw timings. The gzip
data files, the data schema and the split and join code are gone, and
core_probe --record appends one line. record_version moves to 3.

The seven records are rewritten once: each new line equals its old line
joined with its data file, and rebuilds the first-layout file its id
hashes.
2026-10-27 09:57:39 +01:00
Abdessamad Derraz 571e50f2a8 refactor: read measurements from the local tree
The site compiles the index and the SQLite files from measurements/
instead of a pinned clone. The pin, the sync script, the CI clone steps,
the freshness area and its refresh handler are gone. core_probe --record
writes into measurements/ of the checkout by default, and a test fails
when a line or data file HEAD holds changed or vanished.
2026-10-27 09:11:12 +01:00
Abdessamad Derraz 6109d7abd5 feat: split probe records into line and sidecar
A record is now one line of records/<emulator>.jsonl in the measurements repository, under 20 KB, and a gzip sidecar holding every repetition, the core logs and the raw timings, its sha256 in the line. record_version 2, core-probe-data.schema.json for the sidecar. core_probe --record appends to a measurements clone and refuses a directory inside this repository's tracked tree.

The seven records of issue 85 moved to Abdess/retrobios-measurements: each line plus its sidecar rebuilds the first file, and keeps its first id.
2026-10-27 08:30:07 +01:00
Abdessamad Derraz 00b034612f feat: add headless libretro core probe
core_probe.py loads a libretro core with no window, runs it for a
number of frames and reports the files it opened (strace), its log
lines, frame, audio and save-state hashes and its speed. bench repeats
a run after warm-up and gives median, mean, deviation and range; check
swaps one system file between runs and compares one observable.

--record writes a JSON record with the core build, frontend commit,
host and every input by hash, validated against
core-probe-record.schema.json. A busy host is refused, and a
repetition that disagrees with the first is flagged.
2026-10-27 05:18:10 +01:00
Abdessamad Derraz 57811270e3 fix: card setups, pager lines, devices, data size
A system that runs from its SD card (Onion, spruceOS, muOS, KNULLI,
MinUI...) named with --platform on a computer no longer takes that
computer's RetroArch folder: one mounted card of that system is used,
at a terminal the card root is asked, without questions the run stops
on the --dest example. The interface opens the card screen with the
mounted cards first.

The result pager scrolls and counts lines, so its last line is
reachable at every size. The setup filter knows the devices each
setup runs on, from the guides and the firmware pages. A new event
kind, data, gives the bytes of the data trees, and the plan shows
them.
2026-10-26 22:01:02 +01:00
Abdessamad Derraz f3c816eea5 feat: put the retrobios app on a handheld card
--add-app, the result screen and a [y/N] question put the launcher of
muOS, KNULLI or ROCKNIX on the card the BIOS folder is on. The files come
from install/launchers/index.json at the same ref, each checked by size
and SHA-256 before the first is written, and the app event reports it.
2026-10-26 21:01:45 +01:00
Abdessamad Derraz d2ba7e18ba fix: search past a folder this user cannot open
One closed folder under HOME (~/.config/EmuDeck at 000) made the whole
detection return nothing: a RetroArch whose retroarch.cfg was readable
was lost and the run fell back to a folder it does not read. Every
detection probe now goes through _exists, _is_dir or _is_file, which
count a refused place as absent, name the closed folder once, and let
the other probes go on. The decorator stays as the last net.

The refusal is an unsearched event before detect. The interface keeps
the line above the list of setups. At HOME or a drive root A goes into
a visible subfolder, never a card or drive row nor .cache, and with
none it stays on Use this folder and says where it is. A default folder
taken because the platform was not found is no longer called detected.
2026-10-24 02:23:57 +02:00
Abdessamad Derraz 801b8bbda0 fix: one pack answer and fuller reference data
The API reads release_pack from the registry entry, not the inherited config, and the picker gets release-advertised.json. The lookup counts emulators that read a file by name, every record names what was declared, the Data page lists all schemas, the target lists and nested manifest fields, and systems.json items carry their api address.
2026-10-24 00:51:13 +02:00
Abdessamad Derraz 329b9e1bda docs: describe every install manifest field
Nested fields and each omission reason code, read from the code that writes them.
2026-10-24 00:51:13 +02:00
Abdessamad Derraz 3db8ddae2b feat: name the save folder and the pack on screen
The plan and the result now say how many files go to the save folder and
where, from a new saves event. With --from-pack they say the files come
from the pack, not a download. A stop asked for (Start+Select, SIGHUP)
ends on the plain outcome, without the log's last lines.
2026-10-23 22:34:24 +02:00
Abdessamad Derraz 6f958382b7 chore: merge the engine options round 2026-10-23 19:25:34 +02:00
Abdessamad Derraz 8f50015a0c feat: round 4 reference and data 2026-10-23 19:23:58 +02:00
Abdessamad Derraz b34a6058f7 chore: merge main into the engine round 2026-10-23 19:07:44 +02:00
Abdessamad Derraz 35e7300465 feat: add --declared-only and the events schema
--declared-only keeps the entries the platform's own list declares
(cores: null), each slot settled to the platform's copy with no core
alternative left behind. Install, --check, --dry-run, --events,
--list-systems, --system, --region, --emulator, --target,
--installed-cores, --manifest and --from-pack apply it. --core,
--list-cores, --list-emulators, --list-platforms, --list-targets,
--standalone-copies, --restore and --tui refuse it, and the interface
does not open by itself with it. On Batocera it gives 353 files of
14531, the list batocera-systems checks.

Event decision: no schema bump. A `declared` key on `file` would add a
key to an existing kind, which schema 1 forbids. A new kind, `declared`,
follows `check` with the same counts over the platform's own files and
their dests in file-event order. Readers already skip a kind they do
not know (rbevents.py and the interface reader were checked), so
EVENT_SCHEMA stays 1.

An unknown --system is answered with the systems the list files its
key file under: --platform retroarch --system snk-neogeo now suggests
arcade, which holds neogeo.zip, instead of snk-neogeo-cd. neogeo, mvs
and aes follow the same route. Never taken, only suggested. A system
title the manifest carries (SNK Neo Geo CD) is taken for its id when
no other system shares it and no platform name reads the same.

schemas/install-events.schema.json is written from EVENT_KEYS by
scripts/generate_event_schema.py. validate_schemas.py reports a stale
file, and tests compare it with EVENT_KEYS and validate a real run.
Bootstrap pins recomputed.
2026-10-23 19:07:18 +02:00
Abdessamad Derraz 6d938f3d54 feat: reference pages answer every hash and field
A platform or emulator record whose value designates a collection
file carries its sha1 and is filed under the file's other hashes, so
a SHA1 paste finds the setups listing it by MD5. The data page shows
every manifest field, writes URL templates as code, states the EmuDeck
rule, the browser lookup and the empty lists. systems.json gives every
id holding files a page, Files to source splits at 100 rows with an
upload link per file, and MiSTer and BizHawk get no empty core table.
2026-10-23 18:52:13 +02:00
Abdessamad Derraz 00cd203b23 feat: add schema_version to release.json 2026-10-23 18:52:04 +02:00
Abdessamad Derraz 50b42823ca feat: add a per-os data_root to profiles
Apps such as AetherSX2 named their user data folder only in prose, so
the site could neither print it nor show only the OS tabs that apply.
data_root keys android, ios, windows, macos and linux, each with the
path the user finds and the source_ref that builds it.
2026-10-23 18:44:25 +02:00
Abdessamad Derraz 82bf236a5a feat: hold an upstream pin a port vendors
upstream_frozen says why upstream_commit stays: the port vendors that
upstream tree, synced to that release, or merged that commit. The view
is then read at its own pin, the way pin_frozen holds source_commit, and
a bump leaves it. Outside a GitHub fork network nothing else could tell
aps3e's vendored RPCS3 1f080485 from an upstream read on a date.
2026-10-23 17:30:20 +02:00
Abdessamad Derraz eeef4fc6fb feat: packs for the most downloaded setups 2026-10-23 17:18:00 +02:00
Abdessamad Derraz 32c648b67d feat: release packs for marked platforms only
release_pack: true in the registry is the list a release builds: RetroArch,
EmuDeck, RetroBat, Batocera, RomM and MiSTer, 83% of the downloads of
v2026.10.10. generate_pack --all, --verify-packs, the pipeline consistency
step, release_record and check_release read it. Manifests are still written
for every platform.
2026-10-23 17:06:46 +02:00
Abdessamad Derraz 4a1d875f43 feat: name systems in the install manifests
Each manifest carries system_titles, the name the site prints for every system its files list (Sony PlayStation, not sony-playstation or psx), so the installer can show it. Older installers ignore the field.
2026-10-23 15:55:46 +02:00
Abdessamad Derraz 94d18bd7e5 feat: name the system in each lookup record 2026-10-23 10:54:34 +02:00
Abdessamad Derraz e3a54d2423 feat: serve each schema at its own $id 2026-10-23 10:46:05 +02:00
Abdessamad Derraz 0b4504992b feat: read batocera md5 written as a list
batocera.linux 6b08b873 lets a biosFiles item declare its md5 as a
list of strings, and checkBios accepts a file matching any of them.
The scrapers join the values with commas, as for Recalbox, and record
md5_list so the export writes the list back as a list, byte for byte.
2026-10-23 07:39:25 +02:00
Abdessamad Derraz 8b26a8fcac feat: cite sources and files on reference pages
Platform pages link each reader's source line at its pinned revision, name the collection file that serves each entry, show launcher records with their lines, name systems with the id beside them and the list they were read from in words. The largest system lists move to pages of their own so a platform page stays under 500 KB.

Gaps: plain numbers with a bar, a badge only for missing required firmware, Files to source as a heading. Cross-reference: counts per platform, one page per platform, one row per upstream project. Exports: gaps listed once, status, category and the serving file in the CSVs and SQLite, the commit in each API envelope, the provenance Wanted list as CSV. Catalog dates in words on the provenance page.
2026-10-23 02:58:51 +02:00
Abdessamad Derraz 2d26385efe feat: place save-dir files in each system folder
Batocera, KNULLI, Recalbox, RetroBat and RetroDECK hand a libretro core <saves>/<system> at each launch. Each copy lands below every system that runs the core. EmuDeck keeps one unsorted saves/retroarch/saves. The unsupported platforms record why.
2026-10-22 21:16:59 +02:00
Abdessamad Derraz 9a1bf4baa6 feat: record the systems each libretro core runs 2026-10-22 21:16:47 +02:00
Abdessamad Derraz 1cd6c9ee4d feat: place save-dir files in packs and lists
load_from: save_dir entries of a libretro build were left out of every
pack. They now go below the platform's save folder (registry
save_directory: RetroArch saves/, Lakka savefiles/, by core), the
manifest marks them root: save, and verify replays the same placement.
Platforms keeping saves per system or per game list them as omitted.
2026-10-22 19:58:26 +02:00
Abdessamad Derraz 33041b6bd1 feat: find a file by any hash, out of the search
Hashes, link lists, install commands, member tables and long reference
tables leave the site search: the index drops from 21.0 MB (4.88 MB
gzip) to 6.8 MB (1.33 MB gzip). Names and prose stay searchable.

files/?hash= takes a SHA1, MD5, CRC32 or SHA256 and reads one shard of
api/v1/lookup/ (about 22 KB gzip, was 675 KB) listing the collection
card, the platforms that declare the value and the profiles that check
it. files/?sha1= still opens the card, and a sha1- anchor missing from
a page is looked up there, so a card can move between pages.

Excluded blocks are section elements: the search plugin compares
elements by tag, so a div ended at the first nested div.
2026-10-22 12:46:40 +02:00