Commit Graph
144 Commits
Author SHA1 Message Date
Abdessamad Derraz 0605079cda fix: skip a schema check without jsonschema 2026-11-10 06:58:07 +01:00
Abdessamad Derraz ad424f796b chore: merge one-of sets 2026-11-09 18:33:21 +01:00
Abdessamad Derraz 58735286a7 feat: let an archive join a set
Kronos reads kronos/stvbios.zip, else stvbios.zip at the root of the
system directory, and a ST-V game stops without either
(libretro.c:1608-1617, 1676-1681 at the pin). The archive under
archive_prefix and the root copy are one need, but only loose entries
could be members, so the required-only pack carried both.

An archive's own entry now joins a set like a loose file. verify
--emulator and the emulator pack settle archive units with the loose
entries, the required-only pick covers them, find_undeclared_files
and cross_reference record them, and the unit under archive_prefix
ranks before the same name at the root. Its members still settle the
archive by their contents. Kronos' stvbios.zip is grouped stv-bios.

No other profile reads one archive under two paths: odin and
tgpulse_next_m1 place their own entries at the prefix path, and
flycast's naomi.zip has one path per build.
2026-11-09 18:26:05 +01:00
Abdessamad Derraz 315dab82b7 feat: give each set its requirement and one pick
Three changes to the search-list model, one commit because the readers
and the profiles must move together for the schema to hold.

Set requirement in a groups map. pom2 falls back on apple2.rom for the
][ and the ][+, and on apple2e.rom for the //e it boots and for the
unenhanced //e: one required on the file could not say both. A profile
now declares groups: {name: {required}}, a member writes no required
of its own (schema and validate_schemas refuse one) and load_emulator_
profiles derives it, true when one of its sets is. group takes a name,
a list, or a map from set to rank where the rank differs per set. pom2
is grouped from SystemProfile.cpp and MainWindow.cpp:246-296 at its pin.

One member per required set under --required-only. The pick is the
first the code reads among the held members the region filter leaves,
by rank then declaration order; a set the platform declares a member
of is carried by that declaration. The emulator pack, verify
--emulator, the platform pack, its manifest and verify --platform read
alternatives.required_only_spares or core_file_spares.

variant_group is now selection_slot: files a core chooses between by
a rule other than first found (disc territory for PicoDrive and
Pizza Boy SC, the o2em_bios option, a WebAssembly probe in SWF
Player). Where the code takes any one, the field became a group:
ARMSX1 (ArmSxBackend.java:149-157, name order), ePSXe (load_bios
0x433540-0x433621 of the 2.0.5 Linux x64 binary, one configured path,
HLE when BiosHLE is set), FPse and FPse-NG, DXX-Rebirth's HOGs
(inferno.cpp:472-512). odin and zesarux carried it with no effect.
slot.py compares ranks only between candidates that share a set.
2026-11-09 18:19:32 +01:00
Abdessamad Derraz 2c76cc7df9 feat: read a search list as one need
A core that loads the first of several names (yabause: saturn_bios.bin,
sega_101.bin, mpr-17933.bin) needs one of them. Profiles wrote this as
required on the first name and optional on the others, so verify and
the gap report called saturn_bios.bin a missing required file while
sega_101.bin was held, and a set held only through a name marked
optional read as a need met by an optional file.

Model: an explicit group on the file entry, mirroring the platform
field. priority was not reused: it is a rank, read by slot.py, region.py
and truth.py inside a (system, region) slot, and that slot is not the
set. px68k-sdl ranks its IPL list and its font list each from 1 in one
system, pom2 ranks one list per machine model, DuckStation ranks 105
images across every region and takes any of them. A rank cannot say
where one list ends. priority stays the order inside a group.

Readers, through alternatives.settle_sets:
- verify --emulator and the emulator pack: a held member fills the
  set, the others leave the report, a set nobody holds is one gap
  under its first member naming the others.
- find_undeclared_files (verify platform, core complement of packs and
  manifests): same rule, a platform declaration or launcher counts as
  answering the set.
- cross_reference: excused members are set aside as other versions.
- slot.py: ranks compare members of one set only.
- archive members: a required set inside an archive is met by one.
- truth: a member gives no requirement of its own, one_of and
  one_of_required carry the set, and the exporters and the diff leave
  such a line's required as the platform wrote it.
- validate_schemas: members of a group share required.
2026-11-09 17:39:46 +01:00
Abdessamad Derraz b270a24d7b feat: take a platform's own libretro builds
A list naming retroarch now takes the libretro profiles whose built_by names the platform: ROCKNIX ships yabadabadoo with its RetroArch and named it nowhere, so its files never reached the ROCKNIX pack. Only change across the registry: rocknix + yabadabadoo. Also quotes the installer's current unproven-drive message.
2026-11-09 17:18:44 +01:00
Abdessamad Derraz 19264db538 chore: merge issue 87 and platform refresh 2026-11-09 07:24:48 +01:00
Abdessamad Derraz 77a8f171ba feat: read retrobat bios groups
emulatorLauncher 228731e0 lists ten PS2 BIOS in one group: one held file
is enough and Program.cs reports the others OPTIONAL. The scraper keeps
group and optional, verify and the pack check apply the same rule.
2026-11-09 07:22:06 +01:00
Abdessamad Derraz 84cffe3f8d chore: merge origin/main 2026-11-06 16:42:16 +01:00
Abdessamad Derraz 057fd2d0f6 fix: isolate every test module however it runs
tests/__init__.py keeps a run offline, its scratch under tmp/ and its
installer cache its own, but only discover -s tests -t . imported it.
discover tests ran with RETROBIOS_OFFLINE unset, scratch on the /tmp
tmpfs and the user's real installer cache, the setting in which e2e
fixtures had written into the shared .cache/large. Every test module now
imports the package first, and a test fails on a module that does not.
test_verify_listing.py took its line with the previous commit.
2026-11-06 15:14:05 +01:00
Abdessamad Derraz 52b7ee0f46 fix: succeed when an emulator needs no file
--emulator handy --required-only removed the empty ZIP, printed the
path it had just removed and exited 1, while verify reported 0/0 OK and
exited 0. The builder now says no file is needed and succeeds.
2026-11-06 15:06:28 +01:00
Abdessamad Derraz 6bf62d5746 fix: refuse profile narrowings without cores
MiSTer and BizHawk declare cores: []. --region, --one-per-slot and
--no-game-data read the profiles of the cores a platform runs, so their
packs and manifests withdrew nothing under a name and README that said
they had. generate_pack and verify now refuse the flags for a named
platform and leave it out under --all.
2026-11-06 15:01:58 +01:00
Abdessamad Derraz 4d3f589ef4 refactor: drop the archived platform status
RetroPie was the only platform with status: archived, which kept it out
of verify --all, slots, the pipeline and list_platforms unless
--include-archived was typed. release_pack already decides which packs a
release publishes and the installer serves every platform, so the status
set nothing apart. The field, its schema entry and the flag are gone, and
RetroPie is covered by every full run like any other platform.
2026-11-06 11:52:34 +01:00
Abdessamad Derraz fb9b5592a5 feat: let --emulator take an alias name
--emulator mednafen_psx refused with 'use --emulator beetle_psx' although the profile names its target. The alias now selects that profile and says so on stderr, in verify, generate_pack and cross_reference alike. Launchers are still refused, and pipeline step 5 skips aliases since they compare under the profile they name.
2026-11-05 19:26:36 +01:00
Abdessamad Derraz cfc2515909 fix: check the zipped_file member before composite
A one-member archive whose member carries the zipped_file md5 under
another name has that md5 as its composite. The name step returned it as
md5_composite_exact, which slots and the site take as content proof, while
Batocera looks the member up by name and finds nothing.
2026-11-05 11:45:58 +01:00
Abdessamad Derraz 0d510eb4bd chore: merge core versions read from code 2026-10-27 19:42:04 +01:00
Abdessamad Derraz 25c356669b fix: record each platform's own list version
install/<platform>.json wrote the constant "1.0" as the version of the
list it was built from. The manifest now carries the version the platform
YAML records, and generate_pack refuses a YAML without one.

inherits: no longer passes a parent's version to a child: Lakka was
published as RetroArch v1.22.2 in its manifest, pack name and site page.
lakka.yml records v6.1, the Lakka-LibreELEC tag its target list is read
at. The MiSTer scraper refuses a BiosDB without a timestamp instead of
writing an empty version.

A pack's manifest.json records list_version, so an install from the pack
alone knows it too. Manifests regenerated twice and compared: only
version and generated changed.
2026-10-27 19:10:39 +01:00
Abdessamad Derraz 07c82c885f refactor: read the fbneo version from version.h
FBNeo bumps src/burn/version.h without tagging: the scraper wrote the
last tag, v1.0.0.02, while the source said 1.0.0.03. It now reads the
header the way the libretro core prints it. The merge no longer writes
core_version: the hashes come from finalburnneo/FBNeo's tip, which the
libretro/FBNeo pin does not name, so the version stays the one
version_basis reads at that pin. A MAME release profile is written with
its version_basis.
2026-10-27 18:48:16 +01:00
Abdessamad Derraz fa43053362 chore: pin packed zips to the collection bytes 2026-10-23 08:27:16 +02:00
Abdessamad Derraz a0b27d612d fix: align tests with the ten-value pack result
Two test modules still unpacked nine values from verify_pack_against_platform. The category evidence joined readers with semicolons, which the prose rule refuses on generated pages, and three comments used clause semicolons.
2026-10-22 06:23:15 +02:00
Abdessamad Derraz 8b952968d6 feat: add a --no-game-data pack narrowing
The install manifest marks each game data file "category": "game_data":
a platform entry only game data profile entries read, or a core file its
profile declares game data. Firmware carries no category. Every manifest
says "categorized": true so the installer can refuse an older list.

generate_pack.py --no-game-data cuts those files from the platform list
and the core profiles before the build, names each one with the cores that
read it as game data, and tags the pack _NoGameData through _narrowings.
verify.py and the pack check take the same cut, so --verify-packs and the
report describe the pack the builder wrote. Modes without a platform
(--emulator, --system alone, --from-md5) and --all-variants refuse it.
2026-10-22 06:19:05 +02:00
Abdessamad Derraz f66adfdcbc feat: categorize missing platform entries
A platform list can name a game's data files or a file no core it runs opens. Verify gives each missing entry a category read from the profiles of the platform's cores: firmware, game data or unread. Only firmware gaps carry a severity and fail --strict; the others are counted on their own line, and the pack check no longer counts them as pack errors. Missing firmware stays a conformance error.
2026-10-22 05:38:57 +02:00
Abdessamad Derraz b7b407698f fix: count files the collection lacks apart
The pack check failed a pack for a declared file nothing in the collection resolves to, though the builder cannot ship it and verify already reports it missing. Such destinations are now counted as not in the collection; a held file absent from the pack stays an error.
2026-10-22 04:28:43 +02:00
Abdessamad Derraz 055627f39c chore: merge round 2 fixes (tooling) 2026-10-21 10:31:04 +02:00
Abdessamad Derraz 4fec09a014 fix: name the selection mode in --from-md5 packs
BizHawk is both a platform and a profile display name: --from-md5 with --platform bizhawk and with --emulator bizhawk wrote one file in two layouts. The emulator mode now carries _Emulator in the name.
2026-10-21 09:56:45 +02:00
Abdessamad Derraz a3e625cd6a fix: refuse a default value typed out as a flag
The refusal tables compared each flag with its default value, so --source full and --group-by system passed beside the modes that refuse every other value. Both now default to None and take their effective value after the checks, in generate_pack and pipeline.
2026-10-21 09:55:51 +02:00
Abdessamad Derraz d8e9ad23d3 docs: describe short md5 handling as the code does
No platform YAML holds an md5 shorter than 32 characters: the Batocera scraper completes one when a single collection md5 starts with it. The pages said some entries carry 29 characters and that scrapers store them as-is.
2026-10-21 09:54:57 +02:00
Abdessamad Derraz f957b5f893 fix: open no System.dat section libretro lacks
libretro titles each section with the system as it spells it,
"Commodore - Amiga". An addition for a system it has no section for
opened one under our slug: 21 sections came out as commodore-c64,
quake, laseractive or xrick, which no libretro DAT carries. An addition
now lands only in a section the original holds and is otherwise counted
as left out, so the export keeps the original's 66 sections. A platform
entry is always written, under its own section.
2026-10-20 17:21:16 +02:00
Abdessamad Derraz 71b59eb2a3 docs: rewrite half-dash asides in comments
Dropping the double-dash asides left 72 comments and docstrings with
half a dash glued to the next word ("shape -a size or a size range -is
all"). They now read as a colon, a comma or parentheses, and a prose
style test names any that comes back.
2026-10-20 09:17:10 +02:00
Abdessamad Derraz 16f6ef8cbf feat: record the commit a branch scrape read
Six platforms are read from a branch: System.dat, batocera-systems.json, checkBIOS.sh, the MiSTer BiosDB, the RetroPie scriptmodules and the RetroDECK component manifests. Their YAML named the branch, so the native export patched whatever the branch had become since the scrape. Each scraper now resolves the branch head once, reads every file at that commit and writes it into source. The export follows a recorded tarball and a recorded tree as it already followed a raw file URL.
2026-10-19 22:50:45 +02:00
Abdessamad Derraz 4231902a6c docs: write prose without dashes or semicolons
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.

validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
2026-10-19 06:47:31 +02:00
Abdessamad Derraz d6d03617db fix: exit 1 when a hash lookup finds nothing
The documented example looked up a test fixture hash, printed NOT FOUND
and exited 0. The docs now use a hash the database holds and say which
line forms a hash file accepts.
2026-10-18 07:19:01 +02:00
Abdessamad Derraz a40a68c1f7 fix: group a system once whatever its spelling
The platform's files and the core extras for the Master System sat
in two region groups, sega-master-system and sega-mastersystem,
reduced apart. A US pack kept the declared Japanese BIOS. A core
extra's system group now takes the platform's spelling, in the pack,
the report and the installer's manifest fields alike.
2026-10-18 03:51:57 +02:00
Abdessamad Derraz 78902ccc6a fix: name the flag one-per-slot conflicts with
--platform --system --one-per-slot was refused with a message
asking for --platform, which the command carried. The refusal now
names --system or --emulator, and every refusal in the mode matrix
is checked never to ask for a flag it was given.
2026-10-18 02:52:02 +02:00
Abdessamad Derraz d7f382339c chore: name the platform core in export fixtures 2026-10-17 19:38:08 +02:00
Abdessamad Derraz af4b43d64c fix: tell a pack note by its name, not its path 2026-10-17 10:38:18 +02:00
Abdessamad Derraz 921de6a1a6 fix: refuse a truth pack a platform cannot fill 2026-10-17 10:31:02 +02:00
Abdessamad Derraz e09a37d21e feat: keep platform-built cores out of all_libretro
Recalbox compiles libretro/mame at 0.258 and 0.278 itself; typed
libretro, both profiles were resolved for every all_libretro platform
although the buildbot RetroArch installs from has neither. A profile
now names the platforms that build it (built_by), and is reached only
when a platform lists it by name.
2026-10-17 04:44:25 +02:00
Abdessamad Derraz 42950a553c feat: refresh data trees once per online pipeline
Step 2 refreshed the data directories, then the pack build and the
manifest build refreshed them again, so a tree tracking a moving
branch could differ between the pack and the manifest. generate_pack
gains --no-refresh-data, which packs the cached trees and still
fetches release assets; the online pipeline passes it to both builds.
2026-10-17 01:59:20 +02:00
Abdessamad Derraz 9966029cf7 fix: keep the ci test job off the network 2026-10-16 10:18:53 +02:00
Abdessamad Derraz fd8db894d8 fix: test extras grouping through the real chain 2026-10-10 12:16:11 +02:00
Abdessamad Derraz 6a2f9bf2e7 chore: read the three copy counts 2026-10-09 23:18:36 +02:00
Abdessamad Derraz 82bdcfa1b1 fix: read the clone map beside the scanned tree 2026-10-09 22:45:57 +02:00
Abdessamad Derraz 0c39fc479d fix: drop the --include-extras flag nothing read 2026-10-06 12:27:21 +02:00
Abdessamad Derraz c39588807f fix: refuse --include-archived without --all 2026-10-06 12:26:47 +02:00
Abdessamad Derraz 3183ad982c fix: read gaps by normalized system, sha256, file names 2026-10-06 09:32:32 +02:00
Abdessamad Derraz cdecb096a1 fix: refuse --refresh-data where nothing refreshes 2026-10-06 09:24:24 +02:00
Abdessamad Derraz 7b62f7f702 fix: refuse every flag a quick mode does not read 2026-10-06 04:39:08 +02:00
Abdessamad Derraz dbc8d63318 fix: one agnostic substitute rule for pack and verify 2026-10-06 04:19:40 +02:00
Abdessamad Derraz d6afb09663 refactor: one upstream name index for core lookups 2026-10-06 01:47:16 +02:00