Commit Graph
1600 Commits
Author SHA1 Message Date
Abdessamad Derraz 82a3e44afd fix: settle a slot in any case where case folds
declaration_settles tested the exact-case name before the slot, so on a
filesystem that folds case a declaration spelled otherwise never settled
the core's file: RetroBat's 32X_G_BIOS.BIN left BlastEm's 32X_G_BIOS.bin
undeclared, and verify counted ten such entries as core files. A slot a
case-insensitive platform fills now settles the entry whatever the
spelling (case_folded_slots), the slot arbitration judging its bytes as
for the exact name. Lakka and RetroPie, which keep both names, are
unchanged. Measured: 10 entries settle on RetroBat, 21 on RetroArch.

generate_site.py builds its gap report through cross_reference() without
the folded slots; it needs folded_declared_slots() passed in the same way
as the command line now does.
2026-10-21 05:38:11 +02:00
Abdessamad Derraz edda080c59 fix: read muOS RetroArch from its own config
muOS runs RetroArch on a config copied from its default, which reads
/run/muos/storage/bios, bound to MUOS/bios on the second card when that
card holds one, and loads cores from /opt/muos/share/core (MustardOS/internal
f41f4a89, script/control/retroarch.sh:8-20, script/device/bind.sh:116-150,
retroarch.default.cfg:2806,3151). The installer wrote to /mnt/mmc/MUOS/bios
and --installed-cores found no core. The config now decides both.
2026-10-21 05:35:45 +02:00
Abdessamad Derraz 184c822e6e fix: name the file each layer reads in slots.py
Each content-checked line ended with a verdict on the platform's list,
printed at pipeline step 3b. The line now says which emulators read the
other file, and the closing count says the rest follow the platform's
list. The module source carries no such verdict, a test reads it.
2026-10-21 05:33:50 +02:00
Abdessamad Derraz bdcc8f0fb6 fix: lower the pipeline floor to Python 3.9
The 3.12 guard rested on generate_readme f-strings that have since been
rewritten. Every script compiles under CPython 3.9.25, 3.10 and 3.11, and
only generate_pack's parenthesized context managers fail under 3.8. Under
3.9 the offline pipeline (--skip-packs) completed with no change to the
committed artifacts, the 15 manifests matched 3.12 byte for byte, and the
RomM pack was identical. CI now compiles every script under a real 3.9.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz ceb8b752e6 fix: count bundled gaps apart in each heading
cross_reference.py --emulator abuse printed "Gaps: 3 missing" above three
[BUNDLED] files, then "0 missing (need to be sourced)" in the total. The
heading and the total now read one count: bundled files appear as
"N bundled", never among the missing.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz c05d29d941 fix: a held name must carry the declared hash
entry_source took a name or path hit before reading the hashes the entry
declares, so the gap report and the site counted as held a file whose
only homonym has other bytes: zquestclassic's ending.mid and gameover.mid
matched Hydra Castle Labyrinth's. A name or path hit now counts only
when its record carries one of the declared hashes, and the declared
content anywhere in bios/ comes before a data/ copy. Measured over every
profile entry: 254 change, 248 bundled ones (report and site only),
three unsourceable SC-55 revisions now read missing, three data hits
read bios by hash.
2026-10-21 05:32:19 +02:00
Abdessamad Derraz 15cb52d551 fix: find EmuDeck Windows where its scripts look
functions/all.ps1:4-14 (emudeck-we 037460ef) reads the APPDATA
settings.ps1, else the one under the profile, and takes C:\Emulation when
neither names a folder. Only the first was read and a forced platform fell
back to %USERPROFILE%\Emulation\bios. A network install names its share
through $VolName (JSONtoPS1.ps1:166-172), and a UTF-16 settings file is
decoded.
2026-10-21 05:31:35 +02:00
Abdessamad Derraz 0dab8f4706 fix: credit file readers through the data trees
FileConsumers resolved profile entries without the data directory
registry the builder uses. Dolphin's hashless codehandler.bin then took
Dolphin-MMJR's copy by name, and the site listed Dolphin and Triforce as
its readers while every pack serves data/dolphin-sys. blueMSX's RS232.ROM
and MSX2P.rom were credited to bios/ copies the same way. The registry
is now read by default, as the builder reads it.

With it, bundled_only.py --strict (pipeline step 4d) names
bios/Other/dolphin-mmjr/codehandler.bin: only dolphin-mmjr's bundled
entry, which already carries its sha1 and size, reaches that copy.
2026-10-21 05:29:42 +02:00
Abdessamad Derraz d2141ac0bb fix: give the MAME clone test a parsed set
The fixture parsed no BIOS root set, which the scraper now refuses.
2026-10-21 05:28:38 +02:00
Abdessamad Derraz cdb2cfa6f0 feat: find RetroBat through the folder it records
RetroBat.exe writes its folder to HKCU\SOFTWARE\RetroBat
LatestKnownInstallPath on every start (retrobat-executable 702da66a,
Program.cs:57 and 99, RegistryTools.cs:15-50). Only C:\RetroBat and the
profile folder were looked at, so an install on D: was missed and a forced
--platform retrobat wrote into C:\RetroBat\bios. The value is read
through winreg, or reg.exe under WSL, before the fixed folders.
2026-10-21 05:28:35 +02:00
Abdessamad Derraz 5e95574143 fix: the strongest proof takes a shared core path
A destination no platform declares went to the first claimant in profile
order whatever the proof: fbalpha2012 asks for hiscore.dat by name and
took the root of six packs over cores that pin it by sha1. Claims on a
contested path are now ranked by what the collection holds for each (a
held declared hash, then a path or every required archive member, then a
name, then a contradicted hash), and profile order only breaks ties.
Measured: 14 slots change file across the platforms, among them
hiscore.dat, kick13.rom on RetroPie, cgb_boot.bin and dmg_boot.bin on
EmuDeck, and ROCKNIX neogeo.zip and neocdz.zip.
2026-10-21 05:28:10 +02:00
Abdessamad Derraz 8f86842117 feat: keep romsets holding a missing catalog dump
The recipe snapshot kept only the archives a profile or platform names,
so provenance_report --missing named no set for dumps a public romset
carries (fm7740sx holds the FM77AV40SX DICROM, EXTSUB and KANJI2 ROMs).
The importer now also keeps the sets holding a catalog entry the
collection lacks. Reimported from mame0289 and the cached FBNeo DATs:
MAME 1726 to 2148 recipes, FBNeo 104 to 111, none dropped.
2026-10-21 05:27:42 +02:00
Abdessamad Derraz 309231afc2 docs: say what --skip-packs still runs
The help said "Only regenerate DB and verify" while steps 1b, 4d, 7 and
8 still ran and rewrote derivable.json, README.md and docs/. It now names
the steps it skips and points to --skip-docs.
2026-10-21 05:27:07 +02:00
Abdessamad Derraz 94a230a063 chore: frame the triage rows in a refusal test 2026-10-21 05:26:53 +02:00
Abdessamad Derraz 738577b5a4 fix: run verify and generate_pack from any dir
Started outside the repository root, both ended on a FileNotFoundError
traceback: the database and the data registry name files from the root.
They now move to the root after reading the paths the caller typed, which
stay relative to where they were typed.
2026-10-21 05:25:29 +02:00
Abdessamad Derraz 4f20b5daaf fix: name the network when every transfer fails
A run whose downloads were all cut ended on "Location that was being
written to" and "once that is resolved", pointing at the disk. A
failure caused by the connection (lookup, reset, timeout, TLS, a cut
transfer, HTTP 5xx or 429) is now marked as such, and the summary says to
check the connection and HTTPS_PROXY.
2026-10-21 05:25:19 +02:00
Abdessamad Derraz 61f0d033b5 chore: split a test docstring sentence 2026-10-21 05:25:10 +02:00
Abdessamad Derraz 76618af460 docs: shorten two script docstrings
_enrich_hashes repeated "ground truth" five times with capitalised
insistence for a three-step lookup, and the MAME ZIP rebuild listed
three guarantees of which two were the same.
2026-10-21 05:24:34 +02:00
Abdessamad Derraz 0457bd8f40 fix: resolve the pipeline output dir once
Every step runs from the repository root, but the lock and the purge of
stale packs read --output-dir from the caller's directory. Started from
scripts/, the run locked and purged scripts/dist/ and wrote to dist/,
leaving the stale packs there. The directory is now made absolute before
the lock, dist/ at the root by default.
2026-10-21 05:24:25 +02:00
Abdessamad Derraz 82d5ff77cf docs: sentence-case and split README paragraphs
Quick Install was the one Title Case heading left in the published
text. The Files sentence was glued to the paragraph above it, two
paragraphs before the table whose column it describes. It now names
the column and sits just above the table.
2026-10-21 05:24:25 +02:00
Abdessamad Derraz 8eda797562 fix: a hashless entry is a path in any mode
Whether the frontend compares content was decided per platform. EmuDeck
(md5) declares neocdz.zip with no hash and checkBIOS.sh never names it,
yet the proven claims of geolith and FBNeo were dropped and the pack
shipped a neocdz.zip without 000-lo.lo. checked_by_path() decides per
entry: no hash of the kind the frontend compares (any of md5, sha1,
crc32 under RomM's rule) and no zipped_file means a path. Measured: the
EmuDeck neocdz.zip slot and three RetroBat DS slots (dsi_bios7.bin,
dsi_bios9.bin, firmware.bin) now take the file the cores prove.
2026-10-21 05:24:19 +02:00
Abdessamad Derraz b464ed9202 fix: count prose that holds the pin as review work
The triage, --changed-only and --strict counted refs only. Prose that
--realign-prose would move or refuse blocks every recale and bump, and
blastem read 0 to review. Each report now carries that reading. The
triage also gets a header, a closing count, and one sampling notice per
run instead of one per profile.
2026-10-21 05:24:17 +02:00
Abdessamad Derraz 905316061e fix: read retropie and emudeck targets at the pin
The RetroPie target scraper read master and the EmuDeck one the default
branch, while both platform files are pinned on a commit: a module flag
changed in between filtered packages the pinned list does not describe.
Both now read the revision pinned_revision returns, which also accepts the
codeload URL the RetroPie scraper records. EmuDeck Windows stays on the
emudeck-we default branch: the platform file pins no revision of it.
2026-10-21 05:24:04 +02:00
Abdessamad Derraz 06ae666f92 fix: describe WinUAE kickstarts by its own code
Every entry took its names from RetroBat's Uae.Generator.cs, which
the RetroBat launchers: records already hold. Kickstarts are now
labels of the ROM scan, and kick.rom, the name the loader opens under
its data path, is listed.
2026-10-21 05:23:14 +02:00
Abdessamad Derraz 5e70fe7ff5 docs: state each verification rule once
architecture.md drew path_suffix ahead of SHA1 and listed a core
resolution that missed the retroarch list entry and built_by. Both now
link to the one statement that matches common.py. The Recalbox table
read hashMatchMandatory into verify.py, which never reads it, and
md5_composite hashes member contents, not member MD5s
(Zip.cpp:71-96). The validation index is built from the platform's
resolved profiles only. The requirement table gains the fields that
make a platform YAML reversible, and pages that name a platform list
say what changes rather than judging it.
2026-10-21 05:22:38 +02:00
Abdessamad Derraz 9edf785513 docs: refresh installer and release pages
The installer output example showed 5865 files and no Not installable
line, while the RetroArch manifest gives 8069 and omits 11. Exit code 2
also comes from every option argparse refuses. RETROBIOS_REF holds a
file list only until the next release deletes its tag, and large files
always come from large-files. The release commit now carries the
manifests the pipeline regenerates. A RetroPie 4.8 image ships Python
3.7.3, below the bootstrap's floor, and the pages say what to use.
2026-10-21 05:22:28 +02:00
Abdessamad Derraz 32bda05be3 docs: align the testing and profiling guides
The test skeleton indexed verify_platform's dict result as a list and
stopped on KeyError: 0. The verification discipline and the profiling
guide hedged rules the project holds as firm. The profile field table
marked as required fields the schema does not require, and now names
the pin that profile_sync --strict needs.
2026-10-21 05:22:20 +02:00
Abdessamad Derraz 253a41ebc8 feat: list the names --emulator accepts
An unknown --emulator name pointed at --list-cores and counted 441 names
while --list-cores prints 189: the buildbot names --emulator accepts
(mednafen_psx for beetle_psx) were listed nowhere. --list-emulators prints
each accepted name beside the emulator it stands for, and the hint names
it.
2026-10-21 05:22:08 +02:00
Abdessamad Derraz 4fa598645e fix: print pack and lock errors on stderr
The lock refusal, "no pack for", "no manifest for" and "no pack to
verify" went to stdout, so a run redirected to a log exited 1 with
nothing on the terminal. A test reads both scripts and refuses an ERROR
line printed to stdout.
2026-10-21 05:22:05 +02:00
Abdessamad Derraz 0061041466 fix: claim the pack output before refreshing data
generate_pack refreshed the data directories before probing its output
lock. The refresh waits for every run holding a tree, and a pipeline holds
them all: a second build sat silent for hours instead of being refused.
The output lock is now taken first, and a refresh that has to wait for a
tree names it.
2026-10-21 05:21:54 +02:00
Abdessamad Derraz a160a5bc6b fix: name the hash formats --from-md5 accepts
A value of another length was looked up as an MD5 and reported NOT FOUND,
a SHA-256 was dropped, and an input with no hash ended on "No valid
hashes found in input". Digests are now read by length (CRC32, MD5 and
its 29-digit Batocera form, SHA-1, SHA-256), SHA-256 is looked up in the
database, and any other value is refused with the formats accepted.
2026-10-21 05:20:59 +02:00
Abdessamad Derraz 7bcd41e0c4 fix: let taiwan and asia-ntsc answer each other
kronos tags the ST-V BIOS epr-19854.ic8 taiwan, mednafen_stv tags it
asia-ntsc, the SMPC area Mednafen names for Taiwan (ss.cpp:1119,
2363-2364). As siblings under asia they never matched, so a request in
one spelling dropped the file tagged in the other, and --region
taiwan,us dropped the Saturn sega_101.bin. AREA_TERRITORIES relates
each area to the territory it serves, china for asia-pal included, in
region.py and its installer copy.
2026-10-21 05:20:08 +02:00
Abdessamad Derraz e318bb4503 fix: read pcsx_rearmed us slot as the default
CheckCdrom sets PSX_REGION_US before it reads the licence, and only
"Inc." or "Euro pe" move it (libpcsxcore/misc.c:422-453). A Latin
American disc boots the US slot, so its four BIOS images carry
latin-america and --region br,jp keeps them.
2026-10-21 05:20:03 +02:00
Abdessamad Derraz 4e8d375074 fix: serve australian mega cd discs their pal bios
BlastEm and Kega Fusion pick the European Mega CD BIOS for every disc
whose region code is the PAL one, Australian discs included: BlastEm
reads E or bit 8 of the header region (romdb.c:176-215), Fusion the
security byte 0x64 at 0x20B. Both now carry oceania, and a repo-wide
test holds every European Mega CD BIOS to it.

Fusion picks its export Master System BIOS from the CountryOrder
setting or a CRC32 list, never from the cart, so bios_U.sms and
bios_E.sms carry no region.
2026-10-21 05:20:00 +02:00
Abdessamad Derraz 7e6b80d501 fix: keep amiberry to the names its code reads
kick02019.AROS and its .ext were Recalbox configgen names, shipped at
the BIOS root of four packs whose code never opens them. Recalbox
keeps them through its own amiga/bios declarations. amiga-os-310.rom
is a ROM scan label, now found_by content.
2026-10-21 05:19:56 +02:00
Abdessamad Derraz 1c2360926b fix: a refused regional file wins no slot
emulator_region_drops counted as held any file that resolved. A regional
dump whose md5 the core's own check refuses won its region, the fallback
was withdrawn, and the emulator pack then refused the winner: no BIOS for
the system, and verify --emulator, reading the same drops, did not see
it. Held now means what the builder ships, as _ships decides on the
platform side.
2026-10-21 05:19:51 +02:00
Abdessamad Derraz bc68e517cb fix: suggest the closest system for --platform
--platform retroarch --system sony-playstaton listed the first ten systems
of the alphabet. The warning now names the closest systems and the
command that lists the platform's own, as --system alone already did.
2026-10-21 05:19:27 +02:00
Abdessamad Derraz 959934d43c refactor: compute the gap report once per build
The region grouping and phase 2 of each pack and manifest called
find_undeclared_files with the same arguments: 30 reports for the 15
manifests of --all --manifest, 2.8 s, now 15 and 1.7 s. The 15 manifests
are byte-identical before and after.
2026-10-21 05:18:53 +02:00
Abdessamad Derraz 805ca3bb6b fix: pass over a profile revision that won't parse
One unparsable commit in a profile's history stopped the walk and
marked every older prose citation unknown, which refused the recale and
bump for good (lindbergh-loader, df7346b3). The revision is now passed
over unless the pins differ on its two sides while the field's
citations changed across it.
2026-10-21 05:18:49 +02:00
Abdessamad Derraz 38af19091d fix: compare pack paths, not only counts
Step 5 compared (ok, total) and the core file count. RetroArch, Lakka
and RetroPie read 526/526 on both sides while verify credited four paths
below SGB1.sfc and SGB2.sfc that the pack never writes: a declared file
blocks them as a directory. verify now places the platform files in the
builder's order with the same path rules, and a user_provided or
external declaration keeps its slot on both sides, so no core file takes
it in the report only.

verify --destinations writes the paths each pack holds by the report,
and step 5 compares them with the archive members (notes and data tree
members set aside). Every emulator profile, in each build it has, is
built in memory (PackRecord, nothing read or written) and compared with
verify --emulator path by path. Cost measured here: about 1 s per
platform archive and 11 s for the 930 emulator packs.
2026-10-21 05:18:02 +02:00
Abdessamad Derraz 247093d7d2 fix: name BizHawk firmware by its own database
BizHawk finds firmware by SHA-1 scan or a user specification, so the
profile is found_by content. 38 entries used the names RetroBat's
generator writes; they now carry FirmwareDatabase.cs recommended
names, and the pooled NDS firmware entry is split per option.
2026-10-21 05:17:34 +02:00
Abdessamad Derraz bca76e8ace fix: judge a cached large file from one read
_fetch_asset hashed the cache copy, then hashed the path again for the
release check. A copy another run promoted in between answered the second
question, and the caller was told the cache held nothing when it held
the file asked for. Both checks now read one snapshot, taken while the
inode stays the same.
2026-10-21 05:17:16 +02:00
Abdessamad Derraz 79d3a86db7 chore: drop an unused import in export tests 2026-10-21 05:17:10 +02:00
Abdessamad Derraz 31566ce618 fix: read a past pin only for its own repository
A source_commit string from before the field was keyed by mode pinned
the standalone repository, yet was read as a writing pin for every mode.
xenia-edge then saw its libretro pin move under unchanged prose, and
the recale and bump were refused.
2026-10-21 05:17:01 +02:00
Abdessamad Derraz 8a19d647a8 fix: pass -v and every short flag to install.py
A [Parameter()] attribute made install.ps1 an advanced script: its common
parameters took -v (-Verbose), -vb, -d and -ea before install.py saw them,
and the scriptblock form printed PowerShell's own VERBOSE lines. The
wrapper now forwards $args as typed.
2026-10-21 05:16:52 +02:00
Abdessamad Derraz 32dcfcf85a fix: list every copy of a file in the DAT
The collection DAT held the one path per content the database keeps, so
auditing bios/ with it read 2,502 copies (RPG Maker, engine trees, dumps
filed under two consoles) as unknown files. It now lists every path under
bios/ that holds a file, 22,028 roms for the 22,031 files on disk, the
other three being .gitkeep placeholders.
2026-10-21 05:16:19 +02:00
Abdessamad Derraz 1392f00c8d fix: replace a link at a standalone copy's name
A link already at the destination was skipped without a word and counted
as "dir not found" while the folder existed, so the emulator kept the file
the link pointed at. The link now gives way to the file, written beside it
and renamed over the name, and a name leading out of the folder is a named
failure.
2026-10-21 05:16:16 +02:00
Abdessamad Derraz 9b1d378d2f fix: never truncate a copy another run renames
On Windows a verified partial copy is closed before its rename. A second
run that claimed it in that moment truncated it and the first rename failed
with a sharing violation, reported as an unwritable folder. A complete
partial is now left whole there, the rename waits for a sharing violation
to clear, and a sharing violation no longer reads as a refused folder.
2026-10-21 05:16:12 +02:00
Abdessamad Derraz c638c12ca8 fix: replay the builder's held and one-file groups
install.region_drops now skips one-file groups (:file:) and lets no
omitted match displace an installable file, as region.resolve_region_drops
does on the builder side. The property test draws both cases and
compares against the builder.
2026-10-21 05:15:56 +02:00
Abdessamad Derraz 1a098cf975 fix: link revisions on forges profile_sync skips
parse_repo knows only the forges profile_sync fetches from, so the eduke32,
fury, dgen, skyline and splitwolf pages showed no pinned revision and
their citations stayed plain code. The pages now build commit and file
links for voidpoint.io, Bitbucket, SourceForge and Software Heritage.
2026-10-21 05:15:01 +02:00