Commit Graph
2398 Commits
Author SHA1 Message Date
Abdessamad Derraz 184c822e6e fix: name the file each layer reads in slots.py
Each content-checked line ended with a verdict on the platform's list,
printed at pipeline step 3b. The line now says which emulators read the
other file, and the closing count says the rest follow the platform's
list. The module source carries no such verdict, a test reads it.
2026-10-21 05:33:50 +02:00
Abdessamad Derraz ceb8b752e6 fix: count bundled gaps apart in each heading
cross_reference.py --emulator abuse printed "Gaps: 3 missing" above three
[BUNDLED] files, then "0 missing (need to be sourced)" in the total. The
heading and the total now read one count: bundled files appear as
"N bundled", never among the missing.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz c05d29d941 fix: a held name must carry the declared hash
entry_source took a name or path hit before reading the hashes the entry
declares, so the gap report and the site counted as held a file whose
only homonym has other bytes: zquestclassic's ending.mid and gameover.mid
matched Hydra Castle Labyrinth's. A name or path hit now counts only
when its record carries one of the declared hashes, and the declared
content anywhere in bios/ comes before a data/ copy. Measured over every
profile entry: 254 change, 248 bundled ones (report and site only),
three unsourceable SC-55 revisions now read missing, three data hits
read bios by hash.
2026-10-21 05:32:19 +02:00
Abdessamad Derraz 0dab8f4706 fix: credit file readers through the data trees
FileConsumers resolved profile entries without the data directory
registry the builder uses. Dolphin's hashless codehandler.bin then took
Dolphin-MMJR's copy by name, and the site listed Dolphin and Triforce as
its readers while every pack serves data/dolphin-sys. blueMSX's RS232.ROM
and MSX2P.rom were credited to bios/ copies the same way. The registry
is now read by default, as the builder reads it.

With it, bundled_only.py --strict (pipeline step 4d) names
bios/Other/dolphin-mmjr/codehandler.bin: only dolphin-mmjr's bundled
entry, which already carries its sha1 and size, reaches that copy.
2026-10-21 05:29:42 +02:00
Abdessamad Derraz 5e95574143 fix: the strongest proof takes a shared core path
A destination no platform declares went to the first claimant in profile
order whatever the proof: fbalpha2012 asks for hiscore.dat by name and
took the root of six packs over cores that pin it by sha1. Claims on a
contested path are now ranked by what the collection holds for each (a
held declared hash, then a path or every required archive member, then a
name, then a contradicted hash), and profile order only breaks ties.
Measured: 14 slots change file across the platforms, among them
hiscore.dat, kick13.rom on RetroPie, cgb_boot.bin and dmg_boot.bin on
EmuDeck, and ROCKNIX neogeo.zip and neocdz.zip.
2026-10-21 05:28:10 +02:00
Abdessamad Derraz 8eda797562 fix: a hashless entry is a path in any mode
Whether the frontend compares content was decided per platform. EmuDeck
(md5) declares neocdz.zip with no hash and checkBIOS.sh never names it,
yet the proven claims of geolith and FBNeo were dropped and the pack
shipped a neocdz.zip without 000-lo.lo. checked_by_path() decides per
entry: no hash of the kind the frontend compares (any of md5, sha1,
crc32 under RomM's rule) and no zipped_file means a path. Measured: the
EmuDeck neocdz.zip slot and three RetroBat DS slots (dsi_bios7.bin,
dsi_bios9.bin, firmware.bin) now take the file the cores prove.
2026-10-21 05:24:19 +02:00
Abdessamad Derraz 1c2360926b fix: a refused regional file wins no slot
emulator_region_drops counted as held any file that resolved. A regional
dump whose md5 the core's own check refuses won its region, the fallback
was withdrawn, and the emulator pack then refused the winner: no BIOS for
the system, and verify --emulator, reading the same drops, did not see
it. Held now means what the builder ships, as _ships decides on the
platform side.
2026-10-21 05:19:51 +02:00
Abdessamad Derraz 38af19091d fix: compare pack paths, not only counts
Step 5 compared (ok, total) and the core file count. RetroArch, Lakka
and RetroPie read 526/526 on both sides while verify credited four paths
below SGB1.sfc and SGB2.sfc that the pack never writes: a declared file
blocks them as a directory. verify now places the platform files in the
builder's order with the same path rules, and a user_provided or
external declaration keeps its slot on both sides, so no core file takes
it in the report only.

verify --destinations writes the paths each pack holds by the report,
and step 5 compares them with the archive members (notes and data tree
members set aside). Every emulator profile, in each build it has, is
built in memory (PackRecord, nothing read or written) and compared with
verify --emulator path by path. Cost measured here: about 1 s per
platform archive and 11 s for the 930 emulator packs.
2026-10-21 05:18:02 +02:00
Abdessamad Derraz fb639647e6 fix: resolve a loose archive entry by its members
gngeopi and neo-emu declare neogeo.zip loose and, apart, the ROMs they
open inside it. The archive pass took the copy holding them, the loose
entry resolved by name to another copy, and both verify --emulator and
the emulator pack reported the archive missing while the ZIP held it.
with_archive_members() gives the loose entry the members in verify, the
pack, slots and the gap report.
2026-10-21 05:05:54 +02:00
Abdessamad Derraz 348c56e94c docs: put the project's working rules in the wiki
Principles, collection rules and the inverted model in architecture.md, research and citation rules in profiling.md, platform and scraper rules in their pages, tool conventions in tools.md, test and commit expectations in testing-guide.md, installer constraints in installer.md. Each rule sits where a reader looks for it, with its reason. Also fixes the stale release-job line, the suite command without -t ., and the unsourceable definition.
2026-10-21 04:03:31 +02:00
Abdessamad Derraz 392aacb207 fix: hash a large download through its own handle
On Windows the partial copy of a file of 16 MB or more is locked with msvcrt.locking, a LockFile region: the locking process cannot access that region through a second handle (Microsoft LockFile remarks). Hashing reopened the path, the read failed with a sharing violation, and every large file was reported as 'folder not writable' (issue 82). Both digests are now read in one pass through the handle that holds the lock. A read-only file already at the destination had the same misleading report: on Windows its attribute is now cleared before the verified copy replaces it, and a refusal that remains is named for what it is.
2026-10-21 03:46:14 +02:00
Abdessamad Derraz 9fd50a21fd chore: regenerate the database and README
The last pushes changed the data without regenerating them, and Deploy Site requires the committed README to match what the generator writes.
2026-10-21 03:23:57 +02:00
Abdessamad Derraz 16261b448b chore: regenerate the RetroPie manifest and export 2026-10-20 23:05:34 +02:00
Abdessamad Derraz cf54730a17 fix: lay out RetroPie standalone packages
RetroPie's YAML listed no standalone_cores, so every package was laid
out as a libretro core and the standalone files of amiberry, dolphin,
hatari, mame and ppsspp never reached the pack. Packages without the
lr- prefix (bar the retroarch frontend) are now standalone, and names
shipped both ways are dual-mode.
2026-10-20 23:04:42 +02:00
Abdessamad Derraz 027379c065 chore: regenerate the updated platform files 2026-10-20 22:38:32 +02:00
Abdessamad Derraz 6d38e597db fix: keep a RetroPie list's comma before its and
quasi88 writes its list "N88KNJ2.ROM, and N88SUB.ROM". Extended with
N88EXT0.ROM, which QUASI88kai reads, it came back as "N88SUB.ROM and
N88EXT0.ROM": every list was rewritten in one idiom. A list that puts a
comma before its final and keeps it.
2026-10-20 22:38:32 +02:00
Abdessamad Derraz c3af859621 chore: regenerate the updated platform files 2026-10-20 22:20:54 +02:00
Abdessamad Derraz e787efca22 fix: match a size-checked entry to its launcher
EmuDeck passes Complex_4627v1.03.bin as xemu's flash ROM
(emuDeckXemu.sh:60-61), and xemu's entry Complex_4627.bin states no hash,
only the sizes it maps. The layer matched entries by name or content
only, so without the alias the default name was still asked for. A
hashless entry that checks a size now takes a launcher file of a size
it accepts, when no other entry of that core accepts that size too.
2026-10-20 22:20:42 +02:00
Abdessamad Derraz 540374b40d fix: count a launcher's names as read
The removal candidates looked for a reading in the profiles only, so a
name the platform's own launch code opens, links or copies was listed
as read by no core: Batocera's DuckStation generator walks 22 names
(duckstationGenerator.py:501-515). Every name a launcher record names
is now a reading on that platform. Recalbox's unexplained entries drop
from 20 to 13, RetroBat's from 24 to 20, EmuDeck's from 25 to 22.
2026-10-20 22:20:42 +02:00
Abdessamad Derraz 8dc485dcd5 fix: move Hatari's TOS names to the launcher layer
The 43 tos.img aliases in hatari.yml and 10 in hatarib.yml were names Batocera, Recalbox and RetroBat launchers pass to Hatari (hatariGenerator.py, Hatari.Generator.cs, libretroCores.py), now recorded in each platform's launchers: records. With standalone Hatari profiled, those records attach. Measured on Batocera, RetroBat, Recalbox, ROCKNIX and RetroPie: verify statuses, manifest files, omissions and Hatari entries are identical before and after.
2026-10-20 22:03:20 +02:00
Abdessamad Derraz 40e9ed58ad feat: profile standalone Hatari
Batocera (v2.6.1, hatari.mk), RetroBat (v2.4.1 release), Recalbox and ROCKNIX run standalone Hatari, which the profile did not describe, so their launcher records had no build to attach to. Standalone Hatari reads tos.img from its data directory by default (src/configuration.c:810-812, src/paths.c:316-330), retries a missing name with .img, .rom and .tos (src/file.c:275-308, 481-520), and exits when no TOS loads (src/main.c:191-201). The profile is now standalone + libretro, with the keymap file and the window icon as standalone entries and BOOT.ST scoped to libretro.
2026-10-20 21:59:32 +02:00
Abdessamad Derraz ec1a441f4a fix: drop Recalbox launcher names from gsplus
apple2gs1.rom and apple2gs3.rom come only from Recalbox's gsplusGenerator.py:107-110, 140, a platform launcher now recorded in platforms/recalbox.yml. No platform resolves gsplus.yml (Recalbox, Batocera and RetroBat run gsplus-legacy), and verify and the manifests of Recalbox, RetroDECK, ROCKNIX and EmuDeck are identical before and after.
2026-10-20 21:48:43 +02:00
Abdessamad Derraz c445c18bdc fix: map RetroPie's standalone builds to profiles
RetroPie builds minivmac/minivmac, winterheart/quasi88 and TurtleBazooka/px68k under the package names minivmac, quasi88 and px68k (scriptmodules/emulators/minivmac.sh:12, quasi88.sh:12, px68k.sh:12). With built_by retropie and those names in cores, RetroPie resolves them to minivmac37, quasi88kai and px68k-sdl, while its libretro builds stay reached through retroarch. Measured: RetroPie keeps 532 of 532 platform files OK and gains the 14 files the standalone builds read; no other platform's resolution changes.
2026-10-20 21:46:16 +02:00
Abdessamad Derraz eaba9ce901 chore: regenerate the database and manifests 2026-10-20 21:46:10 +02:00
Abdessamad Derraz fbbc0512ac chore: drop a cgrom.dat copy named cgrom.tmp
bios/Sharp/X68000/cgrom.tmp had cgrom.dat's sha1 (8d72c5b4). px68k reads cgrom.tmp only when cgrom.dat is absent (libretro.c:915-919, SDL/SDL2/winx68k.cpp:317-331), and a real cgrom.tmp is a font mkcgrom generates on the user's machine. No platform declares it.
2026-10-20 21:46:10 +02:00
Abdessamad Derraz 3ebbcd5fc2 chore: regenerate the README and manifests 2026-10-20 21:42:45 +02:00
Abdessamad Derraz c702cdf62e fix: pack no launcher file the emulator installs
The launcher layer packed a file under the launcher's name whatever the
platform installs: ROCKNIX's xemu config reads hdd/xbox_hdd.qcow2, and
the profile marks that image installed_on rocknix, yet the pack brought
one. A place whose entry the emulator installs on that platform is now
filled by the install, as validation.deliverable has it for the pack.
2026-10-20 21:28:57 +02:00
Abdessamad Derraz 6295bf4545 refactor: drop the RetroBat LaserActive v1.02 name
RetroBat's ares launcher tries v1.05, v1.02 then v1.01 (Ares.Generator.cs:371-437), now recorded under launchers: in platforms/retrobat.yml. ares itself identifies the image by sha256. No file carries the name, so the database is unchanged.
2026-10-20 21:26:48 +02:00
Abdessamad Derraz b2b8dcd62c feat: read platform launchers in verify and packs
A names_searched list is one place the launcher fills with the first name it finds, so a core file it would never open once another name of the list is held is not reported missing. A place no declared file and no packed core file fills receives, under the launcher's name, a file the core accepts by content. The layer is applied inside find_undeclared_files, which verify and the pack builder both read, and a test reads their source to keep it there.
2026-10-20 21:26:48 +02:00
Abdessamad Derraz 8660892d60 docs: list the RetroPie and RetroDECK tests 2026-10-20 21:13:17 +02:00
Abdessamad Derraz 74a9d7beea feat: read RetroDECK launchers from components
The scraper reads each component's prepare, launcher and functions scripts at the components commit the BIOS list comes from, and the presets they copy: the fixed paths set_setting_value writes (melonDS seven, xemu three, Dolphin GBA), the folders handed over, the dir_prep links (LRPS2 pcsx2/bios back to bios/, Dolphin and PrimeHack GC regions under saves/), package copies into bios/, RetroArch core options naming a file (pcsx2_bios, DuckStation per region), XRoar romlists, the PICO-8 exec and the RPCS3 and Vita3K firmware downloads.

platforms/retrodeck.yml is regenerated at fcd0d729: only the launchers section is new.
2026-10-20 21:13:09 +02:00
Abdessamad Derraz 15ffeddfe3 feat: read RetroPie launchers from scriptmodules
The scraper walks each package's functions the way RetroPie runs them in one shell (assignments, arrays, for loops, brace expansion, $md_id, calls into other packages) and records every link, copy, move and setting that names $biosdir, the addEmulator flags (sdltrs -romfile3), the BIOS read outside $biosdir (basilisk --rom, gngeopi -B) and the launch scripts that exit without their file (fs-uae kick13.rom, reicast dc_boot.bin).

Each package also gets a package record with its rp_module_repo and the systems it registers, which maps a package to the build it compiles. The schema gains package, repo, systems and the moves and package kinds. platforms/retropie.yml is regenerated at 46128cd2: only the launchers section is new.
2026-10-20 21:13:09 +02:00
Abdessamad Derraz 1c23d4daaa chore: regenerate the Recalbox platform file
Scraped at 10.1.1. Only the launchers section is new: 85 records read from configgen.
2026-10-20 20:59:51 +02:00
Abdessamad Derraz fe9a081cfb feat: read Recalbox launchers from configgen
The scraper now records, from configgen at the pinned tag, the names each Recalbox launcher tries and where it hands them: the Hatari TOS cascades (standalone, hatari and hatarib cores), GSplus, Amiberry kickstarts, the PCSX2 region choice, the Dolphin IPL presence test, xemu, the mupen64plus 64DD IPL, the Mini vMac boot disk and the o2em_bios option. The system_directory libretroRetroarch gives each core, and the per-system overrides after it, are passes_path records.
2026-10-20 20:59:51 +02:00
Abdessamad Derraz 4465edcd0a refactor: import launchers from the shared reader 2026-10-20 20:59:41 +02:00
Abdessamad Derraz 98aebed900 refactor: share the configgen launcher reader
Recalbox starts its emulators from a Python configgen too. The parsed source, the record and the ast helpers move to configgen_launchers so both readers use them. Batocera's extractors are unchanged.
2026-10-20 20:59:35 +02:00
Abdessamad Derraz a9881caaf8 docs: list the ROCKNIX and EmuDeck launcher tests 2026-10-20 20:45:19 +02:00
Abdessamad Derraz 01ddbe5509 chore: regenerate the ROCKNIX and EmuDeck files 2026-10-20 20:45:19 +02:00
Abdessamad Derraz 3837d6f097 feat: read EmuDeck launchers from EmuScripts
The scraper reads the EmuScripts at the commit checkBIOS.sh is pinned to: the BIOS paths changeLine, iniFieldUpdate and RetroArch_setConfigOverride write, the ares and melonDS configs whose paths sed rewrites, and the bios/ directories linked to Flycast, Azahar, the Switch emulators and shadPS4. The script to emulator mapping moves beside the extractors so both read one table.
2026-10-20 20:45:19 +02:00
Abdessamad Derraz 57af8e4921 feat: read ROCKNIX launchers from its packages
The scraper reads the start scripts and the configs each package installs at the tag the BIOS list is pinned to: configs that name a BIOS path, the yabasanshiro search, directories handed to DuckStation and the PCSX2 builds, and the ln -s commands that put bios/ where an emulator reads. A package that ships one config per device gets a device field only where the values differ, as melonDS does on AMD64.
2026-10-20 20:45:19 +02:00
Abdessamad Derraz a9c175d322 chore: regenerate the updated platform files 2026-10-20 20:44:42 +02:00
Abdessamad Derraz afc2ce3268 docs: list the RetroBat launcher test module 2026-10-20 20:15:04 +02:00
Abdessamad Derraz ef68cc6615 chore: regenerate the RetroBat platform file
Scraped at emulatorLauncher 48cdc616, the revision already pinned: only the launchers section is new.
2026-10-20 20:14:52 +02:00
Abdessamad Derraz 419d68d1f2 feat: read RetroBat launcher records
The scraper reads emulatorLauncher's generators and es_features.cfg at the commit the BIOS list is pinned to: Ares, Hatari, Mesen, Amiberry, WinUAE, PCSX2, the PSX BIOS cascades, Xemu, openMSX and DuckStation. Each extractor raises when its construct is gone.
2026-10-20 20:14:52 +02:00
Abdessamad Derraz 897fe17ab2 feat: read Batocera launchers from configgen
The scraper parses the configgen generators with ast at the pinned tag: DuckStation and its legacy build, Hatari, xemu, melonDS, mupen64plus 64DD, x16emu, GSplus, and the o2em and neocd core options. An extractor that no longer recognises its code raises instead of writing an empty list.
2026-10-20 20:14:30 +02:00
Abdessamad Derraz 2556a03ed3 feat: describe platform launchers in the schema
A launcher record says what a platform's own launch code looks for and hands to an emulator, read from that code at the pinned revision.
2026-10-20 20:14:30 +02:00
Abdessamad Derraz 7151ea212c chore: regenerate the updated platform files 2026-10-20 20:14:22 +02:00
Abdessamad Derraz 479ada29fe docs: word removal candidates by what was searched
"No core opens it" claimed more than the search shows: what was
checked is that no profile of a core the platform runs names the
entry by name, path or hash. The reason and the page sentence now
say that.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 545e3f3ba1 feat: record each export change before and after
The page named the entries a change touched and never the values, so
a maintainer could not check a correction. The report now carries
changes, one object per change with kind, entry, field, old and new,
read from the platform entry and from the value the render writes
(written() per format: record value for System.dat and RomM, one md5
where a single one is checked). The existing label lists are derived
from it. The page writes "md5 X becomes Y" and names each required
flag with its new value. A test parses the original and updated
System.dat and es_bios.xml and checks the list equals the diff.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 5cba005d67 fix: keep the platform's hash order when unchanged
Where the truth names no value the platform lacks, hashes() still put
the truth's values first and rewrote the list in another order:
Recalbox's msx/CARTS.SHA md5 attribute came out with its two values
swapped, a diff line that changes nothing the frontend checks. The
platform's own list now stands when it already holds every value.
2026-10-20 20:00:54 +02:00