Commit Graph
1264 Commits
Author SHA1 Message Date
Abdessamad Derraz 479ada29fe docs: word removal candidates by what was searched
"No core opens it" claimed more than the search shows: what was
checked is that no profile of a core the platform runs names the
entry by name, path or hash. The reason and the page sentence now
say that.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 545e3f3ba1 feat: record each export change before and after
The page named the entries a change touched and never the values, so
a maintainer could not check a correction. The report now carries
changes, one object per change with kind, entry, field, old and new,
read from the platform entry and from the value the render writes
(written() per format: record value for System.dat and RomM, one md5
where a single one is checked). The existing label lists are derived
from it. The page writes "md5 X becomes Y" and names each required
flag with its new value. A test parses the original and updated
System.dat and es_bios.xml and checks the list equals the diff.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 5cba005d67 fix: keep the platform's hash order when unchanged
Where the truth names no value the platform lacks, hashes() still put
the truth's values first and rewrote the list in another order:
Recalbox's msx/CARTS.SHA md5 attribute came out with its two values
swapped, a diff line that changes nothing the frontend checks. The
platform's own list now stands when it already holds every value.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 0b4f98ce79 feat: report entries no examined core opens
The export keeps every entry a platform declares. An entry the truth
does not pair was still read in 1189 of 2776 cases (alias, hash, a
type: directory tree, a core outside the resolved set), so absence of
a pairing proves nothing. Each report now lists, under
removal_candidates, the unpaired entries that no profile of the cores
the platform runs answers by name, path tail, hash, directory or data
directory, with the cores examined and a reason. A core the platform
names without a profile, or a ROM set under a core that opens sets by
name, leaves the verdict undetermined. The exports page states the
count and lists the entries per platform.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 7e0b3a08b9 fix: keep System.dat lines whose values agree
The rom line renderer upper-cased every crc, so Dinothawr.zip's
683ed4ad, the one lower-case crc libretro writes, changed with no
correction behind it. A line whose values match ours now keeps its
bytes, a corrected crc follows the case of its own line, and an added
line keeps the file's upper case. An empty truth hands System.dat back
byte for byte for RetroArch and Lakka.
2026-10-20 19:39:36 +02:00
Abdessamad Derraz eba5f1dd62 fix: patch Recalbox's attributes, not its elements
A changed element was rendered again, losing its alignment and the
place of its attributes, and an element whose md5 list was only
reordered (neogeo.zip, the same 24 digests) was rewritten too: Recalbox
accepts any digest of the list (Bios.cpp:109-130), so the order is not
content. A changed attribute is now replaced where it stands, one
dropped goes with its space, one added goes before />, and each system's
opening tag keeps its own indentation. An empty truth gives back the
cached es_bios.xml byte for byte.
2026-10-20 19:36:09 +02:00
Abdessamad Derraz aaffcdcb63 fix: patch Batocera's values, not its entries
An entry with one md5 to correct was written again whole: atarist lost
its eight blank lines, every '"x"},' became '"x" },', and most of the
819 lines Batocera's export changed were layout. Each item now keeps
its own text and only the md5 inside it changes; an addition follows
the last item in its indentation and spacing. The patched entry must
read as the one that would have been written, or that one is written.
Batocera's diff to the original drops to 273 lines, ROCKNIX's to 3.
2026-10-20 19:22:58 +02:00
Abdessamad Derraz 659e9b591f fix: keep RetroBat's blank line between systems
RetroBat separates every system after the first with a blank line.
json.dumps dropped all 100 of them: 100 hunks of noise in a diff meant
to show corrections. The separator is written again where the original
has it, and an added system takes it too where the original separates
them all. An empty truth gives back the cached file byte for byte.
2026-10-20 19:07:46 +02:00
Abdessamad Derraz 804534397f fix: write the MiSTer database on one line
The BiosDB is published as a single line of JSON without a final
newline. Written with indent=2 it came out as 1499 changed lines for a
database whose content had not changed, which would bury any real
correction. It is now written in MiSTer's own form: an empty truth
gives back the cached database byte for byte.
2026-10-20 18:54:28 +02:00
Abdessamad Derraz 3ed5f68b82 fix: hand RetroDECK manifests back as written
With nothing to correct, nine of ten manifests still changed: a final
newline was added where the original ends without one, "md5": null was
written 39 times where no md5 key was, and PPSSPP's single bios object
became a list rebuilt from our data. A manifest now ends as the
original does, keeps no key it did not have, and a single object stays
one while nothing is added. An empty truth gives back every cached
manifest byte for byte.
2026-10-20 18:41:26 +02:00
Abdessamad Derraz 85c2433c98 fix: write each export as its bytes and mode
write_text translated line ends on Windows, so FirmwareDatabase.cs's
CRLF would have come out CR CR LF; it replaced the published file in
place, so a run stopped mid-write left it truncated; and it dropped the
mode, while rocknix-systems is 100755 in ROCKNIX's tree. Files are
written atomically as UTF-8 bytes, and an exporter states the mode of
any file the platform does not publish as 0644.
2026-10-20 18:27:22 +02:00
Abdessamad Derraz cbe769e74c fix: read all RetroPie helps, propose only its own
Eight scriptmodules write "bios" in lower case (xm7, np2pi, quasi88,
minivmac, px68k...) and none was examined. A module reported "left as
is" for a name no profile documents was written all the same: the skip
had no continue. The cap of six names came from that case-sensitive
reading, where xm7 writes thirteen: a sentence now grows no longer than
the longest RetroPie writes. lr- was stripped whatever the build, so the
standalone minivmac package matched the libretro profile, which reads
another file. A file of a system RetroPie lacks is proposed to no
package: lr-geargrafx adds pcengine only (lr-geargrafx.sh:41-42) and
was given the LaserActive's pac-n1.bin and pac-n10.bin.
2026-10-20 18:12:56 +02:00
Abdessamad Derraz 95476afb28 fix: make the report directory before writing it
A platform with nothing left to correct writes no file, so its output
directory was never made: RetroPie's report then raised
FileNotFoundError into a fresh --output-dir and stopped --all.
2026-10-20 18:12:45 +02:00
Abdessamad Derraz d92291684c fix: dual-mode only what ROCKNIX runs both ways
The scraper declared every standalone package launched both ways,
reasoning that RetroArch carries every libretro core. package.mk says
otherwise: it adds ARMSX2 as armsx2-sa only (add_emu_core ps2 armsx2
armsx2-sa true), so the libretro build's pcsx2/resources entries
reached ROCKNIX's list as armsx2-sa files the standalone installs
itself. An emulator is now dual-mode only when package.mk also adds it
as a retroarch core: ten of the 63 standalone names.
2026-10-20 17:53:57 +02:00
Abdessamad Derraz a5d7a6572d fix: ask for no file the emulator installs
The pack leaves out what an emulator installs itself
(validation.deliverable); the truth did not. RetroPie's gngeopi help
asked users to copy gngeo_data.zip, which make install puts in the
package's data directory, and amiberry's whdboot tree, scummvm's extra
data and dosbox-x's FREECG98.BMP went into platform lists as files to
bring.

The truth applies deliverable per platform and per build: an entry
nobody has to bring is kept and marked bundled, never added, still
corrected where the platform declares it. A rebuilt archive is
installed only when every member is. The runner written beside an
addition is the build that needs the file (_brought_by), and the
RetroPie sentence names only what its package's core must be given.
2026-10-20 17:39:21 +02:00
Abdessamad Derraz f957b5f893 fix: open no System.dat section libretro lacks
libretro titles each section with the system as it spells it,
"Commodore - Amiga". An addition for a system it has no section for
opened one under our slug: 21 sections came out as commodore-c64,
quake, laseractive or xrick, which no libretro DAT carries. An addition
now lands only in a section the original holds and is otherwise counted
as left out, so the export keeps the original's 66 sections. A platform
entry is always written, under its own section.
2026-10-20 17:21:16 +02:00
Abdessamad Derraz e5174a07bf feat: mark files a core finds by content
DuckStation identifies its BIOS by MD5 in a folder it scans
(bios.cpp:208), so its entries' names are labels. The export wrote them
into Batocera's and ROCKNIX's lists as files to hold: a user with only
scph5501.bin read 104 missing BIOS where DuckStation boots. Labels also
lent their md5 to a name another core opens, and paired by name they
replaced System.dat's scph5000.bin hashes.

found_by: name | content, on a profile or a file and keyed by build like
load_from. The truth records which cores open each name (_named_by). A
label pairs with a platform entry by hash only, is never added, merges
with an opened name only on a shared hash, and is counted in the export
report rather than listed as missing.
2026-10-20 16:32:22 +02:00
Abdessamad Derraz 0eec5ab4be fix: file a truth entry by content, not by name
A name the platform declares filed an entry under that system before
its own system:, and the last declaration of a name won. fmsx's 16 KB
DISK.ROM went to nec-pc-88 for quasi88's 2 KB disk.rom and came out as
a second DISK.ROM line under "NEC - PC-8801"; dosbox's SC-55
ROM1.BIN/ROM2.BIN went to galaksija.

A declaration now files an entry only when it can be the same file: a
shared hash, or no size both admit. The same content declared under two
systems is filed under both, an entry's own system stands against a
refuted homonym, and a systemless profile's placeholder takes only names
the platform does not declare. Merging reads the same size rule, so a
24 KB ROM and an entry capped at 16 KB stay two files.
2026-10-20 16:14:55 +02:00
Abdessamad Derraz f758efd062 fix: write no export its validator refuses
validate() ran after the files were written, so an export the
exporter's own validator refused still replaced the published one in
exports/, and only the exit code said so. The platform now fails before
anything is cleared or written.
2026-10-20 15:55:29 +02:00
Abdessamad Derraz 49b79a6d52 fix: refuse a file read off the pinned revision
pinned_base returned an empty base when the platform's source named no
revision or no longer ended in the exported file, and the exporter's
own branch URL was then patched without a word. Every file an export
reads must now carry the revision the platform YAML records, or the
platform fails and asks for a rescrape. The check covers every file of
every platform: ROCKNIX's second file, RetroDECK's manifests and
RetroPie's tarball were checked by nothing before.
2026-10-20 15:40:28 +02:00
Abdessamad Derraz 4da40c471d fix: isolate an unreadable original per platform
A cached bios_db.json.zip that was not a zip raised BadZipFile out of
run(): --all stopped and the platforms after it were never exported.
fetch() also cached any 200 reply, so an error page replaced a good
archive and was served on every later run, and an IncompleteRead or an
oversized reply escaped the same way. An archive is now read before it
is cached, and an unreadable file fails its own platform with the
--refresh-cache remedy.
2026-10-20 15:24:30 +02:00
Abdessamad Derraz 8de45b6f45 feat: generate MAME device sets from -listxml
A machine that instantiates a device loads the device's ROMs from <shortname>.zip (src/emu/device.cpp:140-152), and a missing required ROM stops that machine (romload.cpp:529-570, 667-671). The BIOS root scan never sees these sets. mame_hash_scraper --devices --listxml reads them from the -listxml of the release mame.yml pins, refuses another release, and cites each set at the ROM_START block holding its ROMs. The BIOS root merge also wrote refs without the upstream prefix and would have rewritten all 171 of them in mame.yml.
2026-10-20 15:12:23 +02:00
Abdessamad Derraz 7a65a2cf83 fix: refuse an export without its original
System.dat, RomM, Recalbox and RetroBat were rebuilt from our data when
their file was not cached: libretro's quoted names, Recalbox's comments
and one of its two zxspectrum blocks were lost, and the run said OK.
needs_original now follows native_sources, so every format we patch
fails without the file it patches and writes nothing.
2026-10-20 15:09:16 +02:00
Abdessamad Derraz 19b88ca055 fix: refuse an export without its truth
dist/ is not committed and exports/ is. On a fresh clone, or with a
mistyped --truth-dir, the export copied each platform's file back
uncorrected, said OK and replaced the corrected export in exports/.
check_freshness then reported a stale export instead of the missing
model. The platform now fails, writes nothing, and names the
generate_truth command.
2026-10-20 14:54:16 +02:00
Abdessamad Derraz e81c63804a fix: correct no hash the code does not check
validation: names what the code verifies. dosbox checks
MT32_CONTROL.ROM against nine sha1 and size; the truth's one md5 and
crc32 describe one of those revisions, and the export wrote them beside
System.dat's own sha1, a rom line no file satisfies, and replaced
Batocera's md5. pcsx_rearmed and DuckStation check scph1001.bin by size,
and the v2.0 md5 platforms name was replaced by v4.1's.

checked_hashes keeps a hash the code does not name only where a checked
hash pins one content. Otherwise it still pairs a renamed file but never
corrects or fills a value the platform wrote. Additions keep every hash.
diff_truth reads the same rule.
2026-10-20 14:40:10 +02:00
Abdessamad Derraz 9bcfa049ba fix: carry a profile destination into the truth
A profile states where the code reads a file as path: or destination:,
and the resolver reads both. The truth copied path only: redream's
dc_bios.bin, read at dc/boot.bin, reached the export with no path, so
System.dat's dc/boot.bin was not seen and a dc_bios.bin line was added,
and Batocera got bios/dc_bios.bin instead of bios/dc/boot.bin.
2026-10-20 14:24:05 +02:00
Abdessamad Derraz 181ad0c630 fix: pair truth entries hash-consistent first
Each truth entry claimed its platform line in turn, so the first one to
arrive won. dosbox-x's system.rom (alias bios.rom, no path) took
Batocera's np2kai/BIOS.ROM by name before np2kai's own np2kai/bios.rom
was tried, and the export wrote an IBM PCjr md5 on it. A path also
outranked a contradicting hash: the truth's MSX.rom joins bluemsx's path
and fmsx's md5, which replaced the blueMSX dump on Recalbox's and
System.dat's Machines/Shared Roms/MSX.rom while fmsx's own line held it.

pair_entries runs each test over every entry before the next, tests that
contradict no hash first. export_native and diff_truth both read it.
2026-10-20 14:24:00 +02:00
Abdessamad Derraz 19974e34b8 docs: state only checked facts on the exports page
The intro gave an opinion on the effort lists take and said nothing is removed, while RetroArch and Lakka lose three hashes and one size. It now says no entry is removed, that dropped values are listed per platform, and names exactly what the tests run on patched code.
2026-10-20 13:33:53 +02:00
Abdessamad Derraz 7a3cbc657d docs: describe other projects by facts only
Profile notes and wiki pages called .info entries phantom or incorrect, and labelled an upstream list a known defect. They now state what each file declares and what the code reads. The wiki claimed Batocera prefix-matches truncated md5s; its script compares the full string (batocera-systems:1088), which the page now says. The gaps column reads "Not read by profiles" and notes that a profile may be incomplete.
2026-10-20 13:33:53 +02:00
Abdessamad Derraz 048334cc82 docs: offer the platform files as peers would
The page called the original "their file" and listed hashes "corrected" and additions "left out", a verdict on work the page only proposes to extend. It now says what each updated file changes in neutral terms built from the report's numbers, gives the reason a file is not added, credits the effort a list takes to keep current, and invites an issue when a change does not fit a platform. A test keeps the judging words off the page.
2026-10-20 13:20:34 +02:00
Abdessamad Derraz 6e746f0f3b docs: drop dashes from the README method list 2026-10-20 12:46:37 +02:00
Abdessamad Derraz 721622c2ea fix: keep bold labels in nested profile blocks
The nested list structure stays, so each label renders bold instead of showing its asterisks. The platform details block becomes a list as well: its labels were paragraphs followed by items with no blank line between them. validate_site now reports any asterisks that reach a rendered page outside code, which runs in the deploy and in pipeline step 8b.
2026-10-20 12:46:37 +02:00
Abdessamad Derraz 90b0313470 fix: render nested profile blocks as lists
A key holding a mapping was written as a paragraph, and its children landed eight spaces deep inside the four-space admonition, where Markdown reads an indented code block: the clownmdemu analysis showed raw asterisks. Each level is now a list item with its children as a sublist, booleans read yes or no, and a null reads none.
2026-10-20 12:32:41 +02:00
Abdessamad Derraz 6a852737a8 fix: one file under two spellings keeps both cores
BlastEm opens 32X_M_BIOS.bin and PicoDrive 32X_M_BIOS.BIN. On a case-insensitive platform the manifest kept the first and dropped the second core's claim, so an install narrowed to PicoDrive (RetroArch on Switch) lost the 32X BIOS its pack carries. The claim now joins the entry kept.
2026-10-20 11:27:08 +02:00
Abdessamad Derraz 6181ee429a chore: keep MAME's support trees whole
hash/ and plugins/ are read by path as a whole, and two MAME versions share most software lists byte for byte. Dedup would leave one tree incomplete.
2026-10-20 10:37:30 +02:00
Abdessamad Derraz 0938c30b65 fix: a platform's own build answers its core name
Recalbox compiles uae4all from Chips-fr/uae4all-rpi, a libretro core, while RetroPie's uae4all package is the standalone uae4all2. A profile whose built_by names the platform now wins the name there, and a bare key match still decides everywhere else.
2026-10-20 10:29:47 +02:00
Abdessamad Derraz 40bc07aa3a feat: record files a launcher never opens
unread_on names the platforms whose launcher points the emulator at other files. RetroBat writes absolute BIOS paths into standalone BlastEm's config, so the default names were reported as required and missing there. Such an entry leaves that platform's gap report and pack, and its emulator page says where it is never opened.
2026-10-20 10:27:15 +02:00
Abdessamad Derraz ef8b372810 chore: drop copies only an install provides
A file that arrives with the emulator is already on the user's machine
and the packs leave it out. 4558 collected copies (1.4 GB) were reached
only by such entries, with no platform, other profile entry or install
manifest needing them. They are removed, and two release-asset copies
leave the ignore list. bundled_only.py now lists every copy of a listed
file, since the database keeps one path per content, and pipeline step
4d fails while one remains.
2026-10-20 10:17:08 +02:00
Abdessamad Derraz f71b00615d fix: keep the system loop's declared entries apart
The per-system lookup named its dictionary declared, which the file loop below reassigns to a list, so the second system of a platform page raised AttributeError.
2026-10-20 10:17:01 +02:00
Abdessamad Derraz f8e0b33314 feat: report a published export that fell behind
exports/ is committed but CI cannot rebuild it, since the originals it
patches live in the local cache. The native freshness check now runs
the export offline and compares it with the committed files, and
refresh_stale writes the export again after refreshing an original.
2026-10-20 09:28:14 +02:00
Abdessamad Derraz b6a1304893 feat: commit the corrected file of each platform
exports/ now holds the twelve platforms' own files, corrected, with a
report of every change, read by the site's corrected files page. Two
summaries are reworded for the page: an addition the format cannot
carry is counted as left out, and RetroPie names each skipped package
by sentence instead of joining reasons with semicolons. Its parser no
longer reads "e.g" as a file name.
2026-10-20 09:26:12 +02:00
Abdessamad Derraz 205ae5fadb fix: a dual-mode name reaches its libretro profile
Batocera, Recalbox and RetroBat launch pcsx2 standalone and as LRPS2.
The name matched the standalone profile's key, so lrps2, which lists
pcsx2 among its cores, was left out and its files never reached those
packs. A name in dual_mode_cores now also resolves to the libretro
profiles that claim it.
2026-10-20 09:21:48 +02:00
Abdessamad Derraz f84eadd954 fix: name the emulators of systems no profile ids
RetroArch, Lakka, RetroPie, EmuDeck and RetroBat group NAOMI and
Atomiswave as sega-dreamcast-arcade and the Enterprise 64 and 128 as
one system. No profile uses those ids, so the emulator cell was empty.
When no id matches, a platform page now names the emulators that read
a file the system lists. Archive members do not count.
2026-10-20 09:20:06 +02:00
Abdessamad Derraz ca13e6036d feat: publish the corrected platform files
export_native now writes into exports/ by default, and beside each
platform's corrected file a report naming every change, where the run
printed five per kind. A new site page lists each platform with its
original file, a download of the corrected one and every corrected,
filled or withheld value. Each platform page links to its row.
2026-10-20 09:20:02 +02:00
Abdessamad Derraz 71b59eb2a3 docs: rewrite half-dash asides in comments
Dropping the double-dash asides left 72 comments and docstrings with
half a dash glued to the next word ("shape -a size or a size range -is
all"). They now read as a colon, a comma or parentheses, and a prose
style test names any that comes back.
2026-10-20 09:17:10 +02:00
Abdessamad Derraz e424a35aee feat: install one emulator's files in one command
install.py --emulator NAME keeps what the named emulators read on the detected platform: their own files and the platform's files of the systems they run, as a pack built for them carries, without the systems no core is mapped to. Each emulator page now gives that command for Linux, macOS and Windows, beside the per-file links.
2026-10-20 09:09:05 +02:00
Abdessamad Derraz 7c94a58825 feat: tell libretro glue from emulator code
A libretro port carries the emulator's own sources beside its glue: pcem cites src/mem.c in the port, which is PCem's code. A citation in the port is labelled libretro glue when it is one of the libretro interface files, emulator code otherwise, and upstream emulator when it is in the original repository.
2026-10-20 09:05:57 +02:00
Abdessamad Derraz 274db1322f fix: the second extras pass reads installed marks
A file a core reads at several destinations is added again at each one by a second pass that worked from the raw profile entries, so files the emulator or the platform's package installs came back into the packs and manifests by that route: the installer's per-target replay disagreed with the builder on six files.
2026-10-20 08:12:03 +02:00
Abdessamad Derraz 954f213d42 fix: one id per card, and render the site locally
A copy filed under two consoles of one maker was listed twice with the same id, and a data directory listed the same bytes twice, so the site deploy refused 20 pages. The first card keeps the id. The pipeline only generated the pages, so it now renders them with mkdocs build --strict and runs validate_site as the deploy does. The resumed-speed test read the clock and failed on a slow runner, it now fixes the elapsed time.
2026-10-20 07:05:01 +02:00
Abdessamad Derraz 6df087e3e9 fix: a platform's own data trees leave its pack
The platform layer names data trees for its systems: retroarch.yml carries dolphin-sys and ppsspp-assets, and Lakka inherits them, while Lakka's core packages install both trees into the system directory. A profile records that with installed_on on its data directory, and the pack and the install manifest now leave those trees out for that platform.
2026-10-20 06:36:32 +02:00