Commit Graph
141 Commits
Author SHA1 Message Date
Abdessamad Derraz 2c76cc7df9 feat: read a search list as one need
A core that loads the first of several names (yabause: saturn_bios.bin,
sega_101.bin, mpr-17933.bin) needs one of them. Profiles wrote this as
required on the first name and optional on the others, so verify and
the gap report called saturn_bios.bin a missing required file while
sega_101.bin was held, and a set held only through a name marked
optional read as a need met by an optional file.

Model: an explicit group on the file entry, mirroring the platform
field. priority was not reused: it is a rank, read by slot.py, region.py
and truth.py inside a (system, region) slot, and that slot is not the
set. px68k-sdl ranks its IPL list and its font list each from 1 in one
system, pom2 ranks one list per machine model, DuckStation ranks 105
images across every region and takes any of them. A rank cannot say
where one list ends. priority stays the order inside a group.

Readers, through alternatives.settle_sets:
- verify --emulator and the emulator pack: a held member fills the
  set, the others leave the report, a set nobody holds is one gap
  under its first member naming the others.
- find_undeclared_files (verify platform, core complement of packs and
  manifests): same rule, a platform declaration or launcher counts as
  answering the set.
- cross_reference: excused members are set aside as other versions.
- slot.py: ranks compare members of one set only.
- archive members: a required set inside an archive is met by one.
- truth: a member gives no requirement of its own, one_of and
  one_of_required carry the set, and the exporters and the diff leave
  such a line's required as the platform wrote it.
- validate_schemas: members of a group share required.
2026-11-09 17:39:46 +01:00
Abdessamad Derraz eb8f60aef7 fix: write retrobat group and optional back
The export dropped the two fields Program.cs reads besides md5 and file:
an empty truth no longer handed RetroBat's list back as written.
2026-11-09 12:45:56 +01:00
Abdessamad Derraz 70f530516d chore: merge pack and resolver fixes 2026-11-07 00:06:15 +01:00
Abdessamad Derraz ed6643022d fix: write the cut md5 rule once
Six places read an md5 shorter than 32 digits, each with its own bound.
digest_verdict accepted any prefix down to one digit, while Batocera
compares the whole string (batocera-systems:1088). A cut value now names
a dump only when one held md5 starts with it (cut_md5_owner, read by the
scraper, the resolver and --from-md5), and every verdict compares whole
digests.
2026-11-06 15:18:38 +01:00
Abdessamad Derraz 6a5b384e02 fix: say when a RetroPie help is updated in part 2026-11-06 14:56:11 +01:00
Abdessamad Derraz 144cd46df1 fix: count one export line per addition path 2026-11-06 14:55:23 +01:00
Abdessamad Derraz 63be8cc3bc fix: find tab-indented batocera-systems entries
The entry pattern required four spaces of indent. EmuELEC indents 23 systems with one tab, so render appended each again whole and the published export carried 21 duplicated keys, the md5 fills only in the second copy. Any indent opens an entry now, and an entry written again keeps its own. The EmuELEC export is regenerated: one copy of each system, the amiga entry also gaining the eight WHDLoad Kickstarts the profiles added since 2026-10-22.
2026-11-06 08:39:40 +01:00
Abdessamad Derraz 39e6afbeb2 fix: a size kept in bios_db.json is not removed
When the truth accepts the BiosDB hash but none of its sizes fits, the MiSTer render keeps the entry's size beside the hash, and the report still counted a size removed. A size the truth does not state is no longer reported as changed.
2026-11-06 04:00:47 +01:00
Abdessamad Derraz 290c0b73ff fix: a container md5 leaves a zippedFile md5 alone
Batocera and KNULLI check a zippedFile item's md5 against the member inside the archive. A profile's container md5 for the archive would have replaced it as a correction. The item's md5 is now read as Recalbox's composite md5 is: neither compared with the container's nor overwritten by it.
2026-11-06 04:00:15 +01:00
Abdessamad Derraz d5e61308e1 fix: additions join the native id their cores read
A platform system spanning several native ids with no native_id filed every addition under the id that sorts first: RetroDECK's arcade wrote MAME device archives as FBNeo BIOS, RetroBat's Amiga put A500 kickstarts under amiga1200. An addition now joins the id whose declared files its cores read, and one the cores do not settle is counted and not written.
2026-11-06 03:57:39 +01:00
Abdessamad Derraz 3d0ee19bcb refactor: index profile entries by name and hash
_matching walked the reader's whole profile for each platform entry, 2,135 scans of MAME in one verify --all. Each profile is now indexed once per kind by name and by accepted hash. Over every registered platform, 10,524 lookups give the same answers as the scan, in 0.13 s instead of 2.42 s.
2026-11-05 19:37:27 +01:00
Abdessamad Derraz 4af0d70e4e fix: report md5 lists Recalbox and RetroDECK grow
Both formats write every accepted md5 when the truth extends the platform's list without contradicting it, but only Batocera answered written_list, so landed_changes reported nothing for a diff the maintainer would find. Both now name the list they write when it holds several values.
2026-11-05 19:36:02 +01:00
Abdessamad Derraz b90f8bb0b9 chore: merge tooling batch 2026-11-05 15:57:19 +01:00
Abdessamad Derraz 5ef7a9fabf refactor: hash each profile entry once 2026-11-05 11:57:08 +01:00
Abdessamad Derraz d7d6f3e412 fix: one hash mismatch rule for diff and export
diff_truth read sha1, md5 and crc32 and let one shared value settle the
entry, while export_native corrected each of the four fields on its own.
contradicted_hashes is now the single decision both read.
2026-11-05 11:50:09 +01:00
Abdessamad Derraz a434c1a8a0 fix: add a retropie file to its own system's list
Only the first BIOS clause of a help string was read, so naomi2.zip and
segasp.zip went into flycast's Dreamcast sentence. A file now joins the
clause whose names the platform files under its system, or is not added.
2026-11-05 11:43:54 +01:00
Abdessamad Derraz 4e5b48ddbe fix: keep the md5 a mister entry already holds
A BiosDB entry is one record: md5, size and url describe one dump. The
first md5 the truth accepts replaced an accepted one with no change
reported. The exporter now writes record_hash, as RomM and System.dat do.
2026-11-05 11:41:18 +01:00
Abdessamad Derraz 2ee777824a feat: name doubled destinations in exports
A platform list that declares one path for two files fails one of the
two checks on every install. The export report lists each declaration
with its hash and the emulators whose profile reads that file.
2026-10-24 02:42:04 +02:00
Abdessamad Derraz 5b083c45b5 feat: export md5 lists the pinned checker reads
The script exporters (Batocera, ROCKNIX, KNULLI, AmberELEC, EmuELEC)
read the checker of the pinned original: when checkBios hands an
item's md5 to a function that tests it with isinstance(..., list),
as md5Matches does since batocera.linux 6b08b873, an entry whose
emulator code accepts several dumps is written with all of them, and
an addition that needed several becomes writable. Items with altmd5
keep a string, since checkBios wraps them in a list of its own.

No cached original reads lists today, so every export is unchanged.
The report records a list as a list, counts the lists written, and
records a list the truth extends without contradicting it.
2026-10-23 08:06:58 +02:00
Abdessamad Derraz 0b4504992b feat: read batocera md5 written as a list
batocera.linux 6b08b873 lets a biosFiles item declare its md5 as a
list of strings, and checkBios accepts a file matching any of them.
The scrapers join the values with commas, as for Recalbox, and record
md5_list so the export writes the list back as a list, byte for byte.
2026-10-23 07:39:25 +02:00
Abdessamad Derraz 14a6518d49 fix: cite the reader that accepts the hash
A platform entry that declares a hash is now answered by the profile entries accepting it. Batocera's scph1001.bin (md5 dc2b9bf8, v2.0) cited DuckStation's bios.cpp:36, the v2.2 line. A same-named entry is kept, marked by name, only when none accepts the hash and its own hashes do not contradict it. Categories are unchanged: verify --all prints the same report.
2026-10-23 02:58:43 +02:00
Abdessamad Derraz 33041b6bd1 feat: find a file by any hash, out of the search
Hashes, link lists, install commands, member tables and long reference
tables leave the site search: the index drops from 21.0 MB (4.88 MB
gzip) to 6.8 MB (1.33 MB gzip). Names and prose stay searchable.

files/?hash= takes a SHA1, MD5, CRC32 or SHA256 and reads one shard of
api/v1/lookup/ (about 22 KB gzip, was 675 KB) listing the collection
card, the platforms that declare the value and the profiles that check
it. files/?sha1= still opens the card, and a sha1- anchor missing from
a page is looked up there, so a card can move between pages.

Excluded blocks are section elements: the search plugin compares
elements by tag, so a div ended at the first nested div.
2026-10-22 12:46:40 +02:00
Abdessamad Derraz baf38ffbee refactor: compute coverage once, index proofs
EntryCategories asked every core in turn: 3.8 million proofs for 7,430
entries. ProofIndex looks each key up once, and verify --all --json is
byte-identical (24 s to 17 s). generate_site and generate_readme now share
one coverage, cached in tmp/coverage-cache keyed by its inputs, so the
README step drops from 25 s to 3 s.
2026-10-22 12:23:26 +02:00
Abdessamad Derraz cd7d993cac feat: settle entries by their system's cores
A system that lists the cores it runs is settled by those cores alone, so an unprofiled core of another system no longer leaves its entries open. Readers whose core runs the entry's system are named first, and cores that only share the file name are listed as such.
2026-10-22 06:30:56 +02:00
Abdessamad Derraz a75ea05b86 feat: document what settles each category
Each missing platform entry now carries its evidence: the profile entries that read it with their source reference, the cores of its system examined without finding a reader, or why the question stays open. verify --verbose prints it, --json carries it and the platform pages show it.
2026-10-22 05:43:06 +02:00
Abdessamad Derraz f66adfdcbc feat: categorize missing platform entries
A platform list can name a game's data files or a file no core it runs opens. Verify gives each missing entry a category read from the profiles of the platform's cores: firmware, game data or unread. Only firmware gaps carry a severity and fail --strict; the others are counted on their own line, and the pack check no longer counts them as pack errors. Missing firmware stays a conformance error.
2026-10-22 05:38:57 +02:00
Abdessamad Derraz 2bdac466a9 feat: add knulli, amberelec and emuelec platforms
Each OS publishes Batocera's systems dict (knulli-systems, and
batocera-systems at AmberELEC's newest prerelease commit and EmuELEC v4.8),
read with the shared parser. Cores come from each OS's own ES list.
AmberELEC and EmuELEC declare neogeo/ and ports/ beside bios/ from
/storage/roms, so their packs extract there and core_destination puts
core files under bios/.
2026-10-22 03:42:20 +02:00
Abdessamad Derraz fd1d1b7b96 chore: merge round 2 fixes (pins)
# Conflicts:
#	install.ps1
#	install.sh
2026-10-21 10:31:09 +02:00
Abdessamad Derraz ccf61b96d7 fix: report names a RetroPie sentence gains
flycast now reads naomi2.zip, which the RetroArch list RetroPie inherits
already declares. The render added it to the flycast help sentences but
landed_changes counted additions only for entries the platform layer
lacks, so the report missed a name the diff carries. names_anew lets an
exporter whose unit is not the entry say so.
2026-10-21 10:26:04 +02:00
Abdessamad Derraz 83a4b21d30 fix: keep platform hashes cores disagree on
When two cores declare different values for one file, the truth keeps the first and marks the entry hash_conflict. Its hashes pair an entry and never replace the platform's: armsx2's GameIndex.yaml no longer overwrites System.dat's line for LRPS2.
2026-10-21 09:57:20 +02:00
Abdessamad Derraz f7b758a41d fix: patch batocera entries past commented items
_patched_entry read '#{ ... }' lines as live items, found no match in the model and gave up, so atarist was written again whole (52 lines changed for 3 additions). A commented item is now left as written.
2026-10-21 09:52:34 +02:00
Abdessamad Derraz 5ec1208a83 fix: write romm keys with a bare file name
RomM lists the platform firmware folder without recursing and looks each file up as slug:file_name (models/firmware.py, base_handler.list_files at 5.4.0). An addition named with a directory (fuse/48.rom) is not written.
2026-10-21 09:51:32 +02:00
Abdessamad Derraz 4205bc2233 fix: count one retrobat addition per path
The render writes one biosFiles item per path, as checkBios reads one md5 per item. A second addition at a held path was dropped from the file and still counted added (78 announced, 75 written).
2026-10-21 09:50:46 +02:00
Abdessamad Derraz 28435230a7 fix: read each named core's own entry in recalbox
An element's mandatory flag came from the one truth entry its md5 paired
with. saturn/saturn_bios.bin names yabause, yabasanshiro and kronos,
which require it, and was paired with the beetle_saturn entry, which
does not. Each named core's entries for the file, found by name, now
decide, except a namesake elsewhere in the same tree (bluemsx KANJI.rom
under Shared Roms is not the one under MSX2+ - European).
2026-10-21 05:39:17 +02:00
Abdessamad Derraz d9548e3522 fix: count a recalbox element kept as written
An element whose md5 the truth contradicts without naming another is
written back whole. It was counted as an entry the format cannot keep.
It now counts as kept, and the md5 it keeps is named as withheld.
2026-10-21 05:14:44 +02:00
Abdessamad Derraz 4a6b654bdc fix: keep a romm entry whole without a size
RomM compares file_size_bytes with the entry's size before any hash. A
new hash the truth gives no size for went out beside the old size and
was reported as a size removal. The entry now stays as written and the
correction is withheld and named.
2026-10-21 05:13:46 +02:00
Abdessamad Derraz 4670e0442e fix: withhold a mister hash only beside a size
An entry that states no size has none to contradict: the hash is written
as before.
2026-10-21 05:13:46 +02:00
Abdessamad Derraz d48e81861c fix: keep a mister entry whole without a size
Downloader rejects an entry without an integer size (db_entity.py:187)
and compares it after each fetch (fetch_file_worker.py:127). A new hash
the truth gives no size for is withheld and named, not written beside
the old size nor reported as a size removal.
2026-10-21 05:12:02 +02:00
Abdessamad Derraz 2e29b3be03 fix: read recalbox mandatory per named core
An element lists its cores. Its mandatory flag now follows whether those
cores need the file, and stays as written where none of them reads it.
cdi/cdibios.zip keeps mandatory=false (same_cdi does not read it), the
two amiberry kickstarts keep theirs, saturn_bios.bin becomes optional
for the three HLE cores it names.
2026-10-21 05:10:54 +02:00
Abdessamad Derraz fbf7cb191b fix: keep recalbox composite md5 on archives
Bios.cpp:64-68 hashes a .zip with Zip::Md5Composite, the md5 of its
members in name order. A profile's md5 on an archive is the container's,
so it neither pairs with, corrects nor fills that field, and an archive
addition with only a container md5 is left out.
2026-10-21 05:08:45 +02:00
Abdessamad Derraz f796155e1a fix: count only the names retropie help gains
The report listed every file a profiled core reads as added (1360),
while the export appends one name to one scriptmodule.
2026-10-21 05:05:00 +02:00
Abdessamad Derraz 6d38e597db fix: keep a RetroPie list's comma before its and
quasi88 writes its list "N88KNJ2.ROM, and N88SUB.ROM". Extended with
N88EXT0.ROM, which QUASI88kai reads, it came back as "N88SUB.ROM and
N88EXT0.ROM": every list was rewritten in one idiom. A list that puts a
comma before its final and keeps it.
2026-10-20 22:38:32 +02:00
Abdessamad Derraz 540374b40d fix: count a launcher's names as read
The removal candidates looked for a reading in the profiles only, so a
name the platform's own launch code opens, links or copies was listed
as read by no core: Batocera's DuckStation generator walks 22 names
(duckstationGenerator.py:501-515). Every name a launcher record names
is now a reading on that platform. Recalbox's unexplained entries drop
from 20 to 13, RetroBat's from 24 to 20, EmuDeck's from 25 to 22.
2026-10-20 22:20:42 +02:00
Abdessamad Derraz a9c175d322 chore: regenerate the updated platform files 2026-10-20 20:44:42 +02:00
Abdessamad Derraz 479ada29fe docs: word removal candidates by what was searched
"No core opens it" claimed more than the search shows: what was
checked is that no profile of a core the platform runs names the
entry by name, path or hash. The reason and the page sentence now
say that.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 545e3f3ba1 feat: record each export change before and after
The page named the entries a change touched and never the values, so
a maintainer could not check a correction. The report now carries
changes, one object per change with kind, entry, field, old and new,
read from the platform entry and from the value the render writes
(written() per format: record value for System.dat and RomM, one md5
where a single one is checked). The existing label lists are derived
from it. The page writes "md5 X becomes Y" and names each required
flag with its new value. A test parses the original and updated
System.dat and es_bios.xml and checks the list equals the diff.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 5cba005d67 fix: keep the platform's hash order when unchanged
Where the truth names no value the platform lacks, hashes() still put
the truth's values first and rewrote the list in another order:
Recalbox's msx/CARTS.SHA md5 attribute came out with its two values
swapped, a diff line that changes nothing the frontend checks. The
platform's own list now stands when it already holds every value.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 0b4f98ce79 feat: report entries no examined core opens
The export keeps every entry a platform declares. An entry the truth
does not pair was still read in 1189 of 2776 cases (alias, hash, a
type: directory tree, a core outside the resolved set), so absence of
a pairing proves nothing. Each report now lists, under
removal_candidates, the unpaired entries that no profile of the cores
the platform runs answers by name, path tail, hash, directory or data
directory, with the cores examined and a reason. A core the platform
names without a profile, or a ROM set under a core that opens sets by
name, leaves the verdict undetermined. The exports page states the
count and lists the entries per platform.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 7e0b3a08b9 fix: keep System.dat lines whose values agree
The rom line renderer upper-cased every crc, so Dinothawr.zip's
683ed4ad, the one lower-case crc libretro writes, changed with no
correction behind it. A line whose values match ours now keeps its
bytes, a corrected crc follows the case of its own line, and an added
line keeps the file's upper case. An empty truth hands System.dat back
byte for byte for RetroArch and Lakka.
2026-10-20 19:39:36 +02:00
Abdessamad Derraz eba5f1dd62 fix: patch Recalbox's attributes, not its elements
A changed element was rendered again, losing its alignment and the
place of its attributes, and an element whose md5 list was only
reordered (neogeo.zip, the same 24 digests) was rewritten too: Recalbox
accepts any digest of the list (Bios.cpp:109-130), so the order is not
content. A changed attribute is now replaced where it stands, one
dropped goes with its space, one added goes before />, and each system's
opening tag keeps its own indentation. An empty truth gives back the
cached es_bios.xml byte for byte.
2026-10-20 19:36:09 +02:00