find_undeclared_files decided per entry whether a core requirement can
be a gap, through a chain that mixed two skips whose difference is easy
to lose. Some record the requirement as settled so no other profile
reconsiders it; the rest leave it open, because the same file can be
libretro-only in one profile and standalone-only in another and the key
carries no emulator.
The chain returns a named verdict now, complexity 60 to 45. Twelve tests
cover the verdicts and two more cover the distinction end to end: a
standalone-only entry seen first must not answer for the profile that
needs the file. Collapsing the two skips into one passes every other
test in the suite and fails that pair.
The markdown report wrote to a directory named after the tooling that
happened to produce it. It takes --report-dir now, defaulting to
reports/, so nothing in the tree names anything but the project.
Em-dashes replaced throughout the sources and tests, rephrased rather
than swapped for a comma where the dash carried an apposition.
Four sites still compared the verification mode to a literal after the
module owning that policy existed. One of them mattered: an unrecognised
mode fell through to MD5 verification while compute_severity was scoring
it as existence, so a typo in a platform YAML produced a report whose
checks and severities described different platforms.
The mode is normalized once per run and the consumers ask for what they
need. A test reads the sources and fails on a literal comparison, so the
next consumer cannot quietly grow a fifth copy.
The builder and the coverage report each grouped their own candidates
before asking which regional alternatives to withdraw. The builder
grouped the platform files and the core extras; the report grouped the
platform files alone, and keyed them on an unsanitized destination.
So a region run withdrew 73 files from a recalbox pack while the report
withdrew 14, and described the other 59 as covered by a pack that would
not carry them.
platform_region_groups builds the grouping once and both sides read it.
The extras it returns are keyed by emulator, name and path: Dolphin
declares three IPL.bin that differ by path alone, and a name-keyed map
withdraws the wrong one. Manifests are byte-identical before and after.
The gap analysis answered from the name index, so a core extra whose
local copy contradicts its declared hash counted as held. Under a
digest mode the builder drops exactly that file, so the coverage report
described a pack that would not contain it: seven files across
Batocera, Recalbox and RetroBat.
An entry that states what its content should be is now resolved by
content; the name still answers for entries that declare nothing to
check against, and existence mode is unchanged because there the
frontend never opens the file and the pack does carry it.
The per-emulator report counted an entry nobody can supply -- a
per-user key, a slot the user fills, a dump that was never made -- as
plainly missing. fpse-ng read 14 missing when twelve were documented as
unobtainable, which invites the wrong repair: dropping the flag,
deleting the entry, or chasing a vendor's whole install tree.
They are listed with the reason the profile records rather than hidden,
and the summary counts them apart. Platform reports are untouched.
The per-emulator report captured the status resolve_local_file returns
and then ignored it: any non-empty path became OK. An entry whose only
candidate contradicts its declared hash therefore read as covered, so a
same-named file from another system stood in for one the collection
does not hold. 36 entries across 14 profiles were affected, among them
config.ini, ROM and rom2.bin, names that collide across systems.
Emulator validation still runs first, since it names the field that
disagrees; the resolution status is consulted only when validation had
nothing to say.
verify.py and generate_pack.py must reach the same verdict on the same
file, and CLAUDE.md calls any divergence critical, but each spelled out
mode == 'existence' in its own words. Both now ask nativemode whether
the frontend reads the file's bytes, which is the one fact the rest
follows from, and a test holds the two answers together.
The BaseScraper contract the wiki asks contributors to implement had no
caller anywhere, so nothing proved compare_with_config, has_changes or
test_connection still worked.
A declared hash that the local dump contradicts is not one situation. An
existence platform never reads the bytes, so withholding the file lets an
upstream list error remove something the frontend would have loaded; a
hash platform would reject it, so shipping it is pointless.
The mode now decides, at every point that had an opinion: pack building,
core complement, emulator packs, manifests, conformance and
_intentional_hash_exclusion. verify.find_undeclared_files follows, since
verify and generate_pack must agree file for file.
Also here: resolution reports which evidence matched rather than a flat
"exact", a path or filename can no longer override a declared hash, and
safe_extract_zip treats a Windows backslash as the separator it is
instead of refusing the archive.
scph3000.bin v2.1J and scph3500.bin v2.2J already existed under
different primary names (scph3500.bin and scph5000.bin respectively).
Add .variants/ entries so by_name resolves both filenames.
verify_single_emulator now calls _find_best_variant on hash mismatch,
matching the platform-level verification path.
generate_db: add by_sha256 index for O(1) variant lookup.
verify: _find_best_variant uses indexed sha256 instead of O(n) scan.
validation: check_file_validation returns (reason, emulators) tuple,
attributing mismatch only to emulators whose check actually failed.
beetle_psx: remove incorrect size field for ps1_rom.bin (code does
not validate size, swanstation is sole size authority).
_find_best_variant now searches by hash (md5, sha1, crc32, sha256)
across the entire database instead of only by filename. Finds
variants stored under different names (e.g. eu_mcd2_9306.bin for
bios_CD_E.bin, scph1001_v20.bin for scph1001.bin).
verify_entry_existence now also calls _find_best_variant to
suppress discrepancies when a matching variant exists in the repo.
Reduces false discrepancies from 22 to 11 (4 unique files where
the variant genuinely does not exist in the repo).
find_undeclared_files was enriching declared_names with DB aliases,
filtering core extras that were never packed by Phase 1 under that
name. Pass strict YAML names to _collect_emulator_extras so alias-
only files (dc_bios.bin, amiga-os-310-a1200.rom, scph102.bin, etc.)
get packed at the emulator's expected path. Also fix truth mode
output message and --all-variants --verify-packs quick-exit bypass.
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).
Run isort --profile black: normalize import ordering across all files.
Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.
3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
Replaces mode: standalone hack with load_from: save_dir on Panda3DS
files. The load_from field documents which libretro directory callback
provides the base path (system_dir default, save_dir, content_dir).
Pack generator and cross-reference skip files not targeting system_dir.
Files with explicit path: null are UI-imported (Dolphin NAND,
Hatari cartridge) and not resolvable by pack placement. Skip
them in find_undeclared_files and cross_reference. Also add
desc.dat (SDLPAL fan-made descriptions) to data/. 149/149 OK.
_build_supplemental_index scans both data/ directories and
contents of bios/ ZIP files. Eliminates 197 false positives
where files existed inside archive ZIPs (neogeo.zip, pgm.zip,
stvbios.zip, etc.) but were counted as missing. True missing
drops from 645 to 448.
_find_in_repo and _name_in_index now scan data/ in addition to
bios/ via database.json. Eliminates 129 false positives from
game data migrated to data/ (OpenTyrian, ScummVM, SDLPAL, Cave
Story, Syobon Action). True missing: 782 -> 653.
Group archived files by archive unit in find_undeclared_files instead
of reporting individual ROMs. Add path-based fallback for descriptive
names (e.g. "SeaBIOS (128 KB)" resolves via path: bios.bin). Update
_collect_extras to use archive name for pack resolution. Regenerate
database with new bios files. 6 new E2E tests covering archive
in_repo, missing archives, descriptive names, and pack extras.
batch re-profiled nekop2 through pokemini. mupen64plus renamed to
mupen64plus_next. new profiles: nes, mupen64plus_next.
validation functions (_build_validation_index, check_file_validation)
consolidated in common.py — single source of truth for verify.py
and generate_pack.py. pipeline 100% consistent on all 6 platforms.
post-generation verification: reopen each ZIP, hash every file,
check against database.json. inject manifest.json inside each pack
(self-documenting: path, sha1, md5, size, status per file).
generate SHA256SUMS.txt alongside packs for download verification.
validation index now uses sets for hashes and sizes to support
multiple valid ROM versions (MT-32 v1.04-v2.07, CM-32L variants).
69 tests pass, pipeline complete.
Abdessamad Derraz