common.py had grown to 1833 lines by accumulation. Six coherent pieces
move out - untrusted parsing, digests, archives, generated artefacts,
release assets, dump catalogues - and common.py re-exports them, so the
sixty existing import sites keep working and migrating them stays
optional.
The site build is now reproducible, which is what made the move
checkable. It deleted its generated directories first, so every page was
new and write_if_changed had no earlier version to compare against: a
deploy republished six hundred pages for the clock alone. Directories
are swept instead, a page is removed only once nothing produces it, and
the body pass compares against the body of the file on disk rather than
against the decorated page. Two consecutive builds on the same inputs
now produce identical bytes; before, 1034 files differed.
A filename-agnostic core accepts any name for its BIOS, so the builder
scans the directory holding the candidates. It picked that directory
from a first-hit lookup by name, the one piece of evidence that lands
in another emulator's tree: five files answer to GameIndex.yaml and one
belongs to an Android package, rom1.bin is a PS2 ROM and a Roland
SC-55 ROM. One wrong match became every file beside it, flattened into
the BIOS root of platforms that do not run that emulator: 45 files in
the Recalbox pack, 170 in the RetroArch one.
Four things decide it now. A destination and the repo layout meet on a
tail, so the path index is tried from the longest tail down and never
to the bare filename; that alone corrects seven files, among them the
Japanese GameCube slot, which held the US dump. A seed has to declare
the shape it is looking for, since no shape means the whole directory.
An ambiguous name needs the profile's other files to agree before the
scan walks anywhere. What the scan emits carries the SHA-1 it selected,
so packing never resolves it by name again.
The builder and the coverage report each grouped their own candidates
before asking which regional alternatives to withdraw. The builder
grouped the platform files and the core extras; the report grouped the
platform files alone, and keyed them on an unsanitized destination.
So a region run withdrew 73 files from a recalbox pack while the report
withdrew 14, and described the other 59 as covered by a pack that would
not carry them.
platform_region_groups builds the grouping once and both sides read it.
The extras it returns are keyed by emulator, name and path: Dolphin
declares three IPL.bin that differ by path alone, and a name-keyed map
withdraws the wrong one. Manifests are byte-identical before and after.
Decorated site pages carry the generation stamp twice: once as the
markdown footer and once as a rendered element. write_if_changed knew
only the first, so every page was rewritten on every run for the clock
alone.
The comparison is what makes the deploy-site freshness guard a real
staleness check rather than a guaranteed failure, and it had no tests.
Loading the emulator profiles is the most expensive step of every
command here, and all forty call sites used the pure-Python scanner
while libyaml sat unused in the same wheel. One shared yaml_load picks
the C loader when pyyaml ships it: the 375 profiles parse in 0.18s
instead of 1.39s, and verify --platform retroarch drops from 2.17s to
0.73s. The loader class is the same restricted one safe_load uses.
es_bios.xml was parsed straight from the network while install.py
already refused a document declaring entities; both now share one
guard. Scrapers reach it through a single path bootstrap in the
package rather than two ad-hoc ones.
A declared hash that the local dump contradicts is not one situation. An
existence platform never reads the bytes, so withholding the file lets an
upstream list error remove something the frontend would have loaded; a
hash platform would reject it, so shipping it is pointless.
The mode now decides, at every point that had an opinion: pack building,
core complement, emulator packs, manifests, conformance and
_intentional_hash_exclusion. verify.find_undeclared_files follows, since
verify and generate_pack must agree file for file.
Also here: resolution reports which evidence matched rather than a flat
"exact", a path or filename can no longer override a declared hash, and
safe_extract_zip treats a Windows backslash as the separator it is
instead of refusing the archive.
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).
Run isort --profile black: normalize import ordering across all files.
Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.
3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
9 new wiki pages: getting-started, faq, troubleshooting,
advanced-usage, verification-modes, adding-a-platform,
adding-a-scraper, testing-guide, release-process.
Updated architecture.md with mermaid diagrams, tools.md with
full pipeline and target/exporter sections, profiling.md with
missing fields, index.md with glossary and nav links.
Expanded CONTRIBUTING.md from stub to full contributor guide.
Filter .old.yml from load_emulator_profiles, generate_db alias
collection, and generate_readme counts. Fix BizHawk sha1 mode
in tools.md, fix RetroPie path, fix export_truth.py typos.
write_if_changed in common.py compares content after stripping
timestamps (generated_at, Auto-generated on, Generated on).
applied to generate_db, generate_readme, generate_site.
eliminates timestamp-only diffs in database.json, README.md,
mkdocs.yml, and 423 docs pages.
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
Add Casio PV-2000 BIOS entry (pv2000.zip) to MAME and MESS profiles and update system lists/counts. Add Funtech Super A'Can BIOS entries (supracan.zip and umc6650.zip) with ROM contents to mamemess. Simplify and condense Vita3K emulator profile (rename fields, update profiled_date, add PSVUPDAT.PUP and optional PSP2UPDAT.PUP file entries, and clarify install/partition behavior). Bump database generated_at timestamp and add a system alias mapping "psvita" -> "sony-playstation-vita" in scripts/common.py.
Regenerate database.json and update README counts/timestamps; add and normalize numerous BIOS entries and hashes. Key changes: update generated_at timestamp and system count (355→357) in README; add OpenBIOS / HLE fallback and additional aliases to beetle_psx, include beetle_psx core name and profiled_date update; add laseractive to ares systems; adjust atari800 systems and source_ref line numbers; mark dinothawr as a system and expand its note; update gsplus upstream/profile date, add apple-iie system and tweak source_refs; add pcsx2 core to lrps2; refresh mame profiled_date and add multiple systems and BIOS root sets. Miscellaneous script changes and other JSON normalization to reflect newly discovered/merged BIOS files.
resolve_local_file step 2 (pure MD5 lookup) now verifies that the
found file's name matches the requested name or is a .variants/
derivative. Prevents serving wrong files when an unrelated file
shares the same MD5 in the index (e.g. spi.zip returned for
a7ports.zip because RetroDECK expected an MD5 we don't have).
Abdessamad Derraz