docs: archive recipes, build command, zlib finding

This commit is contained in:
Abdessamad Derraz committed 2026-11-06 18:38:00 +01:00
1 parent b7645f70ad
commit 91991774da
3 files changed
+102 -18

No files matched your search

+7 -1
View File
@@ -30,7 +30,8 @@ platforms/ one YAML config per platform (scraped from upstream)
_data_dirs.yml data directory definitions (Dolphin Sys, PPSSPP...)
targets/ hardware target configs + _overrides.yml
provenance/ dump-catalog snapshots (redump, no-intro, tosec)
recipes/ romset recipes read from MAME and FBNeo DATs
recipes/ romset recipes read from MAME and FBNeo DATs, and
archives.json, the recipe of every archive held
exports/ each platform's own file, updated, with a report of the changes
measurements/ records of real runs, one JSONL file per emulator
scripts/ all tooling (Python and pyyaml, jsonschema for schema checks)
@@ -318,6 +319,11 @@ How files enter `bios/`, and why.
An upstream romset whose MD5 a platform pins stays verbatim in `.variants/`.
A member renamed by hand in an existing archive gives bytes that match no
distributed romset and that nobody can regenerate.
- **Every archive has a recipe.** `recipes/archives.json` records the members
of each archive the collection holds, by sha1, and whether the TorrentZip
build gives its bytes (`exact`) or only its members (`canonical`), so the
collection can rebuild it from the ROMs it holds
([Archive recipes](tools.md#archive-recipes)).
- **A pack carries what the user must obtain.** A file the emulator installs
itself (`bundled: true`) is not kept once nothing else needs it, and step 4d
of the pipeline fails on such a copy. Its sha1 and size are written into the
+14 -2
View File
@@ -52,7 +52,7 @@ Every module in `tests/`, by the area it guards.
| Area | Fixtures | Modules |
|------|----------|---------|
| Core regression suites | synthetic | `test_e2e`, `test_audit_regressions`, `test_tool_messages` |
| File resolution and identity | synthetic + collection | `test_adler32_resolution`, `test_agnostic_scan`, `test_agnostic_scan_mode`, `test_agnostic_scope`, `test_agnostic_substitute`, `test_archive_choice`, `test_archive_recipes`, `test_bios_companions`, `test_bundled_identity`, `test_casefold_cache`, `test_db_aliases`, `test_declared_names`, `test_derivations`, `test_directory_entries`, `test_extra_identity`, `test_foreign_tree`, `test_foreign_tree_names`, `test_frontend_digest`, `test_game_data_homonyms`, `test_hashing`, `test_identity`, `test_maker_folder_names`, `test_md5_pack_context`, `test_name_tiebreak`, `test_nec_filing`, `test_np2_fonts`, `test_owner_resolution`, `test_path_suffix`, `test_resolution_evidence`, `test_sega_filing`, `test_shipped_content`, `test_twin_declaration_status`, `test_twin_declarations`, `test_validated_choice`, `test_variant_choice`, `test_variant_identity`, `test_zipped_member` |
| File resolution and identity | synthetic + collection | `test_adler32_resolution`, `test_agnostic_scan`, `test_agnostic_scan_mode`, `test_agnostic_scope`, `test_agnostic_substitute`, `test_archive_build`, `test_archive_choice`, `test_archive_recipes`, `test_bios_companions`, `test_bundled_identity`, `test_casefold_cache`, `test_db_aliases`, `test_declared_names`, `test_derivations`, `test_directory_entries`, `test_extra_identity`, `test_foreign_tree`, `test_foreign_tree_names`, `test_frontend_digest`, `test_game_data_homonyms`, `test_hashing`, `test_identity`, `test_maker_folder_names`, `test_md5_pack_context`, `test_name_tiebreak`, `test_nec_filing`, `test_np2_fonts`, `test_owner_resolution`, `test_path_suffix`, `test_resolution_evidence`, `test_sega_filing`, `test_shipped_content`, `test_twin_declaration_status`, `test_twin_declarations`, `test_validated_choice`, `test_variant_choice`, `test_variant_identity`, `test_zipped_member` |
| Verification, cross-reference and profile checks | synthetic + profiles | `test_bios_layout`, `test_cache_independent_counts`, `test_case_insensitive_platforms`, `test_checks_have_values`, `test_core_file_fills_declaration`, `test_core_size_refusal`, `test_cross_reference_sources`, `test_crypto_aes`, `test_crypto_unique_data`, `test_crypto_verify`, `test_entry_categories`, `test_file_consumers`, `test_fileless_audit`, `test_flycast_layout`, `test_gap_settlement`, `test_hidden_extras`, `test_launcher_aliases`, `test_launcher_layer`, `test_native_mode`, `test_no_case_collisions`, `test_path_conflict_status`, `test_path_null_copies`, `test_profile_checks`, `test_profile_names`, `test_profile_paths`, `test_profile_repositories`, `test_romm_hash_rule`, `test_shared_groups`, `test_standalone_links`, `test_standalone_mode`, `test_undeclared_dedup`, `test_unplaced_extras`, `test_validation_scope`, `test_verify_all_dir`, `test_verify_data_dir_notice`, `test_verify_emulator_evidence`, `test_verify_listing`, `test_verify_pack_choice`, `test_verify_release_asset` |
| Packs and manifests | synthetic + real packs | `test_artifact_lock`, `test_atomic_artifacts`, `test_batocera_cores`, `test_core_data_trees`, `test_deterministic_zip`, `test_emulator_pack_counts`, `test_finalize_pack_names`, `test_large_file_cache`, `test_libretro_layout`, `test_manifest_entry`, `test_minui_platform`, `test_no_game_data`, `test_pack_conformance_dirs`, `test_pack_core_extras`, `test_pack_counts`, `test_pack_destinations`, `test_pack_failures`, `test_pack_file_names`, `test_pack_grouping`, `test_pack_integrity`, `test_pack_member_identity`, `test_pack_readme`, `test_platform_inheritance`, `test_platform_keys`, `test_release_packs`, `test_required_only_archives`, `test_required_only_twins`, `test_save_root`, `test_split_extras`, `test_split_names`, `test_split_pack`, `test_system_pack_extras`, `test_system_pack_parity`, `test_torrentzip`, `test_which_pack` |
| Slots and regions | synthetic + profiles | `test_region`, `test_region_agnostic_scan`, `test_region_groups`, `test_region_held`, `test_region_platform_index`, `test_region_required_only`, `test_slot`, `test_slots`, `test_slots_shipped_file` |
@@ -188,9 +188,21 @@ module level, so nothing leaves the machine.
### Tests that read the working tree
Three modules assert on real repository data instead of fixtures. Each skips
Four modules assert on real repository data instead of fixtures. Each skips
when its input is missing, so a partial checkout still runs green.
**test_archive_build.** Reads `database.json` and `recipes/archives.json` and
fails when an archive has no recipe, when a recipe outlives its archive or
names another container, or when a recipe member is held nowhere in the
collection, neither as a loose file nor inside any archive. That list is the
acquisition list, printed by the failure, and it is empty today. With `bios/`
present it also holds each recipe against the archive's central directory:
names in order, sizes and crc32. Its fixture classes build an exact and a
canonical archive, refuse a mismatch without leaving a file, report a member
held nowhere as impossible, and never claim an exact build under zlib-ng.
After adding or changing an archive, `python scripts/romset_recipes.py
--capture` (pipeline step 1c) brings the recipes back in line.
**test_torrentzip.** Rebuilds real MAME romsets through the TorrentZip builder
and asserts the output is byte-identical, which is what keeps arcade ZIP hashes
stable across rebuilds.
+81 -15
View File
@@ -41,7 +41,7 @@ Pipeline steps:
| 0 | Validate profiles and platform YAMLs against the schemas | - (no `jsonschema` installed) |
| 1 | Generate database | - |
| 1b | Dump-catalog coverage report | - (offline, reads `provenance/`) |
| 1c | Romset recipes: identify archives, list rebuild targets | - (offline, reads `recipes/`) |
| 1c | Romset recipes: identify archives, list rebuild targets, then `--capture` writes a recipe for every archive (`recipes/archives.json`) | - (offline, reads `recipes/`) |
| 2 | Refresh data directories | `--offline` |
| 2a | Refresh MAME BIOS hashes | `--offline` |
| 2a2 | Refresh FBNeo BIOS hashes | `--offline` |
@@ -1372,10 +1372,11 @@ before a release (wiki/release-process.md). Its lines stay in
| `sitesources.py` | Record which cited source paths each pinned upstream tree holds, for the site's source links (`--fetch` reads a tree no cache holds) |
| `site_insights.py` | What readers look for and do not find, from totals only: searches that show the site but are rarely clicked, file names searched and whether the collection holds them, site searches that found nothing, broken addresses, "Did this page help?" answers, GitHub traffic and release downloads (`--days`, `--skip gsc,goatcounter,github`, `--output`). Reads Search Console, GoatCounter and GitHub with credentials kept outside the repository. [What the site counts](privacy.md) |
| `validate_site.py` | Validate rendered metadata, heading order, image alternatives, JSON-LD, local resources, links and fragments, and the page size and search index budgets |
| `romset_recipes.py` | Identify which emulator version an arcade archive matches, and rebuild a pinned archive from ROMs already held |
| `romset_recipes.py` | Identify which emulator version an arcade archive matches, rebuild a pinned archive from ROMs already held, record a recipe for every archive the collection holds and build any of them from the members it holds (see [Archive recipes](#archive-recipes)) |
| `archive_build.py` | The archive recipes behind `romset_recipes.py --capture`, `--build` and `--build-all`: capture, member lookup by sha1, verified build |
| `scraper/romset_dat_importer.py` | Fetch and import per-set recipes from MAME `-listxml` or FBNeo DATs into `recipes/`. `--fetch` alone takes the newest MAME release, and FBNeo DATs are refreshed by git blob sha. `--fetch <revision>` reads FBNeo at that tag, branch or commit |
| `deterministic_zip.py` | The fixed date pack members carry. Its archive rebuild is no longer called by the pack, which copies each ZIP byte for byte |
| `torrentzip.py` | Build TorrentZip archives for MAME/FBNeo ROM sets (archive bytes depend only on contents) |
| `torrentzip.py` | Build TorrentZip archives for MAME/FBNeo ROM sets (archive bytes depend only on contents, with stock zlib), streaming one member at a time |
| `crypto_verify.py` | 3DS RSA signature and AES crypto verification |
| `sect233r1.py` | Pure Python ECDSA verification on sect233r1 curve (3DS OTP cert) |
| `check_buildbot_system.py` | Detect stale data directories by comparing with buildbot |
@@ -1410,7 +1411,7 @@ Further options, by script:
| `release_record.py` | `--install-dir` (manifests the archives are held against), `--platforms-dir` (registry the full pack names are rebuilt from: any other pack in the directory is refused), `--output` |
| `release_notes.py` | `--whats-new FILE` (required, appended as written), `--record`, `--platforms-dir`, `--output` |
| `restore_large_files.py` | `--cache` (downloaded assets), `--gitignore`, `--db` |
| `romset_recipes.py` | `--dat-recipes PATH`, a snapshot or a directory of them (default `recipes/`) |
| `romset_recipes.py` | `--dat-recipes PATH`, a snapshot or a directory of them (default `recipes/`). `--archives FILE` (default `recipes/archives.json`), `--output-dir DIR` for `--build` and `--build-all`, `--discard` deletes each build once verified |
| `split_pack.py` | `--max-size 1900M`, the ceiling per part (K, M, G, default 2G) |
| `torrentzip.py` | `--check ZIP` reports conformance, `--rebuild ZIP -o OUT` normalizes an archive |
| `validate_pr.py` | `--changed` takes the BIOS files the branch changes, a folder named checks every file under it, `--place` copies each recognised file from outside `bios/` to its folder and declared name (or its `.variants/` slot), `--markdown`, `--json` |
@@ -1501,18 +1502,28 @@ check compares with.
Snapshots land in `recipes/`, not `provenance/`: the latter holds dump catalogues
and a recipe is not one. Versions accumulate rather than replace each other,
because a platform pins the archive of whichever version its list was built
against. Only sets a profile or platform references are kept, plus the sets
holding a dump-catalog entry the collection lacks (crc32 and size, and sha1
when both sides carry one), so `provenance_report.py --missing` can name the
set that ships it: MAME's `fm7740sx` carries the FM77AV40SX `DICROM.rom`,
`EXTSUB.rom` and `KANJI2.rom` that No-Intro lists. A `romof` parent's
members are merged into the child, and members with no CRC32 are dropped: those
are undumped and a real romset does not carry them.
against. Every MAME release with a `-listxml` asset is imported, 0.177 to
0.289, not one in five. Kept sets:
The store is compacted: 22 MAME versions produce 31,141 entries but only 2,399
distinct recipes, since most sets do not change between releases. Each is kept
once, `dats` listing every version that agrees and `dat` naming the earliest, so
an identification reads as "unchanged since that version".
- the archive names a profile or platform references;
- the archive names the collection holds;
- the sets holding a dump-catalog entry the collection lacks (crc32 and size,
and sha1 when both sides carry one), so `provenance_report.py --missing` can
name the set that ships it: MAME's `fm7740sx` carries the FM77AV40SX
`DICROM.rom`, `EXTSUB.rom` and `KANJI2.rom` that No-Intro lists;
- every set whose member list equals an archive the collection keeps under
another name. `aga.zip` holds exactly MAME's `isa_aga`: the entry carries
the collection's name and keeps the upstream one in `set`. When several sets
share one member list (thirty keyboard sets carry the same ROM), the set
named like the archive is kept, then the first by name.
A `romof` parent's members are merged into the child, and members with no
CRC32 are dropped: those are undumped and a real romset does not carry them.
The store is compacted: 113 MAME versions produce 173,985 entries but only
2,936 distinct recipes, since most sets do not change between releases. Each
is kept once, `dats` listing every version that agrees and `dat` naming the
earliest, so an identification reads as "unchanged since that version".
`--identify` answers what an archive on disk actually is: the version whose
recipe reproduces it byte for byte. Step 1c of the pipeline prints how many
@@ -1524,6 +1535,61 @@ archive whose bytes carry metadata unrelated to its contents cannot be derived
from ROMs by anyone, so those are reported as unreproducible rather than guessed
at. Writing is explicit: the pipeline only ever reports.
### Archive recipes
The collection must regenerate every archive it serves from the ROMs it holds
plus a committed recipe, without a copy of that archive from elsewhere.
`recipes/archives.json` holds one record per archive in `database.json`,
`.variants/` copies included, keyed by path:
- the members in archive order, each with its size, crc32 and sha1;
- the container's sha1, md5 and size;
- `rebuild`: `exact` when `build_torrentzip()` over the members gives the
container byte for byte (proven at capture by rebuilding it), `canonical`
when the archive was written by another tool, so the build gives the same
members in other container bytes. Directory entries are listed under
`directories` and not rebuilt;
- `sources`: the profile `contents:` blocks and the MAME and FBNeo DAT
recipes whose member list equals the archive's, matched by members rather
than by name, then `collection`, the archive itself. A DAT is provenance,
never the only source;
- `bytes_pinned` and `pins` when a platform or a profile pins the container
hash. A `zipped_file` md5 and a Recalbox composite md5 name no container
bytes and are not pins.
```bash
python scripts/romset_recipes.py --capture # pipeline step 1c
python scripts/romset_recipes.py --build neogeo.zip --output-dir tmp/built
python scripts/romset_recipes.py --build "bios/SNK/Neo Geo CD/neocdz.zip" --output-dir tmp/built
python scripts/romset_recipes.py --build-all --output-dir tmp/built --discard
```
`--capture` reads only the archives whose sha1 changed since the last run,
and recomputes sources and pins every time. `--build` takes a path or a file
name (every archive of that name is built, each at its path below `bios/`
inside the output directory). Each member is read by its sha1: a loose file
first, then a member of another archive, then the archive's own copy. The
result is written beside its destination, read back from disk and checked
against the recipe: the recorded sha1 for an exact recipe, the members for a
canonical one. A mismatch is built once more, then refused, and nothing is
moved into place. `--build-all` prints one line per archive (exact, canonical,
or impossible with the member nothing holds) and exits 1 when an archive
failed or was impossible. `--discard` deletes each build once verified, which
proves the whole collection without the disk space.
The summary also counts the archives that read a member from their own copy:
that member is held nowhere else, and would have to become a loose file before
the archive bytes could be dropped from `bios/`.
**Deflate implementation.** TorrentZip bytes are the output of zlib's level-9
deflate (`compressobj(9, DEFLATED, -15)`). Stock zlib 1.2.13 to 1.3.2
reproduced all 1,561 TorrentZip archives tested in November 2026, on Linux and
on Windows. zlib-ng does not: python.org CPython 3.14 and 3.15 on Windows
(zlib-ng 2.2.4) reproduced 479, and zlib-ng 2.3.3 545, the two versions
disagreeing with each other. `--capture` refuses to run under zlib-ng, and
`--build` says so and checks every recipe by its members, never claiming an
exact rebuild. `torrentzip.zlib_ng_version()` is the test.
## Large files
Files too large for git are stored as assets on the `large-files` GitHub