feat: generate MAME device sets from -listxml

A machine that instantiates a device loads the device's ROMs from <shortname>.zip (src/emu/device.cpp:140-152), and a missing required ROM stops that machine (romload.cpp:529-570, 667-671). The BIOS root scan never sees these sets. mame_hash_scraper --devices --listxml reads them from the -listxml of the release mame.yml pins, refuses another release, and cites each set at the ROM_START block holding its ROMs. The BIOS root merge also wrote refs without the upstream prefix and would have rewritten all 171 of them in mame.yml.
This commit is contained in:
Abdessamad Derraz committed 2026-10-20 15:12:23 +02:00
1 parent 7a65a2cf83
commit 8de45b6f45
7 files changed
+974 -13

No files matched your search

+13 -1
View File
@@ -45,7 +45,7 @@ Every module in `tests/`, by the area it guards.
| Hardware targets and systems | synthetic + target files | `test_extract_targets`, `test_lakka_targets`, `test_libretro_slugs`, `test_system_ids`, `test_system_selection`, `test_target_names`, `test_target_scrapers`, `test_target_systems`, `test_target_verify` |
| Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_selection` |
| Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_diff`, `test_truth_layout` |
| Scrapers, upstream and provenance | inline sources, synthetic | `test_check_freshness`, `test_dedup`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_mame_parser`, `test_mame_ref_audit`, `test_profile_sync`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_scraper_contract`, `test_upstream`, `test_upstream_index` |
| Scrapers, upstream and provenance | inline sources, synthetic | `test_check_freshness`, `test_dedup`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_profile_sync`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_scraper_contract`, `test_upstream`, `test_upstream_index` |
| Site, docs and repository contracts | rendered pages, sources | `test_contributing_text`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_optional_dependencies`, `test_prose_style`, `test_site_availability`, `test_site_cores`, `test_site_exports`, `test_site_platform_hashes`, `test_site_system_links`, `test_site_validation`, `test_site_write`, `test_validate_pr`, `test_validate_pr_hashes`, `test_workflow_paths` |
## Test architecture
@@ -99,6 +99,18 @@ sets from driver files. Fixtures are inline C source snippets containing
- Multi-region ROM blocks
- Macro expansion and edge cases
**test_mame_devices.** Tests the device ROM sets read from a MAME `-listxml`
and merged into a profile, on an inline listxml and driver source. Covers:
- Only `isdevice="yes"` sets with a dumped ROM kept, from a plain or zipped listxml
- The `ROM_START` block cited for a set: found through a ROM macro, a block
inside a comment ignored, one dump under two names told apart, the label
`device_rom_region` returns, not found and ambiguous
- Merge: a set added under the generated section in name order, a written
entry's contents refreshed with its manual fields kept, BIOS root sets left
as written, a second run changing nothing, a new BIOS root placed before the
section
**test_fbneo_parser.** Tests the FBNeo C source parser that identifies
`BDF_BOARDROM` sets. Same inline fixture approach.
+21 -2
View File
@@ -622,7 +622,7 @@ Further options, by script:
| `validate_pr.py` | `--changed` takes the BIOS files the branch changes, `--markdown`, `--json` |
| `validate_site.py` | `--site-dir` (rendered site), `--config` (mkdocs.yml), `--max-errors N` |
| `scraper/coreinfo_scraper.py` | `--compare-db database.json` lists the `.info` firmware the database lacks |
| `scraper/mame_hash_scraper.py` | `--release mameNNNN --profile FILE --source-commit SHA --core NAME --built-by PLATFORM` writes the profile of a MAME build a platform compiles |
| `scraper/mame_hash_scraper.py` | `--release mameNNNN --profile FILE --source-commit SHA --core NAME --built-by PLATFORM` writes the profile of a MAME build a platform compiles. `--devices --listxml FILE` documents in `mame.yml` every device set with dumped ROMs that the release's `-listxml` names (`isdevice="yes"`), with `--dry-run` to preview |
| `scraper/romset_dat_importer.py` | `--cache-dir` (downloaded listxml and DATs) |
## Installation tools
@@ -743,9 +743,28 @@ stand alone.
| `misterfpga_scraper` | BiosDB_MiSTer bios_db.json.zip | JSON in ZIP |
| `coreinfo_scraper` | .info files from libretro-core-info | INI-like |
| `bizhawk_scraper` | FirmwareDatabase.cs | C# source |
| `mame_hash_scraper` | mamedev/mame source tree. `--release mameNNNN` writes the profile of one release a platform builds | C source (sparse clone) |
| `mame_hash_scraper` | mamedev/mame source tree. `--release mameNNNN` writes the profile of one release a platform builds. `--devices --listxml FILE` adds the device ROM sets of the release's `-listxml` | C source (sparse clone), listxml |
| `fbneo_hash_scraper` | FBNeo source tree | C source (sparse clone) |
A machine that instantiates a device loads the device's ROMs from
`<device shortname>.zip`, and a missing ROM that is neither optional nor
undumped stops that machine. The BIOS root scan never sees these sets, so
`--devices` reads them from the `-listxml` of the release `mame.yml` is pinned
on and refuses another release. Each set is cited at the `ROM_START` to
`ROM_END` block holding its ROMs, in the file the listxml names, read at the
release tag through the upstream cache. When several blocks qualify, the one
matching every ROM by name and CRC wins, then the one named after the set, then
the one the device class returns from `device_rom_region`. A set still
undecided is written without a `source_ref` and listed in the output. New sets
go under the `Device ROM sets (generated from MAME -listxml)` comment at the end
of `files:` with `required: false`. A set the profile already lists keeps every
field a reader wrote and only gets its contents refreshed.
```bash
python -m scripts.scraper.mame_hash_scraper --devices --listxml .cache/dats/mame0289lx.zip --dry-run
python -m scripts.scraper.mame_hash_scraper --devices --listxml .cache/dats/mame0289lx.zip
```
Internal modules: `base_scraper.py` (abstract base with `_fetch_raw()` caching
and shared CLI), `dat_parser.py` (clrmamepro DAT format parser),
`logiqx_parser.py` (Logiqx XML DAT parser, used by the dump catalogs),