42 Commits

Author SHA1 Message Date
Abdessamad Derraz
40ff2b5307 Merge branch 'main' of https://github.com/Abdess/retrobios 2026-03-30 23:58:20 +02:00
Abdessamad Derraz
d0dd05ddf6 docs: add wiki pages for all audiences, fix .old.yml leak
9 new wiki pages: getting-started, faq, troubleshooting,
advanced-usage, verification-modes, adding-a-platform,
adding-a-scraper, testing-guide, release-process.

Updated architecture.md with mermaid diagrams, tools.md with
full pipeline and target/exporter sections, profiling.md with
missing fields, index.md with glossary and nav links.

Expanded CONTRIBUTING.md from stub to full contributor guide.

Filter .old.yml from load_emulator_profiles, generate_db alias
collection, and generate_readme counts. Fix BizHawk sha1 mode
in tools.md, fix RetroPie path, fix export_truth.py typos.
2026-03-30 23:58:12 +02:00
Abdessamad Derraz
d4b0af0a38 docs: add wiki pages for all audiences, fix .old.yml leak
9 new wiki pages: getting-started, faq, troubleshooting,
advanced-usage, verification-modes, adding-a-platform,
adding-a-scraper, testing-guide, release-process.

Updated architecture.md with mermaid diagrams, tools.md with
full pipeline and target/exporter sections, profiling.md with
missing fields, index.md with glossary and nav links.

Expanded CONTRIBUTING.md from stub to full contributor guide.

Filter .old.yml from load_emulator_profiles, generate_db alias
collection, and generate_readme counts. Fix BizHawk sha1 mode
in tools.md, fix RetroPie path, fix export_truth.py typos.
2026-03-30 23:42:00 +02:00
Abdessamad Derraz
038c3d3b40 feat: enrich MAME/FBNeo profiles with upstream BIOS contents
auto-fetched from mamedev/mame 0.287 and finalburnneo/FBNeo v1.0.0.2.
mame: +20 new BIOS root sets, 96 entries enriched with contents.
mamearcade: 47 entries enriched with contents.
mamemess: 20 entries enriched with contents.
fbneo: +13 new ROM entries from upstream BIOS sets.
2026-03-30 21:39:02 +02:00
Abdessamad Derraz
427fef5669 fix: text-based YAML patching preserves formatting
replace yaml.dump with surgical text edits for contents/source_ref.
preserves comments, block scalars, quoting, indentation.
fix FBNeo new entry detection using parsed keys instead of text search.
2026-03-30 21:35:41 +02:00
Abdessamad Derraz
75e34898ee feat: add MAME/FBNeo hash auto-fetch scrapers
sparse clone upstream repos, parse BIOS root sets from C source,
cache as JSON, merge into emulator profiles with backup.
covers macro expansion, version detection, subset profile protection.
2026-03-30 19:11:26 +02:00
Abdessamad Derraz
94c3ac9834 feat: add hash merge for MAME and FBNeo profiles 2026-03-30 18:33:09 +02:00
Abdessamad Derraz
319a1d2041 feat: add MAME source code parser for BIOS root sets 2026-03-30 18:29:31 +02:00
Abdessamad Derraz
00d7b57884 feat: add FBNeo source parser for BIOS sets 2026-03-30 18:29:06 +02:00
Abdessamad Derraz
caf6285a04 fix: skip entries without md5 in batocera and retrobat exports 2026-03-30 17:46:48 +02:00
Abdessamad Derraz
529cb8a915 fix: recalbox paths from scrape, batocera md5 fallback from scrape 2026-03-30 17:35:39 +02:00
Abdessamad Derraz
1146fdf177 fix: rewrite emudeck exporter to match exact checkBIOS.sh format 2026-03-30 17:21:56 +02:00
Abdessamad Derraz
4fbb3571f8 fix: exporters use _dest fallback, merge colliding systems, per-platform subdirs 2026-03-30 17:15:44 +02:00
Abdessamad Derraz
0be68edad0 feat: add exporters for lakka, retropie, emudeck, retrodeck, romm 2026-03-30 17:07:08 +02:00
Abdessamad Derraz
1ffc4f89ca refactor: registry merge is fully flexible, no hardcoded lists 2026-03-30 16:38:13 +02:00
Abdessamad Derraz
f1ebfff5bd refactor: registry merge uses exclusion list instead of hardcoded fields 2026-03-30 16:36:40 +02:00
Abdessamad Derraz
425ea064ae fix: scrapers merge into existing YAML instead of overwriting 2026-03-30 16:31:40 +02:00
Abdessamad Derraz
6818a18a42 feat: load_platform_config merges all metadata from registry 2026-03-30 16:24:40 +02:00
Abdessamad Derraz
c11de6dba6 fix: restore retroarch.yml fields lost by scraper regeneration 2026-03-30 16:22:16 +02:00
Abdessamad Derraz
c4f3192020 fix: system.dat rom quoting, native_ids, acronym display names 2026-03-30 16:17:50 +02:00
Abdessamad Derraz
e2d0510f4e fix: exporters match exact native formats with display names 2026-03-30 16:09:02 +02:00
Abdessamad Derraz
74269bab84 fix: rewrite exporters to match exact native formats 2026-03-30 15:49:33 +02:00
Abdessamad Derraz
1e6b499602 feat: add batocera, recalbox, retrobat native exporters 2026-03-30 15:31:44 +02:00
Abdessamad Derraz
9b785ec785 feat: add missing laseractive sega pac bios files
v1.05 japan and v1.01 japan from archive.org.
v1.04 us variant alias for pioneer-named file.
resolves retrobat laseractive missing files.
2026-03-30 15:16:23 +02:00
Abdessamad Derraz
d415777f2c feat: add PS3UPDAT.PUP to rpcs3 profile
storage: large_file, validated via SCEUF magic + HMAC-SHA1
per entry (PUP.cpp:23-77). also adds missing standard fields
(cores, core_classification, upstream), removes non-standard
firmware_file/validation/firmware_version fields.
2026-03-30 15:06:51 +02:00
Abdessamad Derraz
eafabd20f3 refactor: skip writing generated files when content unchanged
write_if_changed in common.py compares content after stripping
timestamps (generated_at, Auto-generated on, Generated on).
applied to generate_db, generate_readme, generate_site.
eliminates timestamp-only diffs in database.json, README.md,
mkdocs.yml, and 423 docs pages.
2026-03-30 14:33:44 +02:00
Abdessamad Derraz
2aca4927c0 chore: regenerate database, readme, manifests, site 2026-03-30 14:19:00 +02:00
Abdessamad Derraz
17777f315b feat: agnostic bios mode for filename-agnostic emulators
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
2026-03-30 14:18:54 +02:00
Abdessamad Derraz
692484d32d refactor: remove false aliases from pcsx2 and melonds profiles
aliases must be same-SHA1 alternative names, not distinct files.
pcsx2: 164 different BIOS dumps are separate DB entries, not aliases.
melonds: 6 regional NAND dumps are separate DB entries, not aliases.
also cleans pcsx2 non-standard fields, fixes display_name.
2026-03-30 12:37:32 +02:00
Abdessamad Derraz
a8430940f9 feat: add regional nand aliases to melonds profile
dsi_nand.bin aliases: DSi_Nand_USA/EUR/JPN/AUS/CHN/KOR.bin
for repo resolution. Code loads a single configurable path
(libretro.cpp:836, EmuInstance.cpp:1036-1050), validates
nocash footer (DSi_NAND.cpp:42-111). size + storage added.
2026-03-30 12:20:26 +02:00
Abdessamad Derraz
1f073f521d fix: preserve batocera version when github fetch fails 2026-03-30 11:55:57 +02:00
Abdessamad Derraz
903c49edcf feat: add tests for registry merge, all_libretro expansion, hash fallback, system normalization 2026-03-30 11:33:59 +02:00
Abdessamad Derraz
d3a2224dd2 chore: regenerate database, readme, manifests, site 2026-03-30 09:42:39 +02:00
Abdessamad Derraz
f898f26847 chore: regenerate retrobat.yml with corrected system slugs 2026-03-30 09:11:00 +02:00
Abdessamad Derraz
2712307420 feat: add tandy vis bios root set to mame profiles
vis.zip (p513bk0b.bin + p513bk1b.bin) from src/mame/trs/vis.cpp.
Driver added in MAME 0.180, not present in older cores.
2026-03-30 09:09:58 +02:00
Abdessamad Derraz
54022e9db1 feat: hash-based matching for cross-reference
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
2026-03-30 08:25:54 +02:00
Abdessamad Derraz
4db9e4350c fix: add missing system slugs to batocera and retrobat scrapers 2026-03-30 07:58:46 +02:00
Abdessamad Derraz
6864ce6584 feat: diff hash fallback detects platform renames 2026-03-30 07:53:59 +02:00
Abdessamad Derraz
12196b6445 feat: add 55 missing cores across 6 platform registries 2026-03-30 07:17:57 +02:00
Abdessamad Derraz
7551e41a7b feat: load_platform_config merges cores from registry 2026-03-30 07:13:15 +02:00
Abdessamad Derraz
7b484605d4 feat: add 47 missing cores to batocera platform config 2026-03-30 07:09:52 +02:00
Abdessamad Derraz
b587381f05 feat: resolve_platform_cores expands all_libretro in list 2026-03-30 07:06:32 +02:00
72 changed files with 31047 additions and 3649 deletions

View File

@@ -1,14 +1,109 @@
# Contributing to RetroBIOS
## Add a BIOS file
## Types of contributions
1. Fork this repository
2. Place the file in `bios/Manufacturer/Console/filename`
3. Variants (alternate hashes): `bios/Manufacturer/Console/.variants/`
4. Create a Pull Request - checksums are verified automatically
- **Add a BIOS file** - a great way to get started. Fork, add the file, open a PR.
- **Create an emulator profile** - document what a core actually loads from source code. See the [profiling guide](https://abdess.github.io/retrobios/wiki/profiling/).
- **Add a platform** - integrate a new frontend (scraper + YAML config). See [adding a platform](https://abdess.github.io/retrobios/wiki/adding-a-platform/).
- **Add or fix a scraper** - parse upstream sources for BIOS requirements. See [adding a scraper](https://abdess.github.io/retrobios/wiki/adding-a-scraper/).
- **Fix a bug or improve tooling** - Python scripts in `scripts/`, single dependency (`pyyaml`).
## File conventions
## Local setup
- Files >50 MB go in GitHub release assets (`large-files` release)
- RPG Maker and ScummVM directories are excluded from deduplication
- See the [documentation site](https://abdess.github.io/retrobios/) for full details
```bash
git clone https://github.com/Abdess/retrobios.git
cd retrobios
pip install pyyaml
# run tests
python -m unittest tests.test_e2e -v
# run full pipeline (DB + verify + packs + consistency check)
python scripts/pipeline.py --offline
```
Requires Python 3.10 or later.
## Adding a BIOS file
1. Place the file in `bios/Manufacturer/Console/filename`.
2. Alternate versions (different hash, same purpose) go in `bios/Manufacturer/Console/.variants/`.
3. Files over 50 MB go as assets on the `large-files` GitHub release (git handles them better that way).
4. RPG Maker and ScummVM directories are excluded from deduplication - please keep their structure as-is.
5. Open a pull request. CI validates checksums automatically and posts a report.
## Commit conventions
Format: `type: description` (50 characters max, lowercase start).
Allowed types: `feat`, `refactor`, `chore`, `docs`, `fix`.
```
feat: add panasonic 3do bios files
docs: update architecture diagram
fix: resolve truncated md5 matching
chore: remove unused test fixtures
refactor: extract hash logic to common.py
```
Keep messages factual. No marketing language, no superfluous adjectives.
## Code and documentation quality
The codebase runs on Python 3.10+ with a single dependency (`pyyaml`). All modules
include `from __future__ import annotations` at the top. Type hints on every function
signature, `pathlib` instead of `os.path`, and dataclasses where a plain class would
just hold attributes.
On performance: O(1) or O(n) algorithms are preferred. If something needs O(n^2), a
comment explaining why helps future readers. List comprehensions over explicit loops,
generators when iterating large datasets, and standard default arguments
(`def f(items=None)` over `def f(items=[])`).
File I/O uses context managers. ZIP extraction goes through `safe_extract_zip()` in
`common.py`, which prevents zip-slip path traversal.
The code stays lean. Comments that describe *why* age better than comments that
restate *what*. Unused variables can be deleted rather than renamed with an underscore.
The same spirit applies to documentation and emulator profiles. Straightforward
language, honest labels ("untested" when something is untested).
When a bug is spotted while working on something else, fixing it in the same PR
keeps things tidy. Features ideally ship complete in one pass: code, tests, a
passing pipeline run, and any documentation updates together.
## Pull request process
CI runs four checks on every PR:
| Check | What it does |
|-------|-------------|
| `validate-bios` | hashes changed BIOS files against the database, posts a validation report |
| `validate-configs` | schema-validates platform YAML configs |
| `run-tests` | runs the full E2E test suite |
| `label-pr` | auto-labels the PR based on changed file paths |
All four checks must pass. One maintainer approval is required before merge.
## Project philosophy
This repository is an archive. The goal is a complete collection of every file a
user needs to run every supported console in an emulator, with zero friction. That
scope goes beyond BIOS ROMs: firmware updates, system fonts, data files, encryption
keys, hiscore databases, anything the emulator loads from disk rather than generating
itself. In a hundred years the pack should still work out of the box.
Licensing considerations are secondary to completeness. If an emulator needs a
file to function, the file belongs in the collection. The project's legal position is
documented in the FAQ section of the site.
Integrity matters. Every file, hash, and metadata field is cross-checked
against the emulator's source code. Upstream references like System.dat,
`.info` files, and wiki pages are valuable and generally accurate, though
they can occasionally fall out of date. When an upstream source and the
code disagree, the code at runtime is the tiebreaker.
## Documentation
Full reference docs, profiling guides, and architecture details are on the [documentation site](https://abdess.github.io/retrobios/).

View File

@@ -2,7 +2,7 @@
Complete BIOS and firmware packs for Batocera, BizHawk, EmuDeck, Lakka, Recalbox, RetroArch, RetroBat, RetroDECK, RetroPie, and RomM.
**7,244** verified files across **387** systems, ready to extract into your emulator's BIOS directory.
**7,241** verified files across **396** systems, ready to extract into your emulator's BIOS directory.
## Quick Install
@@ -27,14 +27,14 @@ Pick your platform, download the ZIP, extract to the BIOS path.
| Platform | BIOS files | Extract to | Download |
|----------|-----------|-----------|----------|
| Batocera | 359 | `/userdata/bios/` | [Download](../../releases/latest) |
| Batocera | 362 | `/userdata/bios/` | [Download](../../releases/latest) |
| BizHawk | 118 | `Firmware/` | [Download](../../releases/latest) |
| EmuDeck | 161 | `Emulation/bios/` | [Download](../../releases/latest) |
| Lakka | 448 | `system/` | [Download](../../releases/latest) |
| Recalbox | 346 | `/recalbox/share/bios/` | [Download](../../releases/latest) |
| RetroArch | 448 | `system/` | [Download](../../releases/latest) |
| RetroBat | 331 | `bios/` | [Download](../../releases/latest) |
| RetroDECK | 2007 | `~/retrodeck/bios/` | [Download](../../releases/latest) |
| RetroBat | 339 | `bios/` | [Download](../../releases/latest) |
| RetroDECK | 2006 | `~/retrodeck/bios/` | [Download](../../releases/latest) |
| RetroPie | 448 | `BIOS/` | [Download](../../releases/latest) |
| RomM | 374 | `bios/{platform_slug}/` | [Download](../../releases/latest) |
@@ -44,14 +44,14 @@ BIOS, firmware, and system files for consoles from Atari to PlayStation 3.
Each file is checked against the emulator's source code to match what the code actually loads at runtime.
- **10 platforms** supported with platform-specific verification
- **328 emulators** profiled from source (RetroArch cores + standalone)
- **387 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
- **7,244 files** verified with MD5, SHA1, CRC32 checksums
- **9266 MB** total collection size
- **329 emulators** profiled from source (RetroArch cores + standalone)
- **396 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
- **7,241 files** verified with MD5, SHA1, CRC32 checksums
- **8144 MB** total collection size
## Supported systems
NES, SNES, Nintendo 64, GameCube, Wii, Game Boy, Game Boy Advance, Nintendo DS, Nintendo 3DS, Switch, PlayStation, PlayStation 2, PlayStation 3, PSP, PS Vita, Mega Drive, Saturn, Dreamcast, Game Gear, Master System, Neo Geo, Atari 2600, Atari 7800, Atari Lynx, Atari ST, MSX, PC Engine, TurboGrafx-16, ColecoVision, Intellivision, Commodore 64, Amiga, ZX Spectrum, Arcade (MAME), and 353+ more.
NES, SNES, Nintendo 64, GameCube, Wii, Game Boy, Game Boy Advance, Nintendo DS, Nintendo 3DS, Switch, PlayStation, PlayStation 2, PlayStation 3, PSP, PS Vita, Mega Drive, Saturn, Dreamcast, Game Gear, Master System, Neo Geo, Atari 2600, Atari 7800, Atari Lynx, Atari ST, MSX, PC Engine, TurboGrafx-16, ColecoVision, Intellivision, Commodore 64, Amiga, ZX Spectrum, Arcade (MAME), and 362+ more.
Full list with per-file details: **[https://abdess.github.io/retrobios/](https://abdess.github.io/retrobios/)**
@@ -59,15 +59,15 @@ Full list with per-file details: **[https://abdess.github.io/retrobios/](https:/
| Platform | Coverage | Verified | Untested | Missing |
|----------|----------|----------|----------|---------|
| Batocera | 359/359 (100.0%) | 354 | 5 | 0 |
| Batocera | 356/362 (98.3%) | 349 | 7 | 6 |
| BizHawk | 118/118 (100.0%) | 118 | 0 | 0 |
| EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 |
| Lakka | 443/448 (98.9%) | 443 | 0 | 5 |
| Recalbox | 276/346 (79.8%) | 273 | 3 | 70 |
| RetroArch | 443/448 (98.9%) | 443 | 0 | 5 |
| RetroBat | 330/331 (99.7%) | 326 | 4 | 1 |
| RetroDECK | 1958/2007 (97.6%) | 1932 | 26 | 49 |
| RetroPie | 443/448 (98.9%) | 443 | 0 | 5 |
| Lakka | 442/448 (98.7%) | 442 | 0 | 6 |
| Recalbox | 277/346 (80.1%) | 274 | 3 | 69 |
| RetroArch | 442/448 (98.7%) | 442 | 0 | 6 |
| RetroBat | 339/339 (100.0%) | 335 | 4 | 0 |
| RetroDECK | 1960/2006 (97.7%) | 1934 | 26 | 46 |
| RetroPie | 442/448 (98.7%) | 442 | 0 | 6 |
| RomM | 372/374 (99.5%) | 372 | 0 | 2 |
## Build your own pack
@@ -104,7 +104,7 @@ The [documentation site](https://abdess.github.io/retrobios/) provides:
- **Per-emulator profiles** with source code references for every file
- **Per-system pages** showing which emulators and platforms cover each console
- **Gap analysis** identifying missing files and undeclared core requirements
- **Cross-reference** mapping files across 10 platforms and 328 emulators
- **Cross-reference** mapping files across 10 platforms and 329 emulators
## How it works
@@ -130,4 +130,4 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
This repository provides BIOS files for personal backup and archival purposes.
*Auto-generated on 2026-03-29T21:00:40Z*
*Auto-generated on 2026-03-30T20:16:27Z*

View File

@@ -1,7 +1,7 @@
{
"generated_at": "2026-03-30T04:10:12Z",
"total_files": 7239,
"total_size": 8539795099,
"generated_at": "2026-03-30T13:15:58Z",
"total_files": 7241,
"total_size": 8540057243,
"files": {
"520d3d1b5897800af47f92efd2444a26b7a7dead": {
"path": "bios/3DO Company/3DO/3do_arcade_saot.bin",
@@ -40843,16 +40843,6 @@
"crc32": "11647ca5",
"adler32": "1817f6f4"
},
"26237b333db4a4c6770297fa5e655ea95840d5d9": {
"path": "bios/Pioneer/LaserActive/Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer - Sega)(JP)(en-ja).bin",
"name": "Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer - Sega)(JP)(en-ja).bin",
"size": 131072,
"sha1": "26237b333db4a4c6770297fa5e655ea95840d5d9",
"md5": "a5a2f9aae57d464bc66b80ee79c3da6e",
"sha256": "dca942d977217f703d8d1c6eb1aeb6b32c78ecc421486bbb46c459d385161c94",
"crc32": "00eedb3a",
"adler32": "94a45a21"
},
"aa811861f8874775075bd3f53008c8aaf59b07db": {
"path": "bios/Pioneer/LaserActive/Pioneer LaserActive Sega PAC Boot ROM v1.04 (1993)(Pioneer - Sega)(US).bin",
"name": "Pioneer LaserActive Sega PAC Boot ROM v1.04 (1993)(Pioneer - Sega)(US).bin",
@@ -40863,6 +40853,16 @@
"crc32": "50cd3d23",
"adler32": "72a13133"
},
"26237b333db4a4c6770297fa5e655ea95840d5d9": {
"path": "bios/Pioneer/LaserActive/Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer - Sega)(JP)(en-ja).bin",
"name": "Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer - Sega)(JP)(en-ja).bin",
"size": 131072,
"sha1": "26237b333db4a4c6770297fa5e655ea95840d5d9",
"md5": "a5a2f9aae57d464bc66b80ee79c3da6e",
"sha256": "dca942d977217f703d8d1c6eb1aeb6b32c78ecc421486bbb46c459d385161c94",
"crc32": "00eedb3a",
"adler32": "94a45a21"
},
"6973e2593e66fd21627fedccec98d4a364afaaff": {
"path": "bios/Pioneer/LaserActive/[BIOS] LaserActive PAC-N1 (Japan) (v1.02).bin",
"name": "[BIOS] LaserActive PAC-N1 (Japan) (v1.02).bin",
@@ -40883,6 +40883,26 @@
"crc32": "01223dd5",
"adler32": "74e98625"
},
"bc746df0c5d2b779ca41a94954b60d6ac6a7c2a3": {
"path": "bios/Pioneer/LaserActive/[BIOS] LaserActive PAC-S1 (Japan) (v1.01).bin",
"name": "[BIOS] LaserActive PAC-S1 (Japan) (v1.01).bin",
"size": 131072,
"sha1": "bc746df0c5d2b779ca41a94954b60d6ac6a7c2a3",
"md5": "219dc2aa7cb8d1ad5142c86d50c2ffa5",
"sha256": "3300e3771b468b41818b90c2358ff288da69bada92b8247acd793a437b30731c",
"crc32": "24ad336e",
"adler32": "6993a1dd"
},
"f7101b40cae0484a0f43f3bbee2d55033ff1e3ec": {
"path": "bios/Pioneer/LaserActive/[BIOS] LaserActive PAC-S1 (Japan) (v1.05).bin",
"name": "[BIOS] LaserActive PAC-S1 (Japan) (v1.05).bin",
"size": 131072,
"sha1": "f7101b40cae0484a0f43f3bbee2d55033ff1e3ec",
"md5": "515636778430c8e01027234c38c4ddf8",
"sha256": "369c1e699abbaecd231a3ab7b98443d28097366ff3cdd978698417195d54ca3c",
"crc32": "1493522c",
"adler32": "9e972082"
},
"c1b9202cbe072db12114b223a9ba5374b30718fb": {
"path": "bios/Pioneer/LaserActive/[BIOS] LaserActive PCE-LP1 (Japan) (v1.02).bin",
"name": "[BIOS] LaserActive PCE-LP1 (Japan) (v1.02).bin",
@@ -76480,10 +76500,12 @@
"c500ff71236068e0dc0d0603d265ae76": "5130243429b40b01a14e1304d0394b8459a6fbae",
"f1071cdb0b6b10dde94d3bc8a6146387": "a6120aed50831c9c0d95dbdf707820f601d9452e",
"279008e4a0db2dc5f1c048853b033828": "54b8d2c1317628de51a85fc1c424423a986775e4",
"a5a2f9aae57d464bc66b80ee79c3da6e": "26237b333db4a4c6770297fa5e655ea95840d5d9",
"0e7393cd0951d6dde818fcd4cd819466": "aa811861f8874775075bd3f53008c8aaf59b07db",
"a5a2f9aae57d464bc66b80ee79c3da6e": "26237b333db4a4c6770297fa5e655ea95840d5d9",
"f69f173b251d8bf7649b10a9167a10bf": "6973e2593e66fd21627fedccec98d4a364afaaff",
"f0fb8a4605ac7eefbafd4f2d5a793cc8": "f7412aa822d70a55b2ff3d7095137263dc54f6b6",
"219dc2aa7cb8d1ad5142c86d50c2ffa5": "bc746df0c5d2b779ca41a94954b60d6ac6a7c2a3",
"515636778430c8e01027234c38c4ddf8": "f7101b40cae0484a0f43f3bbee2d55033ff1e3ec",
"761fea207d0eafd4cfd78da7c44cac88": "c1b9202cbe072db12114b223a9ba5374b30718fb",
"69489153dde910a69d5ae6de5dd65323": "f2a9ce387019bf272c6e3459d961b30f28942ac5",
"3fd0d13282b031f4c017cd6bf6597183": "9dda4cbcc1d3f6d38c10fee3de53b9abd5e47ec0",
@@ -91151,18 +91173,24 @@
"jopac.bin": [
"54b8d2c1317628de51a85fc1c424423a986775e4"
],
"Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer - Sega)(JP)(en-ja).bin": [
"26237b333db4a4c6770297fa5e655ea95840d5d9"
],
"Pioneer LaserActive Sega PAC Boot ROM v1.04 (1993)(Pioneer - Sega)(US).bin": [
"aa811861f8874775075bd3f53008c8aaf59b07db"
],
"Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer - Sega)(JP)(en-ja).bin": [
"26237b333db4a4c6770297fa5e655ea95840d5d9"
],
"[BIOS] LaserActive PAC-N1 (Japan) (v1.02).bin": [
"6973e2593e66fd21627fedccec98d4a364afaaff"
],
"[BIOS] LaserActive PAC-N10 (US) (v1.02).bin": [
"f7412aa822d70a55b2ff3d7095137263dc54f6b6"
],
"[BIOS] LaserActive PAC-S1 (Japan) (v1.01).bin": [
"bc746df0c5d2b779ca41a94954b60d6ac6a7c2a3"
],
"[BIOS] LaserActive PAC-S1 (Japan) (v1.05).bin": [
"f7101b40cae0484a0f43f3bbee2d55033ff1e3ec"
],
"[BIOS] LaserActive PCE-LP1 (Japan) (v1.02).bin": [
"c1b9202cbe072db12114b223a9ba5374b30718fb"
],
@@ -101218,6 +101246,9 @@
"g7400.bin": [
"5130243429b40b01a14e1304d0394b8459a6fbae"
],
"[BIOS] LaserActive PAC-S10 (US) (v1.04).bin": [
"aa811861f8874775075bd3f53008c8aaf59b07db"
],
"Battle 1.mid": [
"dc5cc32fafa442b09ab2d814ed32074d02597234"
],
@@ -102371,6 +102402,9 @@
"ts1500.zxpand.ovl": [
"0334b35f164089df7b2a82d46fa0ec6e43fafa90"
],
"tvc64.zip": [
"abf119cf947ea32defd08b29a8a25d75f6bd4987"
],
"TVC22_D7.64K": [
"abf119cf947ea32defd08b29a8a25d75f6bd4987"
],
@@ -102413,6 +102447,24 @@
"EXOS21.ROM": [
"55315b20fecb4441a07ee4bc5dc7153f396e0a2e"
],
"bk0010.zip": [
"4e83a94ae5155bbea14d7331a5a8db82457bd5ae",
"34fa37599f2f9eb607390ef2458a3c22d87f09a9",
"f087af69044432a1ef2431a72ac06946e32f2dd3",
"7e9a30e38d7b78981999821640a68a201bb6df01"
],
"monit10.rom": [
"4e83a94ae5155bbea14d7331a5a8db82457bd5ae"
],
"bas11m_1.rom": [
"34fa37599f2f9eb607390ef2458a3c22d87f09a9"
],
"b11m_ext.rom": [
"f087af69044432a1ef2431a72ac06946e32f2dd3"
],
"b11m_bos.rom": [
"7e9a30e38d7b78981999821640a68a201bb6df01"
],
"sony-playstation:239665b1a3dade1b5a52c06338011044": [
"343883a7b555646da8cee54aadd2795b6e7dd070"
],
@@ -103185,27 +103237,15 @@
"Kickstart v3.0 rev 39.106 (1992)(Commodore)(A4000)[!].rom": [
"f0b4e9e29e12218c2d5bd7020e4e785297d91fd7"
],
"monit10.rom": [
"4e83a94ae5155bbea14d7331a5a8db82457bd5ae"
],
"focal10.rom": [
"6386e58bc1bba5e76baec9e8a1ca4b99dc3c573f"
],
"disk_327.rom": [
"28eefbb63047b26e4aec104aeeca74e2f9d0276c"
],
"b11m_bos.rom": [
"7e9a30e38d7b78981999821640a68a201bb6df01"
],
"b11m_ext.rom": [
"f087af69044432a1ef2431a72ac06946e32f2dd3"
],
"bas11m_0.rom": [
"9d76f3eefd64e032c763fa1ebf9cd3d9bd22317a"
],
"bas11m_1.rom": [
"34fa37599f2f9eb607390ef2458a3c22d87f09a9"
],
"terak.rom": [
"273a9933b68a290c5aedcd6d69faa7b1d22c0344"
],
@@ -107481,10 +107521,12 @@
"e20a9f41": "5130243429b40b01a14e1304d0394b8459a6fbae",
"a318e8d6": "a6120aed50831c9c0d95dbdf707820f601d9452e",
"11647ca5": "54b8d2c1317628de51a85fc1c424423a986775e4",
"00eedb3a": "26237b333db4a4c6770297fa5e655ea95840d5d9",
"50cd3d23": "aa811861f8874775075bd3f53008c8aaf59b07db",
"00eedb3a": "26237b333db4a4c6770297fa5e655ea95840d5d9",
"a8cb694c": "6973e2593e66fd21627fedccec98d4a364afaaff",
"01223dd5": "f7412aa822d70a55b2ff3d7095137263dc54f6b6",
"24ad336e": "bc746df0c5d2b779ca41a94954b60d6ac6a7c2a3",
"1493522c": "f7101b40cae0484a0f43f3bbee2d55033ff1e3ec",
"76116a02": "c1b9202cbe072db12114b223a9ba5374b30718fb",
"4e70e3c0": "f2a9ce387019bf272c6e3459d961b30f28942ac5",
"92bcc762": "9dda4cbcc1d3f6d38c10fee3de53b9abd5e47ec0",
@@ -124501,6 +124543,10 @@
"9451a1a09d8f75944dbd6f91193fc360f1de80ac",
"03bbb386cf530e804363acdfc1d13e64cf28af2e",
"55315b20fecb4441a07ee4bc5dc7153f396e0a2e",
"4e83a94ae5155bbea14d7331a5a8db82457bd5ae",
"34fa37599f2f9eb607390ef2458a3c22d87f09a9",
"f087af69044432a1ef2431a72ac06946e32f2dd3",
"7e9a30e38d7b78981999821640a68a201bb6df01",
"343883a7b555646da8cee54aadd2795b6e7dd070",
"e38466a4ba8005fba7e9e3c7b9efeba7205bee3f",
"ffa7f9a7fb19d773a0c3985a541c8e5623d2c30d",
@@ -124623,13 +124669,9 @@
"4192c505d130f446b2ada6bdc91dae730acafb4c",
"16df8b5fd524c5a1c7584b2457ac15aff9e3ad6d",
"f0b4e9e29e12218c2d5bd7020e4e785297d91fd7",
"4e83a94ae5155bbea14d7331a5a8db82457bd5ae",
"6386e58bc1bba5e76baec9e8a1ca4b99dc3c573f",
"28eefbb63047b26e4aec104aeeca74e2f9d0276c",
"7e9a30e38d7b78981999821640a68a201bb6df01",
"f087af69044432a1ef2431a72ac06946e32f2dd3",
"9d76f3eefd64e032c763fa1ebf9cd3d9bd22317a",
"34fa37599f2f9eb607390ef2458a3c22d87f09a9",
"273a9933b68a290c5aedcd6d69faa7b1d22c0344",
"4891d739a8a8b67923681bad4fb67edab2e90e50",
"6e89d1227581c76441a53d605f9e324185f1da33",
@@ -124956,6 +124998,9 @@
"roms/win486/ALI1429G.AMW": [
"72c60172fb1ba77c9b24b06b7755f0a16f0b3a13"
],
".variants/[BIOS] LaserActive PAC-S10 (US) (v1.04).bin": [
"aa811861f8874775075bd3f53008c8aaf59b07db"
],
"rtp/2003/Battle/Arrow.png": [
"7aa8d4c377efcea1c9fad01924da1ba7b8575e1e"
],

View File

@@ -5,7 +5,7 @@ source: "https://github.com/libretro/FBNeo"
upstream: "https://github.com/finalburnneo/FBNeo"
logo: "https://raw.githubusercontent.com/finalburnneo/FBNeo/master/projectfiles/xcode/Emulator/Assets.xcassets/AppIcon.appiconset/icon_512.png"
profiled_date: "2026-03-23"
core_version: "v1.0.0.03"
core_version: "v1.0.0.2"
display_name: "Arcade (FinalBurn Neo)"
cores:
- fbneo
@@ -1485,3 +1485,94 @@ files:
size: 155000000
note: "Two Tigers with Journey CD audio samples (2 WAVs)"
source_ref: "src/burn/snd/samples.cpp"
- name: "coleco.rom"
archive: cv_coleco.zip
required: true
size: 8192
crc32: "3aa93ef3"
source_ref: "src/burn/drv/coleco/d_coleco.cpp:1079"
- name: "colecoa.rom"
archive: cv_coleco.zip
required: true
size: 8192
crc32: "39bb16fc"
source_ref: "src/burn/drv/coleco/d_coleco.cpp:1079"
- name: "svi603.rom"
archive: cv_coleco.zip
required: true
size: 8192
crc32: "19e91b82"
source_ref: "src/burn/drv/coleco/d_coleco.cpp:1079"
- name: "czz50.rom"
archive: cv_coleco.zip
required: true
size: 16384
crc32: "4999abc6"
source_ref: "src/burn/drv/coleco/d_coleco.cpp:1079"
- name: "fdsbios.nes"
archive: fds_fdsbios.zip
required: true
size: 8192
crc32: "5e607dcf"
source_ref: "src/burn/drv/nes/d_nes.cpp:523"
- name: "st010.bin"
archive: snes_st010.zip
required: true
size: 69632
crc32: "aa11ee2d"
source_ref: "src/burn/drv/snes/d_snes.cpp:577"
- name: "st011.bin"
archive: snes_st011.zip
required: true
size: 69632
crc32: "34d2952c"
source_ref: "src/burn/drv/snes/d_snes.cpp:596"
- name: "msx.rom"
archive: msx_msx.zip
required: true
size: 32768
crc32: "a317e6b4"
source_ref: "src/burn/drv/msx/d_msx.cpp:1781"
- name: "msxj.rom"
archive: msx_msx.zip
required: true
size: 32768
crc32: "071135e0"
source_ref: "src/burn/drv/msx/d_msx.cpp:1781"
- name: "kanji.rom"
archive: msx_msx.zip
required: true
size: 262144
crc32: "1f6406fb"
source_ref: "src/burn/drv/msx/d_msx.cpp:1781"
- name: "supernova_modbios-japan.u10"
archive: skns.zip
required: true
size: 524288
crc32: "b8d3190c"
source_ref: "src/burn/drv/pst90s/d_suprnova.cpp:1865"
- name: "supernova-modbios-korea.u10"
archive: skns.zip
required: true
size: 524288
crc32: "1d90517c"
source_ref: "src/burn/drv/pst90s/d_suprnova.cpp:1865"
- name: "mcu"
archive: bubsys.zip
required: true
size: 4096
crc32: "00000000"
source_ref: "src/burn/drv/konami/d_nemesis.cpp:4539"

View File

@@ -5,6 +5,7 @@
emulator: LRPS2
type: libretro
core_classification: community_fork
bios_mode: agnostic
source: "https://github.com/libretro/ps2"
upstream: "https://github.com/PCSX2/pcsx2"
profiled_date: "2026-03-25"

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -77,11 +77,14 @@ files:
source_ref: "src/SPI.cpp:197-211, src/frontend/Util_ROM.cpp:201-217"
- name: dsi_nand.bin
agnostic: true
system: nintendo-dsi
description: "DSi NAND dump"
required: true
source_ref: "src/frontend/Util_ROM.cpp:224-235, src/DSi_NAND.cpp"
note: "Uses AES keys from ARM7i BIOS offset 0x8308"
size: 251658304
storage: large_file
source_ref: "src/frontend/Util_ROM.cpp:224-235, src/DSi_NAND.cpp:58-97"
note: "Any regional dump works. Nocash footer required (DSi eMMC CID/CPU at EOF-0x40 or 0xFF800). AES keys derived from ARM7i BIOS offset 0x8308."
- name: dsi_sd_card.bin
system: nintendo-dsi

View File

@@ -1,219 +1,74 @@
# PCSX2 emulator BIOS profile
# Generated from source analysis of https://github.com/PCSX2/pcsx2
# Commit analyzed: HEAD as of 2026-03-17
emulator: PCSX2
type: standalone
core_classification: official_port
bios_mode: agnostic
source: "https://github.com/PCSX2/pcsx2"
logo: "https://raw.githubusercontent.com/PCSX2/pcsx2/master/pcsx2-qt/resources/icons/PCSX2logo.svg"
profiled_date: "2026-03-18"
upstream: "https://github.com/PCSX2/pcsx2"
cores:
- pcsx2
profiled_date: "2026-03-30"
core_version: "Git"
display_name: "Sony - PlayStation 2 (LRPS2)"
display_name: "Sony - PlayStation 2 (PCSX2)"
systems: [sony-playstation-2]
bios_directory: "bios/"
bios_detection: "romdir" # scans romdir structure inside binary, looks for RESET/ROMVER/EXTINFO entries
bios_selection: "automatic" # scans all files in bios dir matching 4-8 MB size, validates via romdir
validation:
method: "romdir_parse"
min_size: 4194304 # 4 MB (MIN_BIOS_SIZE = 4 * _1mb)
max_size: 8388608 # 8 MB (MAX_BIOS_SIZE = 8 * _1mb)
required_entries: ["RESET", "ROMVER"]
optional_entries: ["EXTINFO"]
note: "Any file in bios/ between 4-8 MB with valid romdir containing RESET+ROMVER is accepted"
regions:
J: {zone: "Japan", id: 0}
A: {zone: "USA", id: 1}
E: {zone: "Europe", id: 2}
H: {zone: "Asia", id: 4}
C: {zone: "China", id: 6}
T: {zone: "T10K/COH-H", id: 8}
X: {zone: "Test", id: 9}
P: {zone: "Free", id: 10}
notes: |
Filename-agnostic BIOS detection. Scans bios/ for any file between 4-8 MB
with valid romdir structure (RESET + ROMVER entries). No hash validation.
Companion files (.rom1, .rom2, .nvm, .mec) derive paths from selected BIOS.
ROM1 (DVD player) and ROM2 (Chinese extension) silently skipped if missing.
NVM and MEC auto-created with defaults if missing.
files:
# -- Main BIOS binary (required) --
- name: "<user-selected>.bin"
pattern: "*"
- name: ps2-0230a-20080220.bin
required: true
size_range: "4MB-8MB"
source_ref: "pcsx2/ps2/BiosTools.cpp:258-282"
note: >
PCSX2 does not mandate a specific filename. It scans the entire bios/ directory
for any file between 4-8 MB that contains a valid romdir structure (RESET + ROMVER entries).
Common filenames follow the SCPH-XXXXX_BIOS_VYYY_REGION_ZZZ.BIN convention but this is
not enforced. The file is loaded into the 4 MB ROM region of EE memory.
min_size: 4194304
max_size: 8388608
validation: [size]
source_ref: "pcsx2/ps2/BiosTools.cpp:258-362"
note: "Accepts any file 4-8 MB with valid romdir (RESET + ROMVER). Naming convention ps2-VVVVr-YYYYMMDD.bin (version, region, date)."
# -- ROM1 (optional, DVD player) --
- name: "<biosname>.rom1"
pattern: "{biosname}.rom1 or {biosbase}.rom1"
- name: rom1.bin
required: false
max_size: 4194304 # 4 MB (Ps2MemSize::Rom1)
source_ref: "pcsx2/ps2/BiosTools.cpp:214-241"
note: >
DVD player ROM. Loaded via LoadExtraRom("rom1"). PCSX2 tries two naming patterns:
1) Full bios path + ".rom1" appended (e.g. scph70004.bin.rom1)
2) Bios path with extension replaced (e.g. scph70004.rom1)
Mapped to EE memory at ROM1 region (0x1FC00000 + 4MB offset).
Contains DVD player and region detection data (DVDID).
max_size: 4194304
source_ref: "pcsx2/ps2/BiosTools.cpp:214-241,366"
note: "DVD player ROM. Tries {biospath}.rom1 then {biosbase}.rom1. Silently skipped if missing."
# -- ROM2 (optional, Chinese ROM extension) --
- name: "<biosname>.rom2"
pattern: "{biosname}.rom2 or {biosbase}.rom2"
- name: ROM2.BIN
required: false
max_size: 4194304 # 4 MB (Ps2MemSize::Rom2)
source_ref: "pcsx2/ps2/BiosTools.cpp:214-241"
note: >
Chinese ROM extension. Loaded via LoadExtraRom("rom2"). Same naming convention
as rom1: tries appended extension first, then replaced extension.
Only present on Chinese region consoles.
max_size: 4194304
source_ref: "pcsx2/ps2/BiosTools.cpp:214-241,367"
note: "Chinese ROM extension. Same naming convention as rom1. Only present on Chinese region consoles."
# -- NVM / NVRAM (optional, auto-created) --
- name: "<biosname>.nvm"
pattern: "{biosbase}.nvm"
- name: EROM.BIN
required: false
source_ref: "pcsx2/ps2/BiosTools.cpp"
note: "Extended ROM. Present in some BIOS dumps but not loaded by PCSX2 code via LoadExtraRom."
path: null
- name: eeprom.dat
required: false
hle_fallback: true
size: 1024 # NVRAM_SIZE = 1024 bytes
source_ref: "pcsx2/CDVD/CDVD.cpp:160-238"
note: >
EEPROM / NVRAM data. Path derived from BiosPath with extension replaced to ".nvm"
(cdvdGetNVRAMPath). Contains console configuration: language, timezone, iLink ID,
region parameters, OSD settings. Auto-created with defaults if missing.
Two NVM layouts exist: v0.00+ (biosVer 0x000) and v1.70+ (biosVer 0x146).
# -- MEC file (optional, auto-created) --
- name: "<biosname>.mec"
pattern: "{biosbase}.mec"
required: false
hle_fallback: true
size: 4 # u32 s_mecha_version
source_ref: "pcsx2/CDVD/CDVD.cpp:190-204"
note: >
Mechacon (mechanism controller) version file. 4 bytes containing the mecha version
as a u32 value. Auto-created with DEFAULT_MECHA_VERSION (0x00020603) if missing.
Path derived from BiosPath with extension replaced to ".mec".
# -- IRX override (optional, advanced) --
- name: "<custom>.irx"
pattern: "*.irx"
required: false
source_ref: "pcsx2/ps2/BiosTools.cpp:243-256,384-385"
note: >
Custom IOP Reboot eXecutable module. Loaded into ROM at offset 0x3C0000 if
EmuConfig.CurrentIRX is set (path length > 3). Injected at IOP reset (PC=0x1630).
Used for debugging/development, not needed for normal operation.
# -- DEV9 EEPROM (optional, network adapter) --
- name: "eeprom.dat"
required: false
hle_fallback: true
size: 64 # 64 bytes, mmap'd
size: 64
source_ref: "pcsx2/DEV9/DEV9.cpp:110-160"
note: >
DEV9 (network adapter / HDD expansion bay) EEPROM data. Fixed filename "eeprom.dat"
opened from working directory. Contains network adapter configuration.
Falls back to built-in defaults if file not found. Only relevant when using
DEV9 features (online play, HDD).
note: "DEV9 network adapter EEPROM. Falls back to built-in defaults if missing."
common_bios_filenames:
# Japan
- "SCPH-10000_BIOS_V1_JAP_100.BIN"
- "SCPH-15000_BIOS_V3_JAP_120.BIN"
- "SCPH-30000_BIOS_V4_JAP_150.BIN"
- "SCPH-30001R_BIOS_V7_JAP_160.BIN"
- "SCPH-30004R_BIOS_V7_JAP_160.BIN"
- "SCPH-35000_BIOS_V5_JAP_160.BIN"
- "SCPH-50000_BIOS_V9_JAP_170.BIN"
- "SCPH-50004_BIOS_V9_JAP_170.BIN"
- "SCPH-70000_BIOS_V12_JAP_200.BIN"
- "SCPH-75000_BIOS_V14_JAP_220.BIN"
- "SCPH-77000_BIOS_V14_JAP_220.BIN"
- "SCPH-90000_BIOS_V18_JAP_230.BIN"
# USA
- "SCPH-30001_BIOS_V4_USA_150.BIN"
- "SCPH-39001_BIOS_V6_USA_160.BIN"
- "SCPH-50001_BIOS_V9_USA_170.BIN"
- "SCPH-50003_BIOS_V9_USA_170.BIN"
- "SCPH-70002_BIOS_V12_USA_200.BIN"
- "SCPH-70004_BIOS_V12_USA_200.BIN"
- "SCPH-70012_BIOS_V12_USA_200.BIN"
- "SCPH-75001_BIOS_V14_USA_220.BIN"
- "SCPH-77001_BIOS_V14_USA_220.BIN"
- "SCPH-90001_BIOS_V18_USA_230.BIN"
# Europe
- "SCPH-30002_BIOS_V4_EUR_150.BIN"
- "SCPH-30003_BIOS_V4_EUR_150.BIN"
- "SCPH-30004_BIOS_V4_EUR_150.BIN"
- "SCPH-39002_BIOS_V6_EUR_160.BIN"
- "SCPH-39003_BIOS_V6_EUR_160.BIN"
- "SCPH-39004_BIOS_V6_EUR_160.BIN"
- "SCPH-50002_BIOS_V9_EUR_170.BIN"
- "SCPH-50004_BIOS_V9_EUR_170.BIN"
- "SCPH-70002_BIOS_V12_EUR_200.BIN"
- "SCPH-70003_BIOS_V12_EUR_200.BIN"
- "SCPH-70004_BIOS_V12_EUR_200.BIN"
- "SCPH-70008_BIOS_V12_EUR_200.BIN"
- "SCPH-75002_BIOS_V14_EUR_220.BIN"
- "SCPH-75003_BIOS_V14_EUR_220.BIN"
- "SCPH-75004_BIOS_V14_EUR_220.BIN"
- "SCPH-77002_BIOS_V14_EUR_220.BIN"
- "SCPH-77003_BIOS_V14_EUR_220.BIN"
- "SCPH-77004_BIOS_V14_EUR_220.BIN"
- "SCPH-90002_BIOS_V18_EUR_230.BIN"
- "SCPH-90003_BIOS_V18_EUR_230.BIN"
- "SCPH-90004_BIOS_V18_EUR_230.BIN"
# Asia
- "SCPH-50009_BIOS_V9_HK_170.BIN"
- "SCPH-70005_BIOS_V12_HK_200.BIN"
- "SCPH-70006_BIOS_V12_HK_200.BIN"
- "SCPH-70008_BIOS_V12_HK_200.BIN"
# China
- "SCPH-50009_BIOS_V9_CHN_170.BIN"
- "SCPH-70006_BIOS_V12_CHN_200.BIN"
- name: GameIndex.yaml
path: pcsx2/resources/GameIndex.yaml
required: false
mode: libretro
source_ref: "pcsx2/GameDatabase.cpp:48,880"
note: "Game compatibility database. OSD warning if missing."
memory_layout:
ROM: {offset: "0x1FC00000", size: "4 MB", purpose: "Main BIOS binary"}
ROM1: {offset: "ROM + 4MB", size: "4 MB", purpose: "DVD player"}
ROM2: {offset: "ROM + 8MB", size: "4 MB", purpose: "Chinese ROM extension"}
- name: cheats_ws.zip
path: pcsx2/resources/cheats_ws.zip
required: false
mode: libretro
source_ref: "pcsx2/VMManager.cpp:340-353"
note: "Widescreen patches archive."
nvm_layout:
format_0:
applies_to: "BIOS v0.00+"
biosVer: 0x000
config0: 0x280
config1: 0x300
config2: 0x200
consoleId: 0x1C8
ilinkId: 0x1C0
modelNum: 0x1A0
regparams: 0x180
mac: 0x198
format_1:
applies_to: "BIOS v1.70+"
biosVer: 0x146
config0: 0x270
config1: 0x2B0
config2: 0x200
consoleId: 0x1F0
ilinkId: 0x1E0
modelNum: 0x1B0
regparams: 0x180
mac: 0x198
notes: |
PCSX2 is filename-agnostic for the main BIOS. Detection relies on romdir structure
parsing inside the binary itself, not on filename or extension. Any file between 4-8 MB
with a valid romdir (containing at least RESET and ROMVER entries) is accepted.
The ROMVER entry encodes: version (2+2 digits), region letter, console/devel flag,
build date (YYYYMMDD), and is used to determine the BIOS description and region.
Companion files (.nvm, .mec) are auto-created with sane defaults if missing.
ROM1/ROM2 are silently skipped if not found - only the main BIOS binary is strictly required.
PCSX2 no longer ships as a libretro core in official builds. The standalone emulator
is the primary distribution channel.
Devel console BIOSes (< ~2.3 MB) lack the OSD and are handled with NoOSD=true flag.
- name: cheats_ni.zip
path: pcsx2/resources/cheats_ni.zip
required: false
mode: libretro
source_ref: "pcsx2/VMManager.cpp:375-388"
note: "No-interlacing patches archive."

View File

@@ -1,32 +1,21 @@
# RPCS3 emulator firmware profile
# Generated from source analysis of https://github.com/RPCS3/rpcs3
# Commit analyzed: HEAD as of 2026-03-17
emulator: RPCS3
type: standalone
core_classification: official_port
source: "https://github.com/RPCS3/rpcs3"
logo: "https://raw.githubusercontent.com/RPCS3/rpcs3/master/rpcs3/rpcs3.svg"
profiled_date: "2026-03-18"
upstream: "https://github.com/RPCS3/rpcs3"
cores:
- rpcs3
profiled_date: "2026-03-30"
core_version: "0.0.35"
display_name: "RPCS3 (PS3)"
display_name: "Sony - PlayStation 3 (RPCS3)"
systems: [sony-playstation-3]
firmware_file: "PS3UPDAT.PUP"
firmware_source: "https://www.playstation.com/en-us/support/hardware/ps3/system-software/"
firmware_detection: "pup_header" # validates PUP magic bytes, HMAC-SHA1 hash per entry
firmware_install: "extracts dev_flash_* TAR packages from PUP into dev_flash/"
validation:
method: "pup_object"
magic: "SCEUF"
hash_algo: "HMAC-SHA1"
source_ref: "rpcs3/Loader/PUP.cpp:8-114"
note: "PUP file is validated by magic header, file count, HMAC-SHA1 per entry against PUP_KEY"
firmware_version:
path: "dev_flash/vsh/etc/version.txt"
source_ref: "rpcs3/util/sysinfo.cpp:686"
note: "Read at startup, displayed as 'Firmware version: X.XX'. Missing = 'Missing Firmware'"
files:
- name: PS3UPDAT.PUP
required: true
storage: large_file
source_ref: "rpcs3/Loader/PUP.cpp:23-77"
note: "PUP firmware package. Validated via SCEUF magic + HMAC-SHA1 per entry. Extracted to dev_flash/ at install time."
# dev_flash filesystem layout extracted from PUP
dev_flash:

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -3,7 +3,7 @@
"platform": "emudeck",
"display_name": "EmuDeck",
"version": "1.0",
"generated": "2026-03-29T14:02:36Z",
"generated": "2026-03-30T09:46:25Z",
"base_destination": "bios",
"detect": [
{
@@ -50,8 +50,8 @@
}
}
],
"total_files": 270,
"total_size": 913637720,
"total_files": 290,
"total_size": 717377323,
"files": [
{
"dest": "colecovision.rom",
@@ -473,24 +473,6 @@
"Dolphin"
]
},
{
"dest": "scph1000.bin",
"sha1": "343883a7b555646da8cee54aadd2795b6e7dd070",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph1000.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph1001.bin",
"sha1": "10155d8d6e6e832d6ea66db9bc098321fb5e8ebf",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph1001.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph1002a.bin",
"sha1": "20b98f3d80f11cbf5a7bfd0779b0e63760ecc62c",
@@ -500,132 +482,6 @@
"DuckStation"
]
},
{
"dest": "scph1002b.bin",
"sha1": "76cf6b1b2a7c571a6ad07f2bac0db6cd8f71e2cc",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph1002b.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph1002c.bin",
"sha1": "b6a11579caef3875504fcf3831b8e3922746df2c",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph1002c.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "dtlh1100.bin",
"sha1": "73107d468fc7cb1d2c5b18b269715dd889ecef06",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/dtlh1100.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph3000.bin",
"sha1": "b06f4a861f74270be819aa2a07db8d0563a7cc4e",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph3000.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph1001_v20.bin",
"sha1": "649895efd79d14790eabb362e94eb0622093dfb9",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph1001_v20.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph3500.bin",
"sha1": "e38466a4ba8005fba7e9e3c7b9efeba7205bee3f",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph3500.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph1001_v21.bin",
"sha1": "ca7af30b50d9756cbd764640126c454cff658479",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph1001_v21.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph5000.bin",
"sha1": "ffa7f9a7fb19d773a0c3985a541c8e5623d2c30d",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph5000.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph7000.bin",
"sha1": "77b10118d21ac7ffa9b35f9c4fd814da240eb3e9",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph7000.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph7000w.bin",
"sha1": "1b0dbdb23da9dc0776aac58d0755dc80fea20975",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph7000w.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph7001.bin",
"sha1": "14df4f6c1e367ce097c11deae21566b4fe5647a9",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph7001.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph7002.bin",
"sha1": "8d5de56a79954f29e9006929ba3fed9b6a418c1d",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph7002.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph100.bin",
"sha1": "339a48f4fcf63e10b5b867b8c93cfd40945faf6c",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph100.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph101_v44.bin",
"sha1": "7771d6e90980408f753891648685def6dd42ef6d",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph101_v44.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "scph101_v45.bin",
"sha1": "dcffe16bd90a723499ad46c641424981338d8378",
@@ -653,15 +509,6 @@
"DuckStation"
]
},
{
"dest": "scph1000r.bin",
"sha1": "7082bd57141fa0007b3adcd031f7ba23a20108a0",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/scph1000r.bin",
"cores": [
"DuckStation"
]
},
{
"dest": "ps2_scph18000.bin",
"sha1": "d7d6be084f51354bc951d8fa2d8d912aa70abc5e",
@@ -2111,6 +1958,42 @@
"MAME"
]
},
{
"dest": "astrocde.zip",
"sha1": "8641b09be090c0dc45f4ee5459fec3cc6fb9d78e",
"size": 6800,
"repo_path": "bios/Bally/Astrocade/astrocde.zip",
"cores": [
"MAME"
]
},
{
"dest": "apple2gs.zip",
"sha1": "799e2fc90d6bfd8cb74e331e04d5afd36f2f21a1",
"size": 174124,
"repo_path": "bios/Apple/Apple II/apple2gs.zip",
"cores": [
"MAME"
]
},
{
"dest": "casloopy.zip",
"sha1": "144ea9d0f20113632dde6d21c62a01c02cf7666d",
"size": 478191,
"repo_path": "bios/Casio/Loopy/casloopy.zip",
"cores": [
"MAME"
]
},
{
"dest": "pv2000.zip",
"sha1": "c18f460301177b64bbfcbedc3010e734d0803d4c",
"size": 13622,
"repo_path": "bios/Casio/PV-2000/pv2000.zip",
"cores": [
"MAME"
]
},
{
"dest": "adam.zip",
"sha1": "dc4b77cd3ac45cf448e427eff5c0a7b138fd9d81",
@@ -2219,6 +2102,303 @@
"MAME"
]
},
{
"dest": "beena.zip",
"sha1": "f63eca9a1b1e92ee1582ff5e4c0db55193f97e33",
"size": 85176,
"repo_path": "bios/Sega/Beena/beena.zip",
"cores": [
"MAME"
]
},
{
"dest": "rx78.zip",
"sha1": "b3a26e21574395a279a37922238802b349c303b7",
"size": 6397,
"repo_path": "bios/Bandai/RX-78/rx78.zip",
"cores": [
"MAME"
]
},
{
"dest": "segaai.zip",
"sha1": "39a0fec854e438f6757882604a8dc56e62e401d9",
"size": 343220,
"repo_path": "bios/Sega/AI/segaai.zip",
"cores": [
"MAME"
]
},
{
"dest": "segaai_soundbox.zip",
"sha1": "d90e0b3545e60c59c20a7fb32d0785f6c3abbf60",
"size": 47380,
"repo_path": "bios/Arcade/MAME/segaai_soundbox.zip",
"cores": [
"MAME"
]
},
{
"dest": "lynx48k.zip",
"sha1": "64947e9b7d17870839aba5d93217183d480ff897",
"size": 25078,
"repo_path": "bios/Camputers/Lynx/lynx48k.zip",
"cores": [
"MAME"
]
},
{
"dest": "lynx96k.zip",
"sha1": "df95ea702606e88b4c906a0233c8fec54c02ff01",
"size": 32888,
"repo_path": "bios/Camputers/Lynx/lynx96k.zip",
"cores": [
"MAME"
]
},
{
"dest": "lynx128k.zip",
"sha1": "5300a352007976102d61d53f1e8c48063d2f3026",
"size": 24796,
"repo_path": "bios/Camputers/Lynx/lynx128k.zip",
"cores": [
"MAME"
]
},
{
"dest": "crvision.zip",
"sha1": "87526fb12fabbcc01292dd01dc698fdc762ab7dc",
"size": 441619,
"repo_path": "bios/VTech/CreatiVision/crvision.zip",
"cores": [
"MAME"
]
},
{
"dest": "laser310.zip",
"sha1": "9fa5f366c4ec43d7c23f03f054733894bf42912f",
"size": 42630,
"repo_path": "bios/VTech/Laser 310/laser310.zip",
"cores": [
"MAME"
]
},
{
"dest": "socrates.zip",
"sha1": "6bbaa8f73027eaca2c6988ee3b9b3f3b0b29d18e",
"size": 218299,
"repo_path": "bios/VTech/Socrates/socrates.zip",
"cores": [
"MAME"
]
},
{
"dest": "vsmile.zip",
"sha1": "65e526ae3e4795a9186c53c7428d8b946170befe",
"size": 3078731,
"repo_path": "bios/VTech/V.Smile/vsmile.zip",
"cores": [
"MAME"
]
},
{
"dest": "gamate.zip",
"sha1": "e99667ea5cfe6a5eceb53faaa39cdda0cbf69c69",
"size": 5072,
"repo_path": "bios/Bit Corporation/Gamate/gamate.zip",
"cores": [
"MAME"
]
},
{
"dest": "gamepock.zip",
"sha1": "83e94c56f2fc4ab60de94021a1283cca09d65ee5",
"size": 3455,
"repo_path": "bios/Epoch/Game Pocket/gamepock.zip",
"cores": [
"MAME"
]
},
{
"dest": "gmaster.zip",
"sha1": "a89263689f19e046dab6dadea24c5d07270548a6",
"size": 1686,
"repo_path": "bios/Hartung/Game Master/gmaster.zip",
"cores": [
"MAME"
]
},
{
"dest": "gamecom.zip",
"sha1": "e0f5e2eced447abf9948342b2facc40179f0f527",
"size": 145138,
"repo_path": "bios/Tiger/Game.com/gamecom.zip",
"cores": [
"MAME"
]
},
{
"dest": "gp32.zip",
"sha1": "35def9830e797fb0788740b5da0a7202933554df",
"size": 1378301,
"repo_path": "bios/GamePark/GP32/gp32.zip",
"cores": [
"MAME"
]
},
{
"dest": "fm7.zip",
"sha1": "1c29ba16151255cd916f49ea25570956f1353b71",
"size": 104276,
"repo_path": "bios/Fujitsu/FM Towns/fm7.zip",
"cores": [
"MAME"
]
},
{
"dest": "fm77av.zip",
"sha1": "b558bb5472cfa727dccab92229c66afa0d64404f",
"size": 46921,
"repo_path": "bios/Fujitsu/FM Towns/fm77av.zip",
"cores": [
"MAME"
]
},
{
"dest": "pegasus.zip",
"sha1": "fc10ef402bcac78c70e1cff57d51613fa12202f9",
"size": 27305,
"repo_path": "bios/Arcade/MAME/pegasus.zip",
"cores": [
"MAME"
]
},
{
"dest": "pcw8256.zip",
"sha1": "f8a11b4e03bb8c3caee604ca591676eab6535d1b",
"size": 1710,
"repo_path": "bios/Amstrad/PCW/pcw8256.zip",
"cores": [
"MAME"
]
},
{
"dest": "pcw9512.zip",
"sha1": "844988dc5a36917b002d0140d3a7692d1407f783",
"size": 2625,
"repo_path": "bios/Amstrad/PCW/pcw9512.zip",
"cores": [
"MAME"
]
},
{
"dest": "vis.zip",
"sha1": "beff39b4edb449c66b704e1c64ee6ed58e30a868",
"size": 605047,
"repo_path": "bios/Tandy/VIS/vis.zip",
"cores": [
"MAME"
]
},
{
"dest": "trs80.zip",
"sha1": "a9b8931c42303e8dfb475112fc18ff200b7252ce",
"size": 14558,
"repo_path": "bios/Tandy/TRS-80/trs80.zip",
"cores": [
"MAME"
]
},
{
"dest": "trs80m3.zip",
"sha1": "b804a031c8db6def59e077a4b6938dcac25093d7",
"size": 32854,
"repo_path": "bios/Tandy/TRS-80/trs80m3.zip",
"cores": [
"MAME"
]
},
{
"dest": "trs80m4.zip",
"sha1": "bc58ac28be4854fb198b895c7fd999f599b5a961",
"size": 25371,
"repo_path": "bios/Tandy/TRS-80/trs80m4.zip",
"cores": [
"MAME"
]
},
{
"dest": "trs80m4p.zip",
"sha1": "497eea6e0ed80501ca3beee714d71a5815495b07",
"size": 4880,
"repo_path": "bios/Tandy/TRS-80/trs80m4p.zip",
"cores": [
"MAME"
]
},
{
"dest": "ti99_4a.zip",
"sha1": "e05575b630bea7ff98b9ca1f083d745abb3110b6",
"size": 21377,
"repo_path": "bios/Texas Instruments/TI-99/ti99_4a.zip",
"cores": [
"MAME"
]
},
{
"dest": "ti99_speech.zip",
"sha1": "b58fe2dbbc254d363c2ec4a459e6ec1d91b2ac86",
"size": 31206,
"repo_path": "bios/Arcade/Arcade/ti99_speech.zip",
"cores": [
"MAME"
]
},
{
"dest": "tutor.zip",
"sha1": "273dbb7a93eb0ab83c2e13e9db51b897cc18f838",
"size": 34653,
"repo_path": "bios/Tomy/Tutor/tutor.zip",
"cores": [
"MAME"
]
},
{
"dest": "vg5k.zip",
"sha1": "5ad489a58fc457d2bfb7a348f4a312c3b49daad1",
"size": 26356,
"repo_path": "bios/Arcade/MAME/vg5k.zip",
"cores": [
"MAME"
]
},
{
"dest": "qsound.zip",
"sha1": "f1ccedb2c6e8f15cfebab52c3923e4b35f4df43f",
"size": 2654,
"repo_path": "bios/Arcade/Arcade/qsound.zip",
"cores": [
"MAME"
]
},
{
"dest": "ym2413.zip",
"sha1": "900e8c6cd152704a79905b6838fe3229c9499f8c",
"size": 286,
"repo_path": "bios/Arcade/MAME/ym2413.zip",
"cores": [
"MAME"
]
},
{
"dest": "ym2608.zip",
"sha1": "06fc753d015b43ca1787f4cfd9331b1674202e64",
"size": 7609,
"repo_path": "bios/Arcade/Arcade/ym2608.zip",
"cores": [
"MAME"
]
},
{
"dest": "dsi_bios9.bin",
"sha1": "db61fa39ddbc5f5ed71fb19cda47609ef0201723",
@@ -2246,17 +2426,6 @@
"melonDS"
]
},
{
"dest": "dsi_nand.bin",
"sha1": "b48f44194fe918aaaec5298861479512b581d661",
"size": 251658304,
"repo_path": ".cache/large/dsi_nand.bin",
"cores": [
"melonDS"
],
"storage": "release",
"release_asset": "dsi_nand.bin"
},
{
"dest": "dsi_sd_card.bin",
"sha1": "3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3",
@@ -2365,6 +2534,17 @@
"shadps4"
]
},
{
"dest": "psvita/PSP2UPDAT.PUP",
"sha1": "3ae832c9800fcaa007eccfc48f24242967c111f8",
"size": 56768512,
"repo_path": "bios/Sony/PlayStation Vita/.variants/PSP2UPDAT.PUP",
"cores": [
"Vita3K"
],
"storage": "release",
"release_asset": "PSP2UPDAT.PUP"
},
{
"dest": "Complex_4627.bin",
"sha1": "3944392c954cfb176d4210544e88353b3c5d36b1",

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -3,7 +3,7 @@
"platform": "romm",
"display_name": "RomM",
"version": "1.0",
"generated": "2026-03-29T14:04:16Z",
"generated": "2026-03-30T09:49:20Z",
"base_destination": "bios",
"detect": [
{
@@ -13,8 +13,8 @@
}
],
"standalone_copies": [],
"total_files": 545,
"total_size": 1197706137,
"total_files": 638,
"total_size": 1316397979,
"files": [
{
"dest": "3do/3do_arcade_saot.bin",
@@ -2623,12 +2623,21 @@
"cores": null
},
{
"dest": "psxonpsp660.bin",
"sha1": "96880d1ca92a016ff054be5159bb06fe03cb4e14",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/psxonpsp660.bin",
"dest": "BB01R4_OS.ROM",
"sha1": "decde89fbae90adb591ad2fc553d35f49030c129",
"size": 16384,
"repo_path": "bios/Atari/400-800/BB01R4_OS.ROM",
"cores": [
"Beetle PSX (Mednafen PSX)"
"Atari800"
]
},
{
"dest": "XEGAME.ROM",
"sha1": "a107db7f16a1129cf9d933c9cf4f013b068c9e82",
"size": 8192,
"repo_path": "bios/Atari/400-800/XEGAME.ROM",
"cores": [
"Atari800"
]
},
{
@@ -2640,6 +2649,98 @@
"Beetle PSX (Mednafen PSX)"
]
},
{
"dest": "openbios.bin",
"sha1": "389df7981873d9e6e46c84c20cd43af0e4226cf8",
"size": 524288,
"repo_path": "bios/Sony/PlayStation/openbios.bin",
"cores": [
"Beetle PSX (Mednafen PSX)"
]
},
{
"dest": "cromwell_1024.bin",
"sha1": "4e1c2c2ee308ca4591542b3ca48653f65fae6e0f",
"size": 1048576,
"repo_path": "bios/Microsoft/Xbox/cromwell_1024.bin",
"cores": [
"DirectXBox"
]
},
{
"dest": "GC/dsp_rom.bin",
"sha1": "f4f683a49d7eb4155566f793f2c1c27e90159992",
"size": 8192,
"repo_path": "bios/Nintendo/GameCube/Sys/GC/dsp_rom.bin",
"cores": [
"Dolphin"
]
},
{
"dest": "GC/dsp_coef.bin",
"sha1": "c116d867ba001dcd6bf6d399ff4bf38d340f556c",
"size": 4096,
"repo_path": "bios/Nintendo/GameCube/.variants/dsp_coef.bin.c116d867",
"cores": [
"Dolphin"
]
},
{
"dest": "Wii/shared2/sys/SYSCONF",
"sha1": "3256c026284a24fb99d2ec1558d95db3b5dcc2e9",
"size": 16384,
"repo_path": "bios/Nintendo/Wii/SYSCONF",
"cores": [
"Dolphin"
]
},
{
"dest": "Wii/title/00000001/00000002/data/setting.txt",
"sha1": "077a31ce116de322f089a627c5eb7ff5c2236a5d",
"size": 256,
"repo_path": "bios/Nintendo/Wii/setting.txt",
"cores": [
"Dolphin"
]
},
{
"dest": "Load/WiiSD.raw",
"sha1": "90d83d6c084deceeeb0466ac00723723ccfd0c6d",
"size": 134217728,
"repo_path": "bios/Nintendo/Wii/WiiSD.raw",
"cores": [
"Dolphin"
],
"storage": "release",
"release_asset": "WiiSD.raw"
},
{
"dest": "Wii/clientca.pem",
"sha1": "4b937d4e81de11574b926386dd5f768aa23bf177",
"size": 1005,
"repo_path": "bios/Nintendo/Wii/clientca.pem",
"cores": [
"Dolphin"
]
},
{
"dest": "Wii/clientcakey.pem",
"sha1": "a2c11886e6d12a135e1d5eb50d5fe92e028577d9",
"size": 609,
"repo_path": "bios/Nintendo/Wii/clientcakey.pem",
"cores": [
"Dolphin"
]
},
{
"dest": "Wii/rootca.pem",
"sha1": "b5229455dd26c1f53c73060e9089b391389e1f75",
"size": 897,
"repo_path": "bios/Nintendo/Wii/rootca.pem",
"cores": [
"Dolphin"
]
},
{
"dest": "MT32_CONTROL.ROM",
"sha1": "b083518fffb7f66b03c23b7eb4f868e62dc5a987",
@@ -2694,6 +2795,33 @@
"DOSBox Pure"
]
},
{
"dest": "ep128emu/roms/exdos14isdos10uk.rom",
"sha1": "b82e21b6e3214432b6dc13f650e97de88fc90a72",
"size": 32768,
"repo_path": "bios/Enterprise/64-128/exdos14isdos10uk.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/epdos16f.rom",
"sha1": "9a9eac31e601a1ab0f7a5d2b13175d092fa84bd3",
"size": 32768,
"repo_path": "bios/Enterprise/64-128/epdos16f.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/zx128.rom",
"sha1": "16375d42ea109b47edded7a16028de7fdb3013a1",
"size": 32768,
"repo_path": "bios/Enterprise/64-128/zx128.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "fbalpha2012/hiscore.dat",
"sha1": "7381472bf046126257e51a0124e4553282f020e5",
@@ -3063,6 +3191,53 @@
"Handy"
]
},
{
"dest": "hatari/BOOT.ST",
"sha1": "5bcabba35bb8fbfe5a65b85efccf5ed657388308",
"size": 737280,
"repo_path": "bios/Atari/ST/hatari/BOOT.ST",
"cores": [
"Hatari"
]
},
{
"dest": "Wii/sd.raw",
"sha1": "8c8134f08b2e3baa603206ede30d3935365009b8",
"size": 134217728,
"repo_path": "bios/Nintendo/Wii/sd.raw",
"cores": [
"Ishiiruka"
],
"storage": "release",
"release_asset": "sd.raw"
},
{
"dest": "pcsx2/resources/GameIndex.yaml",
"sha1": "b22389650f6c0a1e276b48213fed8a9c1c6476ce",
"size": 2669341,
"repo_path": "bios/Sony/PlayStation 2/GameIndex.yaml",
"cores": [
"LRPS2"
]
},
{
"dest": "pcsx2/resources/cheats_ws.zip",
"sha1": "773b4279d8c8e182dc57e444dbf448a1d115e7e0",
"size": 1273296,
"repo_path": "bios/Sony/PlayStation 2/cheats_ws.zip",
"cores": [
"LRPS2"
]
},
{
"dest": "pcsx2/resources/cheats_ni.zip",
"sha1": "dec5e4f137890c338babe13cfb9d41ff869fa721",
"size": 42525,
"repo_path": "bios/Sony/PlayStation 2/cheats_ni.zip",
"cores": [
"LRPS2"
]
},
{
"dest": "mame2003/cheat.dat",
"sha1": "32fc78415114a976af9c2ee53ea13de9f40b55ee",
@@ -3234,17 +3409,6 @@
"melonDS"
]
},
{
"dest": "dsi_nand.bin",
"sha1": "b48f44194fe918aaaec5298861479512b581d661",
"size": 251658304,
"repo_path": ".cache/large/dsi_nand.bin",
"cores": [
"melonDS"
],
"storage": "release",
"release_asset": "dsi_nand.bin"
},
{
"dest": "dsi_sd_card.bin",
"sha1": "3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3",
@@ -3254,6 +3418,15 @@
"melonDS"
]
},
{
"dest": "MacIIx.ROM",
"sha1": "753b94351d94c369616c2c87b19d568dc5e2764e",
"size": 262144,
"repo_path": "bios/Apple/Macintosh II/MacIIx.ROM",
"cores": [
"Mini vMac"
]
},
{
"dest": "Mupen64plus/IPL.n64",
"sha1": "bf861922dcb78c316360e3e742f4f70ff63c9bc3",
@@ -3272,6 +3445,33 @@
"Mupen64Plus-Next"
]
},
{
"dest": "np2/bios9821.rom",
"sha1": "2e92346727b0355bc1ec9a7ded1b444a4917f2b9",
"size": 98304,
"repo_path": "bios/NEC/PC-98/bios9821.rom",
"cores": [
"nekop2"
]
},
{
"dest": "np2/scsi.rom",
"sha1": "3d7166f05daad1b022fa04c2569e788580158095",
"size": 8192,
"repo_path": "bios/NEC/PC-98/scsi.rom",
"cores": [
"nekop2"
]
},
{
"dest": "np2/sasi.rom",
"sha1": "b607707d74b5a7d3ba211825de31a8f32aec8146",
"size": 4096,
"repo_path": "bios/NEC/PC-98/sasi.rom",
"cores": [
"nekop2"
]
},
{
"dest": "custom.pal",
"sha1": "c7635019127aa28d84d28ae48304bfca6d5baefb",
@@ -3281,6 +3481,33 @@
"Nestopia UE"
]
},
{
"dest": "np2kai/ide.rom",
"sha1": "0877ffb4b4d1c18283468be3579b72ed8c22e3ac",
"size": 8192,
"repo_path": "bios/NEC/PC-98/ide.rom",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/pci.rom",
"sha1": "1246203bebf9f04e3bac2df7fc64719304f9f1bd",
"size": 32768,
"repo_path": "bios/NEC/PC-98/pci.rom",
"cores": [
"NP2kai"
]
},
{
"dest": "eeprom.dat",
"sha1": "ffc6261e487efa8c7442069f71acfc4aa826993d",
"size": 64,
"repo_path": "bios/Sony/PlayStation 2/eeprom.dat",
"cores": [
"PCSX2"
]
},
{
"dest": "carthw.cfg",
"sha1": "d3fe0b958705e4cb5eeb7822e60d474204df4bd3",
@@ -3515,6 +3742,15 @@
"PUAE (P-UAE)"
]
},
{
"dest": "keropi/cgrom.tmp",
"sha1": "8d72c5b4d63bb14c5dbdac495244d659aa1498b6",
"size": 786432,
"repo_path": "bios/Sharp/X68000/cgrom.dat",
"cores": [
"px68k"
]
},
{
"dest": "same_cdi/bios/cdimono1.zip",
"sha1": "5d0b1b55b0d0958a5c9069c3219d4da5a87a6b93",
@@ -3561,12 +3797,93 @@
]
},
{
"dest": "BIOS.col",
"sha1": "45bedc4cbdeac66c7df59e9e599195c778d86a92",
"size": 8192,
"repo_path": "bios/Coleco/ColecoVision/BIOS.col",
"dest": "squirreljme-0.3.0-fast.jar",
"sha1": "7c4cd0a5451eedeac9b328f48408dbc312198ccf",
"size": 2367241,
"repo_path": "bios/Other/SquirrelJME/squirreljme-0.3.0-fast.jar",
"cores": [
"SMS Plus GX"
"SquirrelJME"
]
},
{
"dest": "squirreljme-0.3.0.jar",
"sha1": "8a2ddc37df2366d4e205e9904cd8bac22db2afd5",
"size": 15181774,
"repo_path": "bios/Java/J2ME/squirreljme.sqc",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme-0.3.0-test.jar",
"sha1": "600cf0440594b0c69339ba4b626a47288b342d77",
"size": 3396757,
"repo_path": "bios/Java/J2ME/squirreljme-test.jar",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme-0.3.0-slow.jar",
"sha1": "3a23de6999b94fc088bdd9c9e4a73781a1d42233",
"size": 3129653,
"repo_path": "bios/Other/SquirrelJME/squirreljme-0.3.0-slow.jar",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme-0.3.0-slow-test.jar",
"sha1": "b411c128bfd8411f339a49bbcb1c9e97189c2161",
"size": 3772058,
"repo_path": "bios/Other/SquirrelJME/squirreljme-0.3.0-slow-test.jar",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme-fast.jar",
"sha1": "8a2ddc37df2366d4e205e9904cd8bac22db2afd5",
"size": 15181774,
"repo_path": "bios/Java/J2ME/squirreljme.sqc",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme.jar",
"sha1": "6933bbe1fc29f58581a7f64f859f756db9ef78a7",
"size": 4700395,
"repo_path": "bios/Java/J2ME/squirreljme.jar",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme-test.jar",
"sha1": "600cf0440594b0c69339ba4b626a47288b342d77",
"size": 3396757,
"repo_path": "bios/Java/J2ME/squirreljme-test.jar",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme-slow.jar",
"sha1": "8a2ddc37df2366d4e205e9904cd8bac22db2afd5",
"size": 15181774,
"repo_path": "bios/Java/J2ME/squirreljme.sqc",
"cores": [
"SquirrelJME"
]
},
{
"dest": "squirreljme-slow-test.jar",
"sha1": "8a2ddc37df2366d4e205e9904cd8bac22db2afd5",
"size": 15181774,
"repo_path": "bios/Java/J2ME/squirreljme.sqc",
"cores": [
"SquirrelJME"
]
},
{
@@ -3722,6 +4039,51 @@
"VICE xvic"
]
},
{
"dest": "xmil/IPLROM.X1",
"sha1": "c4db9a6e99873808c8022afd1c50fef556a8b44d",
"size": 4096,
"repo_path": "bios/Sharp/X1/.variants/IPLROM.X1.c4db9a6e",
"cores": [
"X Millennium"
]
},
{
"dest": "xmil/IPLROM.X1T",
"sha1": "44620f57a25f0bcac2b57ca2b0f1ebad3bf305d3",
"size": 32768,
"repo_path": "bios/Sharp/X1/.variants/IPLROM.X1T.44620f57",
"cores": [
"X Millennium"
]
},
{
"dest": "xmil/FNT0816.X1",
"sha1": "4f06d20c997a79ee6af954b69498147789bf1847",
"size": 4096,
"repo_path": "bios/Sharp/X1/FNT0816.X1",
"cores": [
"X Millennium"
]
},
{
"dest": "xmil/FNT1616.X1",
"sha1": "b9e6c320611f0842df6f45673c47c3e23bc14272",
"size": 306176,
"repo_path": "bios/Sharp/X1/FNT1616.X1",
"cores": [
"X Millennium"
]
},
{
"dest": "xbox_hdd.qcow2",
"sha1": "9da5f9ecfb1c9c32efa616f0300d02e8f702244d",
"size": 1638400,
"repo_path": "bios/Microsoft/Xbox/xbox_hdd.qcow2",
"cores": [
"Xemu"
]
},
{
"dest": "channelf.zip",
"sha1": "1cb23b462b990241013deb4b5e07ce741af28267",
@@ -4127,6 +4489,213 @@
"FinalBurn Neo"
]
},
{
"dest": "GC/USA/IPL.bin",
"sha1": "a1837968288253ed541f2b11440b68f5a9b33875",
"size": 2097152,
"repo_path": "bios/Nintendo/GameCube/GC/JAP/IPL.bin",
"cores": [
"Dolphin"
]
},
{
"dest": "GC/EUR/IPL.bin",
"sha1": "a1837968288253ed541f2b11440b68f5a9b33875",
"size": 2097152,
"repo_path": "bios/Nintendo/GameCube/GC/JAP/IPL.bin",
"cores": [
"Dolphin"
]
},
{
"dest": "GC/JAP/IPL.bin",
"sha1": "a1837968288253ed541f2b11440b68f5a9b33875",
"size": 2097152,
"repo_path": "bios/Nintendo/GameCube/GC/JAP/IPL.bin",
"cores": [
"Dolphin"
]
},
{
"dest": "GBA/gba_bios.bin",
"sha1": "300c20df6731a33952ded8c436f7f186d25d3492",
"size": 16384,
"repo_path": "bios/Nintendo/Game Boy Advance/GBA_bios.rom",
"cores": [
"Dolphin"
]
},
{
"dest": "ep128emu/roms/exos21.rom",
"sha1": "55315b20fecb4441a07ee4bc5dc7153f396e0a2e",
"size": 32768,
"repo_path": "bios/Enterprise/64-128/exos21.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/exos20.rom",
"sha1": "6033a0535136c40c47137e4d1cd9273c06d5fdff",
"size": 32768,
"repo_path": "bios/Enterprise/64-128/exos20.bin",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/exos24uk.rom",
"sha1": "cf12e971623a54bf8c4f891ca3a36d969f205c49",
"size": 65536,
"repo_path": "bios/Enterprise/64-128/exos24uk.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/basic21.rom",
"sha1": "03bbb386cf530e804363acdfc1d13e64cf28af2e",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/basic21.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/basic20.rom",
"sha1": "61d0987b906146e21b94f265d5b51b4938c986a9",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/basic20.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/exdos13.rom",
"sha1": "cb43ab3676b93c279f1ed8ffcb0d4dcd4b34e631",
"size": 32768,
"repo_path": "bios/Enterprise/64-128/exdos13.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/epfileio.rom",
"sha1": "2f9077bcd89b1ec42dbdcd55d335bdbaf361eff3",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/epfileio.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/zt19uk.rom",
"sha1": "b7af62f0bc95fdca4b31d236f8327dafc80f83b7",
"size": 32768,
"repo_path": "bios/Enterprise/64-128/zt19uk.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/hun.rom",
"sha1": "325a5e28c2a0d896711f8829e7ff14fed5dd4103",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/hun.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/brd.rom",
"sha1": "f34f0c330b44dbf2548329bea954d5991dec30ca",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/brd.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/tvc22_sys.rom",
"sha1": "f2572ee83d09fc08f4de4a62f101c8bb301a9505",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/tvc22_sys.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/tvc22_ext.rom",
"sha1": "abf119cf947ea32defd08b29a8a25d75f6bd4987",
"size": 8192,
"repo_path": "bios/Enterprise/64-128/tvc22_ext.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/tvcfileio.rom",
"sha1": "98889c3a56b11dedf077f866ed2e12d51b604113",
"size": 8192,
"repo_path": "bios/Enterprise/64-128/tvcfileio.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/tvc_dos12d.rom",
"sha1": "072c6160d4e7d406f5d8f5b1b66066c797d35561",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/tvc_dos12d.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/cpc464.rom",
"sha1": "56d39c463da60968d93e58b4ba0e675829412a20",
"size": 32768,
"repo_path": "bios/Amstrad/CPC/cpc464.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/cpc664.rom",
"sha1": "073a7665527b5bd8a148747a3947dbd3328682c8",
"size": 32768,
"repo_path": "bios/Amstrad/CPC/cpc664.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/cpc6128.rom",
"sha1": "5977adbad3f7c1e0e082cd02fe76a700d9860c30",
"size": 32768,
"repo_path": "bios/Amstrad/CPC/cpc6128.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/cpc_amsdos.rom",
"sha1": "39102c8e9cb55fcc0b9b62098780ed4a3cb6a4bb",
"size": 16384,
"repo_path": "bios/Amstrad/CPC/amsdos.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "ep128emu/roms/zx48.rom",
"sha1": "5ea7c2b824672e914525d1d5c419d71b84a426a2",
"size": 16384,
"repo_path": "bios/Enterprise/64-128/zx48.rom",
"cores": [
"ep128emu-core"
]
},
{
"dest": "fbneo/hiscore.dat",
"sha1": "7381472bf046126257e51a0124e4553282f020e5",
@@ -4180,6 +4749,276 @@
"cores": [
"MAME 2003-Plus"
]
},
{
"dest": "np2/bios.rom",
"sha1": "1f5f7013f18c08ff50d7942e76c4fbd782412414",
"size": 65536,
"repo_path": "bios/IBM/PC/ibmpcjr/bios.rom",
"cores": [
"nekop2"
]
},
{
"dest": "np2/font.bmp",
"sha1": "b4f14e58030ed40fff2dc312b58ea4440bdf8cc5",
"size": 524350,
"repo_path": "bios/NEC/PC-98/font.bmp",
"cores": [
"nekop2"
]
},
{
"dest": "np2/sound.rom",
"sha1": "d5dbc4fea3b8367024d363f5351baecd6adcd8ef",
"size": 16384,
"repo_path": "bios/NEC/PC-98/sound.rom",
"cores": [
"nekop2"
]
},
{
"dest": "np2/2608_bd.wav",
"sha1": "0a56c142ef40cec50f3ee56a6e42d0029c9e2818",
"size": 19192,
"repo_path": "bios/NEC/PC-98/2608_bd.wav",
"cores": [
"nekop2"
]
},
{
"dest": "np2/2608_sd.wav",
"sha1": "3c79663ef74c0b0439d13351326eb1c52a657008",
"size": 15558,
"repo_path": "bios/NEC/PC-98/2608_sd.wav",
"cores": [
"nekop2"
]
},
{
"dest": "np2/2608_top.wav",
"sha1": "aa4a8f766a86b830687d5083fd3b9db0652f46fc",
"size": 57016,
"repo_path": "bios/NEC/PC-98/2608_top.wav",
"cores": [
"nekop2"
]
},
{
"dest": "np2/2608_hh.wav",
"sha1": "12f676cef249b82480b6f19c454e234b435ca7b6",
"size": 36722,
"repo_path": "bios/NEC/PC-98/2608_hh.wav",
"cores": [
"nekop2"
]
},
{
"dest": "np2/2608_tom.wav",
"sha1": "9513fb4a3f41e75a972a273a5104cbd834c1e2c5",
"size": 23092,
"repo_path": "bios/NEC/PC-98/2608_tom.wav",
"cores": [
"nekop2"
]
},
{
"dest": "np2/2608_rim.wav",
"sha1": "c65592330c9dd84011151daed52f9aec926b7e56",
"size": 5288,
"repo_path": "bios/NEC/PC-98/2608_rim.wav",
"cores": [
"nekop2"
]
},
{
"dest": "np2kai/bios.rom",
"sha1": "1f5f7013f18c08ff50d7942e76c4fbd782412414",
"size": 65536,
"repo_path": "bios/IBM/PC/ibmpcjr/bios.rom",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/font.bmp",
"sha1": "b4f14e58030ed40fff2dc312b58ea4440bdf8cc5",
"size": 524350,
"repo_path": "bios/NEC/PC-98/font.bmp",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/sound.rom",
"sha1": "d5dbc4fea3b8367024d363f5351baecd6adcd8ef",
"size": 16384,
"repo_path": "bios/NEC/PC-98/sound.rom",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/2608_bd.wav",
"sha1": "0a56c142ef40cec50f3ee56a6e42d0029c9e2818",
"size": 19192,
"repo_path": "bios/NEC/PC-98/2608_bd.wav",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/2608_sd.wav",
"sha1": "3c79663ef74c0b0439d13351326eb1c52a657008",
"size": 15558,
"repo_path": "bios/NEC/PC-98/2608_sd.wav",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/2608_top.wav",
"sha1": "aa4a8f766a86b830687d5083fd3b9db0652f46fc",
"size": 57016,
"repo_path": "bios/NEC/PC-98/2608_top.wav",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/2608_hh.wav",
"sha1": "12f676cef249b82480b6f19c454e234b435ca7b6",
"size": 36722,
"repo_path": "bios/NEC/PC-98/2608_hh.wav",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/2608_tom.wav",
"sha1": "9513fb4a3f41e75a972a273a5104cbd834c1e2c5",
"size": 23092,
"repo_path": "bios/NEC/PC-98/2608_tom.wav",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/2608_rim.wav",
"sha1": "c65592330c9dd84011151daed52f9aec926b7e56",
"size": 5288,
"repo_path": "bios/NEC/PC-98/2608_rim.wav",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/scsi.rom",
"sha1": "3d7166f05daad1b022fa04c2569e788580158095",
"size": 8192,
"repo_path": "bios/NEC/PC-98/scsi.rom",
"cores": [
"NP2kai"
]
},
{
"dest": "np2kai/sasi.rom",
"sha1": "b607707d74b5a7d3ba211825de31a8f32aec8146",
"size": 4096,
"repo_path": "bios/NEC/PC-98/sasi.rom",
"cores": [
"NP2kai"
]
},
{
"dest": "keropi/iplrom.dat",
"sha1": "0ed038ed2133b9f78c6e37256807424e0d927560",
"size": 131072,
"repo_path": "bios/Sharp/X68000/iplrom.dat",
"cores": [
"px68k"
]
},
{
"dest": "keropi/iplrom30.dat",
"sha1": "239e9124568c862c31d9ec0605e32373ea74b86a",
"size": 131072,
"repo_path": "bios/Sharp/X68000/iplrom30.dat",
"cores": [
"px68k"
]
},
{
"dest": "keropi/iplromco.dat",
"sha1": "77511fc58798404701f66b6bbc9cbde06596eba7",
"size": 131072,
"repo_path": "bios/Sharp/X68000/iplromco.dat",
"cores": [
"px68k"
]
},
{
"dest": "keropi/iplromxv.dat",
"sha1": "e33cdcdb69cd257b0b211ef46e7a8b144637db57",
"size": 131072,
"repo_path": "bios/Sharp/X68000/iplromxv.dat",
"cores": [
"px68k"
]
},
{
"dest": "keropi/cgrom.dat",
"sha1": "8d72c5b4d63bb14c5dbdac495244d659aa1498b6",
"size": 786432,
"repo_path": "bios/Sharp/X68000/cgrom.dat",
"cores": [
"px68k"
]
},
{
"dest": "scummvm/extra/MT32_CONTROL.ROM",
"sha1": "b083518fffb7f66b03c23b7eb4f868e62dc5a987",
"size": 65536,
"repo_path": "bios/Commodore/Amiga/mt32-roms/mt32_control.rom",
"cores": [
"ScummVM"
]
},
{
"dest": "scummvm/extra/MT32_PCM.ROM",
"sha1": "f6b1eebc4b2d200ec6d3d21d51325d5b48c60252",
"size": 524288,
"repo_path": "bios/Commodore/Amiga/mt32-roms/pcm_mt32.rom",
"cores": [
"ScummVM"
]
},
{
"dest": "scummvm/extra/CM32L_CONTROL.ROM",
"sha1": "a439fbb390da38cada95a7cbb1d6ca199cd66ef8",
"size": 65536,
"repo_path": "bios/Commodore/Amiga/mt32-roms/cm32l_control.rom",
"cores": [
"ScummVM"
]
},
{
"dest": "scummvm/extra/CM32L_PCM.ROM",
"sha1": "289cc298ad532b702461bfc738009d9ebe8025ea",
"size": 1048576,
"repo_path": "bios/Commodore/Amiga/mt32-roms/pcm_cm32l.rom",
"cores": [
"ScummVM"
]
},
{
"dest": "xmil/FNT0808.X1",
"sha1": "1c1a0d8c9f4c446ccd7470516b215ddca5052fb2",
"size": 2048,
"repo_path": "bios/Sharp/X1/FNT0808.X1",
"cores": [
"X Millennium"
]
}
]
}

View File

@@ -37,7 +37,11 @@ markdown_extensions:
- toc:
permalink: true
- pymdownx.details
- pymdownx.superfences
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_code_format
- pymdownx.tabbed:
alternate_style: true
plugins:
@@ -132,7 +136,7 @@ nav:
- ZC: systems/zc.md
- Emulators:
- Overview: emulators/index.md
- Official ports (61):
- Official ports (63):
- amiarcadia: emulators/amiarcadia.md
- Amiberry: emulators/amiberry.md
- Ardens: emulators/ardens.md
@@ -176,9 +180,11 @@ nav:
- mGBA: emulators/mgba.md
- Mr.Boom: emulators/mrboom.md
- Panda3DS: emulators/panda3ds.md
- PCSX2: emulators/pcsx2.md
- PicoDrive: emulators/picodrive.md
- play: emulators/play.md
- PPSSPP: emulators/ppsspp.md
- RPCS3: emulators/rpcs3.md
- Rustation: emulators/rustation.md
- RVVM: emulators/rvvm.md
- SameBoy: emulators/sameboy.md
@@ -428,7 +434,7 @@ nav:
- PCSX-ReARMed: emulators/pcsx_rearmed.md
- Launchers (1):
- Dolphin Launcher: emulators/dolphin_launcher.md
- Other (24):
- Other (23):
- ares: emulators/ares.md
- Beetle GBA (Mednafen): emulators/beetle_gba.md
- BigPEmu: emulators/bigpemu.md
@@ -440,12 +446,11 @@ nav:
- Lexaloffle: emulators/lexaloffle.md
- Model 2 Emulator: emulators/model2.md
- openMSX: emulators/openmsx.md
- PCSX2: emulators/pcsx2.md
- Redream: emulators/redream.md
- RPCS3: emulators/rpcs3.md
- Ryujinx: emulators/ryujinx.md
- shadps4: emulators/shadps4.md
- Supermodel: emulators/supermodel.md
- ti99sim: emulators/ti99sim.md
- tsugaru: emulators/tsugaru.md
- VBA-M: emulators/vba_m.md
- VICE: emulators/vice.md
@@ -457,8 +462,17 @@ nav:
- Gap Analysis: gaps.md
- Wiki:
- Overview: wiki/index.md
- Getting started: wiki/getting-started.md
- FAQ: wiki/faq.md
- Troubleshooting: wiki/troubleshooting.md
- Architecture: wiki/architecture.md
- Tools: wiki/tools.md
- Profiling guide: wiki/profiling.md
- Advanced usage: wiki/advanced-usage.md
- Verification modes: wiki/verification-modes.md
- Data model: wiki/data-model.md
- Profiling guide: wiki/profiling.md
- Adding a platform: wiki/adding-a-platform.md
- Adding a scraper: wiki/adding-a-scraper.md
- Testing guide: wiki/testing-guide.md
- Release process: wiki/release-process.md
- Contributing: contributing.md

View File

@@ -1,231 +1,870 @@
# Platform Registry
# Central configuration for all supported platforms and their scraper sources.
# Adding a new platform = adding an entry here + creating its YAML config.
#
# status: active | archived
# active -- included in automated releases, scraped weekly/monthly
# archived -- config preserved, user can generate pack manually, excluded from CI releases
platforms:
retroarch:
config: retroarch.yml
status: active
logo: "https://raw.githubusercontent.com/libretro/RetroArch/master/media/retroarch-vector_invader-only.svg"
logo: https://raw.githubusercontent.com/libretro/RetroArch/master/media/retroarch-vector_invader-only.svg
scraper: libretro
source_url: "https://raw.githubusercontent.com/libretro/libretro-database/master/dat/System.dat"
source_url: https://raw.githubusercontent.com/libretro/libretro-database/master/dat/System.dat
source_format: clrmamepro_dat
hash_type: sha1
verification_mode: existence
base_destination: system
case_insensitive_fs: true
schedule: weekly
cores: all_libretro
target_scraper: retroarch_targets
target_source: "https://buildbot.libretro.com/nightly/"
target_source: https://buildbot.libretro.com/nightly/
install:
detect:
- os: linux
method: config_file
config: "$HOME/.var/app/org.libretro.RetroArch/config/retroarch/retroarch.cfg"
parse_key: system_directory
- os: linux
method: config_file
config: "$HOME/.config/retroarch/retroarch.cfg"
parse_key: system_directory
- os: darwin
method: config_file
config: "$HOME/Library/Application Support/RetroArch/retroarch.cfg"
parse_key: system_directory
- os: windows
method: config_file
config: "%APPDATA%\\RetroArch\\retroarch.cfg"
parse_key: system_directory
- os: linux
method: config_file
config: $HOME/.var/app/org.libretro.RetroArch/config/retroarch/retroarch.cfg
parse_key: system_directory
- os: linux
method: config_file
config: $HOME/.config/retroarch/retroarch.cfg
parse_key: system_directory
- os: darwin
method: config_file
config: $HOME/Library/Application Support/RetroArch/retroarch.cfg
parse_key: system_directory
- os: windows
method: config_file
config: '%APPDATA%\RetroArch\retroarch.cfg'
parse_key: system_directory
batocera:
config: batocera.yml
status: active
logo: "https://raw.githubusercontent.com/batocera-linux/batocera-emulationstation/master/resources/splash_batocera.svg"
logo: https://raw.githubusercontent.com/batocera-linux/batocera-emulationstation/master/resources/splash_batocera.svg
scraper: batocera
source_url: "https://raw.githubusercontent.com/batocera-linux/batocera.linux/master/package/batocera/core/batocera-scripts/scripts/batocera-systems"
source_url: https://raw.githubusercontent.com/batocera-linux/batocera.linux/master/package/batocera/core/batocera-scripts/scripts/batocera-systems
source_format: python_dict
hash_type: md5
schedule: weekly
cores: [81, a5200, abuse, arduous, atari800, azahar, bennugd, bk, bluemsx, bsnes, bstone, cannonball, cap32, catacombgl, cdogs, cemu, cgenius, citron, clk, corsixth, demul, devilutionx, dhewm3, dice, dolphin, dosbox_pure, dxx-rebirth, easyrpg, ecwolf, eduke32, eka2l1, emuscv, etlegacy, fake08, fallout1-ce, fallout2-ce, fbneo, fceumm, flatpak, flycast, freechaf, freeintv, fury, fuse, gambatte, gearsystem, genesisplusgx, glide64mk2, gong, gsplus, gw, gzdoom, hatari, hcl, hurrican, hypseus-singe, ikemen, ioquake3, iortcw, jazz2-native, lindbergh-loader, lowresnx, lutro, mame, mame078plus, mednafen_lynx, mednafen_ngp, mednafen_supergrafx, mednafen_wswan, melonds, mgba, minivmac, model2emu, moonlight, mrboom, neocd, np2kai, nxengine, o2em, odcommander, openbor6412, openjazz, openjk, openjkdf2, openmohaa, opera, pce_fast, pcfx, pcsx2, pcsx_rearmed, pd777, picodrive, play, pokemini, potator, ppsspp, prboom, prosystem, puae, px68k, pygame, pyxel, quasi88, raze, reminiscence, rpcs3, ruffle, samcoupe, sameduck, scummvm, sdlpop, sh, shadps4, snes9x, solarus, sonic2013, sonic3-air, sonic-mania, steam, stella, superbroswar, supermodel, taradino, tgbdual, theforceengine, theodore, thextech, tic80, tr1x, tr2x, tsugaru, tyrian, tyrquake, uqm, uzem, vb, vecx, vice_x64, vircon32, virtualjaguar, vita3k, vox_official, vpinball, wasm4, wine-tkg, x1, x128, x16emu, xash3d_fwgs, xemu, xenia-canary, xpet, xplus4, xrick, xvic, yabasanshiro, yquake2, zc210]
cores:
- '81'
- a5200
- abuse
- amiberry
- applewin
- arduous
- atari800
- azahar
- beetle-saturn
- bennugd
- bigpemu
- bk
- blastem
- bluemsx
- boom3
- bsnes
- bstone
- cannonball
- cap32
- catacombgl
- cdogs
- cemu
- cgenius
- citron
- clk
- corsixth
- demul
- desmume
- devilutionx
- dhewm3
- dice
- dolphin
- dosbox
- dosbox_pure
- duckstation
- dxx-rebirth
- easyrpg
- ecwolf
- eduke32
- eka2l1
- emuscv
- ep128emu-core
- etlegacy
- fake08
- fallout1-ce
- fallout2-ce
- fbneo
- fceumm
- flatpak
- flycast
- fmsx
- freechaf
- freeintv
- freej2me
- fsuae
- fury
- fuse
- gambatte
- gearcoleco
- gearsystem
- genesisplusgx
- glide64mk2
- gong
- gpsp
- gsplus
- gw
- gzdoom
- handy
- hatari
- hcl
- holani
- hurrican
- hypseus-singe
- ikemen
- ioquake3
- iortcw
- jazz2-native
- kronos
- lindbergh-loader
- lowresnx
- lutro
- mame
- mame078plus
- mamemess
- mednafen_lynx
- mednafen_ngp
- mednafen_psx
- mednafen_supergrafx
- mednafen_wswan
- melonds
- mesen
- mesen-s
- mgba
- minivmac
- model2emu
- moonlight
- mrboom
- mupen64plus-next
- neocd
- nestopia
- np2kai
- nxengine
- o2em
- odcommander
- openbor6412
- openjazz
- openjk
- openjkdf2
- openmohaa
- openmsx
- opera
- parallel_n64
- pce_fast
- pcfx
- pcsx2
- pcsx_rearmed
- pd777
- picodrive
- play
- pokemini
- potator
- ppsspp
- prboom
- prosystem
- puae
- puae2021
- px68k
- pygame
- pyxel
- quasi88
- raze
- redream
- reminiscence
- rpcs3
- ruffle
- ryujinx
- samcoupe
- same_cdi
- sameduck
- scummvm
- sdlpop
- sh
- shadps4
- smsplus
- snes9x
- solarus
- sonic-mania
- sonic2013
- sonic3-air
- squirreljme
- steam
- stella
- stella2014
- superbroswar
- supermodel
- swanstation
- taradino
- tgbdual
- theforceengine
- theodore
- thextech
- tic80
- tr1x
- tr2x
- tsugaru
- tyrian
- tyrquake
- uae4arm
- uqm
- uzem
- vb
- vba-m
- vecx
- vemulator
- vice
- vice_x128
- vice_x64
- vice_x64sc
- vice_xpet
- vice_xplus4
- vice_xscpu64
- vice_xvic
- vircon32
- virtualjaguar
- vita3k
- vitaquake2
- vox_official
- vpinball
- wasm4
- wine-tkg
- x1
- x128
- x16emu
- xash3d_fwgs
- xemu
- xenia
- xenia-canary
- xpet
- xplus4
- xrick
- xroar
- xvic
- yabasanshiro
- ymir
- yquake2
- zc210
target_scraper: batocera_targets
target_source: "https://github.com/batocera-linux/batocera.linux"
target_source: https://github.com/batocera-linux/batocera.linux
install:
detect:
- os: linux
method: file_exists
file: /etc/batocera-version
bios_path: /userdata/bios
- os: linux
method: file_exists
file: /etc/batocera-version
bios_path: /userdata/bios
recalbox:
config: recalbox.yml
status: active
logo: "https://raw.githubusercontent.com/homarr-labs/dashboard-icons/main/svg/recalbox.svg"
logo: https://raw.githubusercontent.com/homarr-labs/dashboard-icons/main/svg/recalbox.svg
scraper: recalbox
source_url: "https://gitlab.com/recalbox/recalbox/-/raw/master/board/recalbox/fsoverlay/recalbox/share_init/system/.emulationstation/es_bios.xml"
source_url: https://gitlab.com/recalbox/recalbox/-/raw/master/board/recalbox/fsoverlay/recalbox/share_init/system/.emulationstation/es_bios.xml
source_format: xml
hash_type: md5
schedule: monthly
target_scraper: null
target_source: null
cores: ["2048", 81, a5200, advancemame, amiberry, applewin, arduous, atari800, b2, beebem, bk, bluemsx, boom3, bsnes, bsneshd, cannonball, cap32, cdi2015, corsixth, craft, crocods, daphne, desmume, dice, dinothawr, dirksimple, dolphin, dolphin-gui, dosbox, dosbox_pure, duckstation, easyrpg, ecwolf, emuscv, fake08, fba2x, fbneo, fceumm, flycast, flycast-next, fmsx, freechaf, freeintv, frotz, fuse, gambatte, gearcoleco, geargrafx, gearsystem, genesisplusgx, genesisplusgx_ex, genesisplusgxwide, geolith, glide64mk2, gliden64, gliden64_20, gong, gpsp, gsplus, gw, handy, hatari, hatarib, holani, imageviewer, julius, kronos, lowresnx, lutro, mame0258, mame0278, mame2000, mame2003, mame2003_plus, mame2010, mame2015, mame2016, mednafen_lynx, mednafen_ngp, mednafen_pce_fast, mednafen_pcfx, mednafen_psx, mednafen_psx_hw, mednafen_saturn, mednafen_supafaust, mednafen_supergrafx, mednafen_vb, mednafen_wswan, melonds, mesen, mesen_s, meteor, mgba, minivmac, mojozork, moonlight, mrboom, mu, mupen64plus, mupen64plus_next, n64_gles2, neocd, nestopia, np2kai, nxengine, o2em, openbor, openlara, opera, oricutron, parallel_n64, pcsx2, pcsx_rearmed, pico8, picodrive, pisnes, pokemini, potator, ppsspp, prboom, prosystem, ps2, puae, px68k, quasi88, quicknes, race, rb5000, reicast, reminiscence, retro8, retrodream, rice, rice_gles2, sameboy, same_cdi, sameduck, scummvm, sdlpop, simcoupe, snes9x, snes9x2002, snes9x2005, snes9x2010, solarus, stella, stella2014, stonesoup, supermodel, swanstation, tamalibretro, tgbdual, theodore, thepowdertoy, ti99sim, tic80, tyrquake, uae4all, uae4arm, uzem, vecx, vice_x128, vice_x64, vice_x64sc, vice_xcbm2, vice_xcbm5x0, vice_xpet, vice_xplus4, vice_xscpu64, vice_xvic, virtualjaguar, vitaquake2, vitaquake3, vitavoyager, vpinball, vvvvvv, wasm4, x1, x128, x64, x64sx, xcbm2, xcbm5x0, xemu, xpet, xplus4, xrick, xroar, xscpu64, xvic, yabasanshiro, yabause]
cores:
- '2048'
- '81'
- a5200
- advancemame
- amiberry
- applewin
- arduous
- atari800
- b2
- beebem
- bk
- bluemsx
- boom3
- bsnes
- bsneshd
- cannonball
- cap32
- cdi2015
- corsixth
- craft
- crocods
- daphne
- desmume
- dice
- dinothawr
- dirksimple
- dolphin
- dolphin-gui
- dosbox
- dosbox_pure
- duckstation
- easyrpg
- ecwolf
- emuscv
- fake08
- fba2x
- fbneo
- fceumm
- flycast
- flycast-next
- fmsx
- freechaf
- freeintv
- frotz
- fuse
- gambatte
- gearcoleco
- geargrafx
- gearsystem
- genesisplusgx
- genesisplusgx_ex
- genesisplusgxwide
- geolith
- glide64mk2
- gliden64
- gliden64_20
- gong
- gpsp
- gsplus
- gw
- handy
- hatari
- hatarib
- holani
- imageviewer
- julius
- kronos
- lowresnx
- lutro
- mame
- mame0258
- mame0278
- mame2000
- mame2003
- mame2003_plus
- mame2010
- mame2015
- mame2016
- mamemess
- mednafen_lynx
- mednafen_ngp
- mednafen_pce_fast
- mednafen_pcfx
- mednafen_psx
- mednafen_psx_hw
- mednafen_saturn
- mednafen_supafaust
- mednafen_supergrafx
- mednafen_vb
- mednafen_wswan
- melonds
- mesen
- mesen_s
- meteor
- mgba
- minivmac
- mojozork
- moonlight
- mrboom
- mu
- mupen64plus
- mupen64plus_next
- n64_gles2
- neocd
- nestopia
- np2kai
- nxengine
- o2em
- openbor
- openlara
- opera
- oricutron
- parallel_n64
- pcsx2
- pcsx_rearmed
- pico8
- picodrive
- pisnes
- pokemini
- potator
- ppsspp
- prboom
- prosystem
- ps2
- puae
- px68k
- quasi88
- quicknes
- race
- rb5000
- reicast
- reminiscence
- retro8
- retrodream
- rice
- rice_gles2
- same_cdi
- sameboy
- sameduck
- scummvm
- sdlpop
- simcoupe
- snes9x
- snes9x2002
- snes9x2005
- snes9x2010
- solarus
- stella
- stella2014
- stonesoup
- supermodel
- swanstation
- tamalibretro
- tgbdual
- theodore
- thepowdertoy
- ti99sim
- tic80
- tyrquake
- uae4all
- uae4arm
- uzem
- vecx
- vice_x128
- vice_x64
- vice_x64sc
- vice_xcbm2
- vice_xcbm5x0
- vice_xpet
- vice_xplus4
- vice_xscpu64
- vice_xvic
- virtualjaguar
- vitaquake2
- vitaquake3
- vitavoyager
- vpinball
- vvvvvv
- wasm4
- x1
- x128
- x64
- x64sx
- xcbm2
- xcbm5x0
- xemu
- xpet
- xplus4
- xrick
- xroar
- xscpu64
- xvic
- yabasanshiro
- yabause
install:
detect:
- os: linux
method: file_exists
file: /usr/bin/recalbox-settings
bios_path: /recalbox/share/bios
- os: linux
method: file_exists
file: /usr/bin/recalbox-settings
bios_path: /recalbox/share/bios
retrobat:
config: retrobat.yml
status: active
logo: "https://raw.githubusercontent.com/RetroBat-Official/retrobat/main/system/resources/retrobat_logo_notext.png"
logo: https://raw.githubusercontent.com/RetroBat-Official/retrobat/main/system/resources/retrobat_logo_notext.png
scraper: retrobat
source_url: "https://raw.githubusercontent.com/RetroBat-Official/emulatorlauncher/master/batocera-systems/Resources/batocera-systems.json"
source_url: https://raw.githubusercontent.com/RetroBat-Official/emulatorlauncher/master/batocera-systems/Resources/batocera-systems.json
source_format: json
hash_type: md5
schedule: weekly
cores: [81, a5200, abuse, arduous, atari800, azahar, bennugd, bk, bluemsx, bsnes, bstone, cannonball, cap32, catacombgl, cdogs, cemu, cgenius, citron, clk, corsixth, demul, devilutionx, dhewm3, dice, dolphin, dosbox_pure, dxx-rebirth, easyrpg, ecwolf, eduke32, eka2l1, emuscv, etlegacy, fake08, fallout1-ce, fallout2-ce, fbneo, fceumm, flatpak, flycast, freechaf, freeintv, fury, fuse, gambatte, gearsystem, genesisplusgx, glide64mk2, gong, gsplus, gw, gzdoom, hatari, hcl, hurrican, hypseus-singe, ikemen, ioquake3, iortcw, jazz2-native, lindbergh-loader, lowresnx, lutro, mame, mame078plus, mednafen_lynx, mednafen_ngp, mednafen_supergrafx, mednafen_wswan, melonds, mgba, minivmac, model2emu, moonlight, mrboom, neocd, np2kai, nxengine, o2em, odcommander, openbor6412, openjazz, openjk, openjkdf2, openmohaa, opera, pce_fast, pcfx, pcsx2, pcsx_rearmed, pd777, picodrive, play, pokemini, potator, ppsspp, prboom, prosystem, puae, px68k, pygame, pyxel, quasi88, raze, reminiscence, rpcs3, ruffle, samcoupe, sameduck, scummvm, sdlpop, sh, shadps4, snes9x, solarus, sonic2013, sonic3-air, sonic-mania, steam, stella, superbroswar, supermodel, taradino, tgbdual, theforceengine, theodore, thextech, tic80, tr1x, tr2x, tsugaru, tyrian, tyrquake, uqm, uzem, vb, vecx, vice_x64, vircon32, virtualjaguar, vita3k, vox_official, vpinball, wasm4, wine-tkg, x1, x128, x16emu, xash3d_fwgs, xemu, xenia-canary, xpet, xplus4, xrick, xvic, yabasanshiro, yquake2, zc210]
cores:
- '81'
- a5200
- abuse
- arduous
- ares
- atari800
- azahar
- bennugd
- bk
- bluemsx
- bsnes
- bstone
- cannonball
- cap32
- catacombgl
- cdogs
- cemu
- cgenius
- citron
- clk
- corsixth
- demul
- devilutionx
- dhewm3
- dice
- dolphin
- dosbox_pure
- dxx-rebirth
- easyrpg
- ecwolf
- eduke32
- eka2l1
- emuscv
- etlegacy
- fake08
- fallout1-ce
- fallout2-ce
- fbalpha2012
- fbalpha2012_neogeo
- fbneo
- fceumm
- flatpak
- flycast
- freechaf
- freeintv
- freej2me
- fury
- fuse
- gambatte
- geargrafx
- gearsystem
- genesisplusgx
- glide64mk2
- gong
- gsplus
- gw
- gzdoom
- hatari
- hcl
- hurrican
- hypseus-singe
- ikemen
- ioquake3
- iortcw
- jazz2-native
- lindbergh-loader
- lowresnx
- lutro
- mame
- mame078plus
- mamemess
- mednafen_lynx
- mednafen_ngp
- mednafen_supergrafx
- mednafen_wswan
- melonds
- mgba
- minivmac
- model2emu
- moonlight
- mrboom
- mupen64plus_next
- neocd
- np2kai
- nxengine
- o2em
- odcommander
- openbor6412
- openjazz
- openjk
- openjkdf2
- openmohaa
- opera
- parallel_n64
- pce_fast
- pcfx
- pcsx2
- pcsx_rearmed
- pd777
- picodrive
- play
- pokemini
- potator
- ppsspp
- prboom
- prosystem
- puae
- px68k
- pygame
- pyxel
- quasi88
- raze
- reminiscence
- rpcs3
- ruffle
- samcoupe
- sameduck
- scummvm
- sdlpop
- sh
- shadps4
- snes9x
- solarus
- sonic-mania
- sonic2013
- sonic3-air
- squirreljme
- steam
- stella
- superbroswar
- supermodel
- taradino
- tgbdual
- theforceengine
- theodore
- thextech
- tic80
- tr1x
- tr2x
- tsugaru
- tyrian
- tyrquake
- uqm
- uzem
- vb
- vecx
- vice_x64
- vircon32
- virtualjaguar
- vita3k
- vox_official
- vpinball
- wasm4
- wine-tkg
- x1
- x128
- x16emu
- xash3d_fwgs
- xemu
- xenia-canary
- xpet
- xplus4
- xrick
- xroar
- xvic
- yabasanshiro
- yquake2
- zc210
target_scraper: null
target_source: null
install:
detect:
- os: windows
method: path_exists
path: "%USERPROFILE%\\RetroBat\\bios"
- os: windows
method: path_exists
path: '%USERPROFILE%\RetroBat\bios'
emudeck:
config: emudeck.yml
status: active
logo: "https://raw.githubusercontent.com/dragoonDorise/EmuDeck/main/icons/EmuDeck.png"
logo: https://raw.githubusercontent.com/dragoonDorise/EmuDeck/main/icons/EmuDeck.png
scraper: emudeck
source_url: "https://raw.githubusercontent.com/dragoonDorise/EmuDeck/main/functions/checkBIOS.sh"
source_wiki: "https://raw.githubusercontent.com/EmuDeck/emudeck.github.io/main/docs/tables/"
source_url: https://raw.githubusercontent.com/dragoonDorise/EmuDeck/main/functions/checkBIOS.sh
source_wiki: https://raw.githubusercontent.com/EmuDeck/emudeck.github.io/main/docs/tables/
source_format: bash_script+csv
hash_type: md5
schedule: weekly
target_scraper: emudeck_targets
target_source: "https://github.com/dragoonDorise/EmuDeck"
# dragoonDorise/EmuDeck = official repo (creator's account, 3.4k stars)
# EmuDeck/emudeck.github.io = official wiki (org account)
target_source: https://github.com/dragoonDorise/EmuDeck
install:
detect:
- os: linux
method: config_file
config: "$HOME/.config/EmuDeck/settings.sh"
parse_key: emulationPath
bios_subdir: bios
- os: linux
method: path_exists
path: "$HOME/Emulation/bios"
- os: windows
method: config_file
config: "%APPDATA%\\EmuDeck\\settings.ps1"
parse_key: "$emulationPath"
bios_subdir: bios
- os: linux
method: config_file
config: $HOME/.config/EmuDeck/settings.sh
parse_key: emulationPath
bios_subdir: bios
- os: linux
method: path_exists
path: $HOME/Emulation/bios
- os: windows
method: config_file
config: '%APPDATA%\EmuDeck\settings.ps1'
parse_key: $emulationPath
bios_subdir: bios
standalone_copies:
- file: prod.keys
targets:
linux:
- "$HOME/.local/share/yuzu/keys"
- "$HOME/.local/share/eden/keys"
- "$HOME/.config/Ryujinx/system"
windows:
- "%APPDATA%\\yuzu\\keys"
- "%APPDATA%\\eden\\keys"
- file: aes_keys.txt
targets:
linux:
- "$HOME/Emulation/bios/citra/keys"
- file: prod.keys
targets:
linux:
- $HOME/.local/share/yuzu/keys
- $HOME/.local/share/eden/keys
- $HOME/.config/Ryujinx/system
windows:
- '%APPDATA%\yuzu\keys'
- '%APPDATA%\eden\keys'
- file: aes_keys.txt
targets:
linux:
- $HOME/Emulation/bios/citra/keys
lakka:
config: lakka.yml
status: active
logo: "https://raw.githubusercontent.com/libretro/retroarch-assets/master/src/xmb/flatui/lakka.svg"
logo: https://raw.githubusercontent.com/libretro/retroarch-assets/master/src/xmb/flatui/lakka.svg
scraper: libretro
inherits_from: retroarch
cores: all_libretro
schedule: weekly
target_scraper: lakka_targets
target_source: "https://buildbot.libretro.com/nightly/"
target_source: https://buildbot.libretro.com/nightly/
install:
detect:
- os: linux
method: os_release
id: lakka
bios_path: /storage/system
- os: linux
method: os_release
id: lakka
bios_path: /storage/system
retrodeck:
config: retrodeck.yml
status: active
logo: "https://raw.githubusercontent.com/RetroDECK/RetroDECK/main/res/icon.svg"
logo: https://raw.githubusercontent.com/RetroDECK/RetroDECK/main/res/icon.svg
scraper: retrodeck
source_url: "https://github.com/RetroDECK/components"
source_url: https://github.com/RetroDECK/components
source_format: github_component_manifests
hash_type: md5
schedule: monthly
cores: [azahar, cemu, dolphin, duckstation, gzdoom, mame, melonds, openbor, pcsx2, pico-8, ppsspp, primehack, retroarch, rpcs3, ruffle, solarus, vita3k, xemu, xroar]
cores:
- azahar
- cemu
- clk
- dolphin
- duckstation
- gsplus
- gzdoom
- mame
- melonds
- openbor
- pcsx2
- pico-8
- ppsspp
- primehack
- retroarch
- rpcs3
- ruffle
- solarus
- vita3k
- xemu
- xroar
target_scraper: null
target_source: null
# Each component/<name>/component_manifest.json declares BIOS requirements
# Scraper enumerates top-level dirs via GitHub API, fetches each manifest directly
install:
detect:
- os: linux
method: path_exists
path: "$HOME/.var/app/net.retrodeck.retrodeck"
bios_path: "$HOME/retrodeck/bios"
- os: linux
method: path_exists
path: $HOME/.var/app/net.retrodeck.retrodeck
bios_path: $HOME/retrodeck/bios
romm:
config: romm.yml
status: active
logo: "https://avatars.githubusercontent.com/u/168586850"
logo: https://avatars.githubusercontent.com/u/168586850
scraper: romm
source_url: "https://raw.githubusercontent.com/rommapp/romm/master/backend/models/fixtures/known_bios_files.json"
source_url: https://raw.githubusercontent.com/rommapp/romm/master/backend/models/fixtures/known_bios_files.json
source_format: json
hash_type: sha1
schedule: monthly
inherits_from: emulatorjs # cores inherited from emulatorjs.yml
inherits_from: emulatorjs
target_scraper: null
target_source: null
install:
detect:
- os: linux
method: path_exists
path: /romm/library/bios
- os: linux
method: path_exists
path: /romm/library/bios
cores:
- atari800
- clk
- directxbox
- dolphin
- dolphin_launcher
- ecwolf
- ep128emu
- ep128emu_core
- freej2me
- hatari
- ishiiruka
- lrps2
- minivmac
- nekop2
- np2kai
- o2em
- pcsx2
- play
- pokemini
- primehack
- px68k
- scummvm
- squirreljme
- x1
- xemu
retropie:
config: retropie.yml
status: archived # Last release: v4.8 (March 2022) - no update in 4 years
logo: "https://avatars.githubusercontent.com/u/11378204"
status: archived
logo: https://avatars.githubusercontent.com/u/11378204
scraper: null
cores: all_libretro
schedule: null
target_scraper: retropie_targets
target_source: "https://retropie.org.uk/stats/pkgflags/"
target_source: https://retropie.org.uk/stats/pkgflags/
install:
detect:
- os: linux
method: path_exists
path: "$HOME/RetroPie/BIOS"
- os: linux
method: path_exists
path: $HOME/RetroPie/BIOS
bizhawk:
config: bizhawk.yml
status: active
logo: "https://raw.githubusercontent.com/TASEmulators/BizHawk/master/Assets/bizhawk.ico"
logo: https://raw.githubusercontent.com/TASEmulators/BizHawk/master/Assets/bizhawk.ico
scraper: bizhawk
source_url: "https://raw.githubusercontent.com/TASEmulators/BizHawk/master/src/BizHawk.Emulation.Common/Database/FirmwareDatabase.cs"
source_url: https://raw.githubusercontent.com/TASEmulators/BizHawk/master/src/BizHawk.Emulation.Common/Database/FirmwareDatabase.cs
source_format: csharp_firmware_database
hash_type: sha1
schedule: monthly
cores: [gambatte, mgba, sameboy, melonds, snes9x, bsnes, beetle_psx, beetle_saturn, beetle_pce, beetle_pcfx, beetle_wswan, beetle_vb, beetle_ngp, opera, stella, picodrive, ppsspp, handy, quicknes, genesis_plus_gx, ares, mupen64plus_next, puae, prboom, virtualjaguar, vice_x64, mame]
cores:
- applewin
- ares
- azahar
- beetle_ngp
- beetle_pce
- beetle_pcfx
- beetle_psx
- beetle_saturn
- beetle_vb
- beetle_wswan
- bsnes
- citra
- citra2018
- citra_canary
- clk
- fbneo
- freechaf
- freeintv
- gambatte
- genesis_plus_gx
- handy
- mame
- melonds
- mgba
- mupen64plus_next
- numero
- o2em
- opera
- panda3ds
- picodrive
- ppsspp
- prboom
- prosystem
- puae
- quicknes
- sameboy
- snes9x
- stella
- trident
- vecx
- vice_x64
- virtualjaguar
target_scraper: null
target_source: null
install:
detect:
- os: windows
method: path_exists
path: "%USERPROFILE%\\BizHawk\\Firmware"
- os: linux
method: path_exists
path: "$HOME/.config/BizHawk/Firmware"
- os: windows
method: path_exists
path: '%USERPROFILE%\BizHawk\Firmware'
- os: linux
method: path_exists
path: $HOME/.config/BizHawk/Firmware

File diff suppressed because it is too large Load Diff

View File

@@ -4,10 +4,8 @@ dat_version: v1.19.0
homepage: https://www.retroarch.com
source: https://github.com/libretro/libretro-database/blob/master/dat/System.dat
base_destination: system
cores: all_libretro
hash_type: sha1
verification_mode: existence
case_insensitive_fs: true
systems:
3do:
files:
@@ -102,6 +100,7 @@ systems:
md5: 35fa1a1ebaaeea286dc5cd15487c13ea
crc32: d5cbc509
size: 1048576
native_id: 3DO Company, The - 3DO
core: opera
manufacturer: Panasonic|GoldStar|Sanyo
docs: https://docs.libretro.com/library/opera/
@@ -135,6 +134,7 @@ systems:
md5: 25629dfe870d097469c217b95fdc1c95
crc32: 1fe22ecd
size: 16384
native_id: Amstrad - CPC
arcade:
files:
- name: bubsys.zip
@@ -249,6 +249,10 @@ systems:
- name: aes.zip
destination: aes.zip
required: true
native_id: Arcade
core: fbneo
manufacturer: Various
docs: https://docs.libretro.com/library/fbneo/
data_directories:
- ref: fbneo-hiscore
destination: ''
@@ -256,9 +260,6 @@ systems:
destination: fbneo
- ref: fbneo-samples
destination: fbneo
core: fbneo
manufacturer: Various
docs: https://docs.libretro.com/library/fbneo/
atari-400-800:
files:
- name: ATARIBAS.ROM
@@ -303,6 +304,7 @@ systems:
md5: d7eb37aec6960cba36bc500e0e5d00bc
crc32: bdca01fb
size: 8192
native_id: Atari - 400-800
atari-5200:
files:
- name: 5200.rom
@@ -312,6 +314,7 @@ systems:
md5: 281f20ea4320404ec820fb7ec0693b38
crc32: 4248d3e3
size: 2048
native_id: Atari - 5200
core: a5200
manufacturer: Atari
docs: https://docs.libretro.com/library/a5200/
@@ -331,6 +334,7 @@ systems:
md5: 0763f1ffb006ddbe32e52d497ee848ae
crc32: 5d13730c
size: 4096
native_id: Atari - 7800
core: prosystem
manufacturer: Atari
docs: https://docs.libretro.com/library/prosystem/
@@ -343,6 +347,7 @@ systems:
md5: fcd403db69f54290b51035d82f835e7b
crc32: 0d973c9d
size: 512
native_id: Atari - Lynx
core: handy
manufacturer: Atari
docs: https://docs.libretro.com/library/handy/
@@ -355,6 +360,7 @@ systems:
md5: c1c57ce48e8ee4135885cee9e63a68a2
crc32: d3c32283
size: 196608
native_id: Atari - ST
core: hatari
manufacturer: Atari
docs: https://docs.libretro.com/library/hatari/
@@ -376,6 +382,7 @@ systems:
- name: bioscv.rom
destination: bioscv.rom
required: true
native_id: Coleco - ColecoVision
commodore-amiga:
files:
- name: kick33180.A500
@@ -462,6 +469,7 @@ systems:
md5: bb72565701b1b6faece07d68ea5da639
crc32: 87746be2
size: 524288
native_id: Commodore - Amiga
core: puae
manufacturer: Commodore
docs: https://docs.libretro.com/library/puae/
@@ -510,6 +518,7 @@ systems:
destination: scpu-dos-2.04.bin
required: true
md5: b2869f8678b8b274227f35aad26ba509
native_id: Commodore - C128
core: vice_x128
manufacturer: Commodore
docs: https://docs.libretro.com/library/vice_x128/
@@ -522,6 +531,7 @@ systems:
md5: a2e891e330d146c4046c2b622fc31462
crc32: 683ed4ad
size: 5763199
native_id: Dinothawr
dos:
files:
- name: MT32_CONTROL.ROM
@@ -552,6 +562,7 @@ systems:
md5: 08cdcfa0ed93e9cb16afa76e6ac5f0a4
crc32: 4b961eba
size: 1048576
native_id: DOS
elektronika-bk:
files:
- name: B11M_BOS.ROM
@@ -610,6 +621,7 @@ systems:
md5: 95f8c41c6abf7640e35a6a03cecebd01
crc32: 26c6e8a0
size: 8192
native_id: Elektronika - BK-0010/BK-0011(M)
enterprise-64-128:
files:
- name: hun.rom
@@ -696,6 +708,7 @@ systems:
md5: 55af78f877a21ca45eb2df68a74fcc60
crc32: c099a5e3
size: 65536
native_id: Enterprise - 64/128
includes:
- ep128emu
epoch-scv:
@@ -707,6 +720,7 @@ systems:
md5: 635a978fd40db9a18ee44eff449fc126
crc32: 7ac06182
size: 4096
native_id: EPOCH/YENO Super Cassette Vision
fairchild-channel-f:
files:
- name: sl31253.bin
@@ -730,6 +744,7 @@ systems:
md5: 95d339631d867c8f1d15a5f2ec26069d
crc32: 015c1e38
size: 1024
native_id: Fairchild Channel F
doom:
files:
- name: prboom.wad
@@ -739,6 +754,7 @@ systems:
md5: 72ae1b47820fcc93cc0df9c428d0face
crc32: a5751b99
size: 143312
native_id: Id Software - Doom
j2me:
files:
- name: freej2me-lr.jar
@@ -762,6 +778,7 @@ systems:
md5: 29a92d0867da2917275b7c6c805d256f
crc32: ffb98ffa
size: 552039
native_id: J2ME
core: freej2me
manufacturer: Java
docs: https://docs.libretro.com/library/freej2me/
@@ -774,6 +791,7 @@ systems:
md5: 66223be1497460f1e60885eeb35e03cc
crc32: 4df6d054
size: 262144
native_id: MacII
magnavox-odyssey2:
files:
- name: o2rom.bin
@@ -804,6 +822,7 @@ systems:
md5: 279008e4a0db2dc5f1c048853b033828
crc32: 11647ca5
size: 1024
native_id: Magnavox - Odyssey2
core: o2em
manufacturer: Magnavox|Philips
docs: https://docs.libretro.com/library/o2em/
@@ -823,6 +842,7 @@ systems:
md5: 0cd5946c6473e42e8e4c2137785e427f
crc32: 683a4158
size: 2048
native_id: Mattel - Intellivision
data_directories:
- ref: freeintv-overlays
destination: freeintv_overlays
@@ -933,6 +953,7 @@ systems:
md5: 279efd1eae0d358eecd4edc7d9adedf3
crc32: ab6874f8
size: 16640
native_id: Microsoft - MSX
core: bluemsx
manufacturer: Spectravideo|Philips|Al Alamiah|Sony|Sanyo|Mitsubishi|Toshiba|Hitachi|Panasonic|Canon|Casio|Pioneer|Fujitsu|Yamaha|JVC|Kyocera|GoldStar|Samsung|Daewoo|Gradiente|Sharp|Talent|NTT|ACVS/CIEL|DDX|AGE
Labs
@@ -991,6 +1012,7 @@ systems:
md5: 0754f903b52e3b3342202bdafb13efa5
crc32: 2b5b75fe
size: 262144
native_id: NEC - PC Engine - TurboGrafx 16 - SuperGrafx
core: mednafen_pce_fast
manufacturer: NEC
docs: https://docs.libretro.com/library/mednafen_pce_fast/
@@ -1073,6 +1095,7 @@ systems:
md5: 524473c1a5a03b17e21d86a0408ff827
crc32: fe9f57f2
size: 16384
native_id: NEC - PC-98
core: np2kai
manufacturer: NEC
docs: https://docs.libretro.com/library/np2kai/
@@ -1115,6 +1138,7 @@ systems:
md5: e2fb7c7220e3a7838c2dd7e401a7f3d8
crc32: 236102c9
size: 1048576
native_id: NEC - PC-FX
core: mednafen_pcfx
manufacturer: NEC
docs: https://docs.libretro.com/library/mednafen_pcfx/
@@ -1134,6 +1158,7 @@ systems:
md5: 7f98d77d7a094ad7d069b74bd553ec98
crc32: 4c514089
size: 24592
native_id: Nintendo - Famicom Disk System
nintendo-gb:
files:
- name: dmg_boot.bin
@@ -1150,6 +1175,7 @@ systems:
md5: 32fbbd84168d3482956eb3c5051637f5
crc32: 59c8598e
size: 256
native_id: Nintendo - Gameboy
core: gambatte
manufacturer: Nintendo
docs: https://docs.libretro.com/library/gambatte/
@@ -1162,6 +1188,7 @@ systems:
md5: a860e8c0b6d573d191e4ec7db1b1e4f6
crc32: '81977335'
size: 16384
native_id: Nintendo - Game Boy Advance
core: gpsp
manufacturer: Nintendo
docs: https://docs.libretro.com/library/gpsp/
@@ -1181,6 +1208,7 @@ systems:
md5: dbfce9db9deaa2567f6a84fde55f9680
crc32: 41884e46
size: 2304
native_id: Nintendo - Gameboy Color
nintendo-gamecube:
files:
- name: gc-dvd-20010608.bin
@@ -1274,6 +1302,7 @@ systems:
- name: font_japanese.bin
destination: dolphin-emu/Sys/GC/font_japanese.bin
required: false
native_id: Nintendo - GameCube
core: dolphin
manufacturer: Nintendo
docs: https://docs.libretro.com/library/dolphin/
@@ -1289,6 +1318,7 @@ systems:
md5: 8d3d9f294b6e174bc7b1d2fd1c727530
crc32: 7f933ce2
size: 4194304
native_id: Nintendo - Nintendo 64DD
nintendo-ds:
files:
- name: bios7.bin
@@ -1324,6 +1354,7 @@ systems:
- name: dsi_nand.bin
destination: dsi_nand.bin
required: true
native_id: Nintendo - Nintendo DS
core: desmume
manufacturer: Nintendo
docs: https://docs.libretro.com/library/desmume/
@@ -1332,10 +1363,11 @@ systems:
- name: NstDatabase.xml
destination: NstDatabase.xml
required: true
sha1: f92312bae56e29c5bf00a5103105fce78472bf5c
md5: 0ee6cbdc6f5c96ce9c8aa5edb59066f4
crc32: 0e4d552b
sha1: 26322f182540211e9b5e3647675b7c593706ae2b
md5: 7bfe8c0540ed4bd6a0f1e2a0f0118ced
crc32: ebb2196c
size: 1009534
native_id: Nintendo - Nintendo Entertainment System
core: fceumm
manufacturer: Nintendo
docs: https://docs.libretro.com/library/fceumm/
@@ -1348,6 +1380,7 @@ systems:
md5: 1e4fb124a3a886865acb574f388c803d
crc32: aed3c14d
size: 4096
native_id: Nintendo - Pokemon Mini
nintendo-satellaview:
files:
- name: BS-X.bin
@@ -1371,6 +1404,7 @@ systems:
md5: 4ed9648505ab33a4daec93707b16caba
crc32: 8c573c7e
size: 1048576
native_id: Nintendo - Satellaview
nintendo-sufami-turbo:
files:
- name: STBIOS.bin
@@ -1380,6 +1414,7 @@ systems:
md5: d3a44ba7d42a74d3ac58cb9c14c6a5ca
crc32: 9b4ca911
size: 262144
native_id: Nintendo - SuFami Turbo
nintendo-sgb:
files:
- name: SGB1.sfc
@@ -1441,6 +1476,7 @@ systems:
- name: sgb.boot.rom
destination: sgb.boot.rom
required: false
native_id: Nintendo - Super Game Boy
nintendo-snes:
files:
- name: cx4.data.rom
@@ -1562,6 +1598,7 @@ systems:
md5: dda40ccd57390c96e49d30a041f9a9e7
crc32: f73d5e10
size: 131072
native_id: Nintendo - Super Nintendo Entertainment System
core: bsnes
manufacturer: Nintendo
docs: https://docs.libretro.com/library/bsnes/
@@ -1588,6 +1625,7 @@ systems:
md5: 279008e4a0db2dc5f1c048853b033828
crc32: 11647ca5
size: 1024
native_id: Phillips - Videopac+
sega-dreamcast:
files:
- name: dc_boot.bin
@@ -1611,6 +1649,7 @@ systems:
md5: 0a93f7940c455905bea6e392dfde92a4
crc32: c611b498
size: 131072
native_id: Sega - Dreamcast
core: flycast
manufacturer: Sega
docs: https://docs.libretro.com/library/flycast/
@@ -1668,6 +1707,7 @@ systems:
- name: segasp.zip
destination: dc/segasp.zip
required: true
native_id: Sega - Dreamcast-based Arcade
sega-game-gear:
files:
- name: bios.gg
@@ -1677,6 +1717,7 @@ systems:
md5: 672e104c3be3a238301aceffc3b23fd6
crc32: 0ebea9d4
size: 1024
native_id: Sega - Game Gear
sega-master-system:
files:
- name: bios.sms
@@ -1707,6 +1748,7 @@ systems:
md5: 840481177270d5642a14ca71ee72844c
crc32: 0072ed54
size: 8192
native_id: Sega - Master System - Mark III
sega-mega-cd:
files:
- name: bios_CD_E.bin
@@ -1730,6 +1772,7 @@ systems:
md5: 2efd74e3232ff260e371b99f84024f7f
crc32: c6d10268
size: 131072
native_id: Sega - Mega CD - Sega CD
sega-mega-drive:
files:
- name: areplay.bin
@@ -1774,6 +1817,7 @@ systems:
md5: b4e76e416b887f4e7413ba76fa735f16
crc32: 4dcfd55c
size: 262144
native_id: Sega - Mega Drive - Genesis
core: genesis_plus_gx
manufacturer: Sega
docs: https://docs.libretro.com/library/genesis_plus_gx/
@@ -1859,6 +1903,7 @@ systems:
- name: stvbios.zip
destination: kronos/stvbios.zip
required: true
native_id: Sega - Saturn
core: kronos
manufacturer: Sega
docs: https://docs.libretro.com/library/kronos/
@@ -1880,6 +1925,7 @@ systems:
md5: 851e4a5936f17d13f8c39a980cf00d77
crc32: e3995a57
size: 2048
native_id: Sharp - X1
core: x1
manufacturer: Sharp
docs: https://docs.libretro.com/library/x1/
@@ -1920,6 +1966,7 @@ systems:
md5: 0617321daa182c3f3d6f41fd02fb3275
crc32: 00eeb408
size: 131072
native_id: Sharp - X68000
core: px68k
manufacturer: Sharp
docs: https://docs.libretro.com/library/px68k/
@@ -2270,6 +2317,7 @@ systems:
md5: 85fede415f4294cc777517d7eada482e
crc32: 2cbe8995
size: 32768
native_id: Sinclair - ZX Spectrum
core: fuse
manufacturer: Sinclair|Amstrad
docs: https://docs.libretro.com/library/fuse/
@@ -2352,6 +2400,7 @@ systems:
md5: 08ca8b2dba6662e8024f9e789711c6fc
crc32: ff3abc59
size: 524288
native_id: SNK - NeoGeo CD
core: neocd
manufacturer: SNK
docs: https://docs.libretro.com/library/neocd/
@@ -2511,6 +2560,7 @@ systems:
md5: 81bbe60ba7a3d1cea1d48c14cbcc647b
crc32: 2f53b852
size: 524288
native_id: Sony - PlayStation
core: duckstation
manufacturer: Sony
docs: https://docs.libretro.com/library/duckstation/
@@ -3027,6 +3077,7 @@ systems:
md5: d3e81e95db25f5a86a7b7474550a2155
crc32: 4e8c160c
size: 4194304
native_id: Sony - PlayStation 2
sony-psp:
files:
- name: ppge_atlas.zim
@@ -3036,6 +3087,7 @@ systems:
md5: 866855cc330b9b95cc69135fb7b41d38
crc32: 7b57fa78
size: 666530
native_id: Sony - PlayStation Portable
core: ppsspp
manufacturer: Sony
docs: https://docs.libretro.com/library/ppsspp/
@@ -3065,6 +3117,7 @@ systems:
md5: d4448d09bbfde687c04f9e3310e023ab
crc32: 4bf05697
size: 262144
native_id: Texas Instruments TI-83
core: numero
manufacturer: Texas Instruments
docs: https://docs.libretro.com/library/numero/
@@ -3098,6 +3151,7 @@ systems:
md5: 88dc7876d584f90e4106f91444ab23b7
crc32: 1466aed4
size: 16384
native_id: Videoton - TV Computer
wolfenstein-3d:
files:
- name: ecwolf.pk3
@@ -3107,6 +3161,7 @@ systems:
md5: c011b428819eea4a80b455c245a5a04d
crc32: 26dc3fba
size: 178755
native_id: Wolfenstein 3D
scummvm:
files:
- name: scummvm.zip
@@ -3116,6 +3171,7 @@ systems:
md5: a17e0e0150155400d8cced329563d9c8
crc32: a93f1c4b
size: 9523360
native_id: ScummVM
core: scummvm
manufacturer: Various
docs: https://docs.libretro.com/library/scummvm/
@@ -3141,3 +3197,5 @@ systems:
core: xrick
manufacturer: Other
docs: https://docs.libretro.com/library/xrick/
case_insensitive_fs: true
cores: all_libretro

View File

@@ -1,170 +1,10 @@
platform: RetroBat
version: 7.5.3
homepage: "https://www.retrobat.org"
source: "https://raw.githubusercontent.com/RetroBat-Official/emulatorlauncher/master/batocera-systems/Resources/batocera-systems.json"
homepage: https://www.retrobat.org
source: https://raw.githubusercontent.com/RetroBat-Official/emulatorlauncher/master/batocera-systems/Resources/batocera-systems.json
base_destination: bios
hash_type: md5
verification_mode: md5
case_insensitive_fs: true
cores:
- 81
- a5200
- abuse
- arduous
- atari800
- azahar
- bennugd
- bk
- bluemsx
- bsnes
- bstone
- cannonball
- cap32
- catacombgl
- cdogs
- cemu
- cgenius
- citron
- clk
- corsixth
- demul
- devilutionx
- dhewm3
- dice
- dolphin
- dosbox_pure
- dxx-rebirth
- easyrpg
- ecwolf
- eduke32
- eka2l1
- emuscv
- etlegacy
- fake08
- fallout1-ce
- fallout2-ce
- fbneo
- fceumm
- flatpak
- flycast
- freechaf
- freeintv
- fury
- fuse
- gambatte
- gearsystem
- genesisplusgx
- glide64mk2
- gong
- gsplus
- gw
- gzdoom
- hatari
- hcl
- hurrican
- hypseus-singe
- ikemen
- ioquake3
- iortcw
- jazz2-native
- lindbergh-loader
- lowresnx
- lutro
- mame
- mame078plus
- mednafen_lynx
- mednafen_ngp
- mednafen_supergrafx
- mednafen_wswan
- melonds
- mgba
- minivmac
- model2emu
- moonlight
- mrboom
- neocd
- np2kai
- nxengine
- o2em
- odcommander
- openbor6412
- openjazz
- openjk
- openjkdf2
- openmohaa
- opera
- pce_fast
- pcfx
- pcsx2
- pcsx_rearmed
- pd777
- picodrive
- play
- pokemini
- potator
- ppsspp
- prboom
- prosystem
- puae
- px68k
- pygame
- pyxel
- quasi88
- raze
- reminiscence
- rpcs3
- ruffle
- samcoupe
- sameduck
- scummvm
- sdlpop
- sh
- shadps4
- snes9x
- solarus
- sonic2013
- sonic3-air
- sonic-mania
- steam
- stella
- superbroswar
- supermodel
- taradino
- tgbdual
- theforceengine
- theodore
- thextech
- tic80
- tr1x
- tr2x
- tsugaru
- tyrian
- tyrquake
- uqm
- uzem
- vb
- vecx
- vice_x64
- vircon32
- virtualjaguar
- vita3k
- vox_official
- vpinball
- wasm4
- wine-tkg
- x1
- x128
- x16emu
- xash3d_fwgs
- xemu
- xenia-canary
- xpet
- xplus4
- xrick
- xvic
- yabasanshiro
- yquake2
- zc210
systems:
3do:
files:
@@ -425,12 +265,12 @@ systems:
md5: 281f20ea4320404ec820fb7ec0693b38
atari7800:
files:
- name: "7800 BIOS (E).rom"
destination: "7800 BIOS (E).rom"
- name: 7800 BIOS (E).rom
destination: 7800 BIOS (E).rom
required: true
md5: 397bb566584be7b9764e7a68974c4263
- name: "7800 BIOS (U).rom"
destination: "7800 BIOS (U).rom"
- name: 7800 BIOS (U).rom
destination: 7800 BIOS (U).rom
required: true
md5: 0763f1ffb006ddbe32e52d497ee848ae
- name: ProSystem.dat
@@ -548,6 +388,40 @@ systems:
- name: bbcmc_flop.xml
destination: mame/hash/bbcmc_flop.xml
required: true
bk:
files:
- name: B11M_BOS.ROM
destination: bk/B11M_BOS.ROM
required: true
md5: fe4627d1e3a1535874085050733263e7
- name: B11M_EXT.ROM
destination: bk/B11M_EXT.ROM
required: true
md5: dc52f365d56fa1951f5d35b1101b9e3f
- name: BAS11M_0.ROM
destination: bk/BAS11M_0.ROM
required: true
md5: 946f6f23ded03c0e26187f0b3ca75993
- name: BAS11M_1.ROM
destination: bk/BAS11M_1.ROM
required: true
md5: 1e6637f32aa7d1de03510030cac40bcf
- name: DISK_327.ROM
destination: bk/DISK_327.ROM
required: true
md5: 5015228eeeb238e65da8edcd1b6dfac7
- name: BASIC10.ROM
destination: bk/BASIC10.ROM
required: true
md5: 3fa774326d75410a065659aea80252f0
- name: FOCAL10.ROM
destination: bk/FOCAL10.ROM
required: true
md5: 5737f972e8638831ab71e9139abae052
- name: MONIT10.ROM
destination: bk/MONIT10.ROM
required: true
md5: 95f8c41c6abf7640e35a6a03cecebd01
loopy:
files:
- name: casloopy.zip
@@ -845,34 +719,30 @@ systems:
md5: a6f31483d1da4558cc19025e21f95c1d
jaguar:
files:
- name: "[BIOS] Atari Jaguar (World).j64"
destination: "[BIOS] Atari Jaguar (World).j64"
- name: '[BIOS] Atari Jaguar (World).j64'
destination: '[BIOS] Atari Jaguar (World).j64'
required: true
md5: bcfe348c565d9dedb173822ee6850dea
laseractive:
files:
- name: "[BIOS] LaserActive PAC-N1 (Japan) (v1.02).bin"
destination: "laseractive/[BIOS] LaserActive PAC-N1 (Japan) (v1.02).bin"
- name: '[BIOS] LaserActive PAC-N1 (Japan) (v1.02).bin'
destination: laseractive/[BIOS] LaserActive PAC-N1 (Japan) (v1.02).bin
required: true
md5: f69f173b251d8bf7649b10a9167a10bf
- name: "[BIOS] LaserActive PAC-N10 (US) (v1.02).bin"
destination: "laseractive/[BIOS] LaserActive PAC-N10 (US) (v1.02).bin"
- name: '[BIOS] LaserActive PAC-N10 (US) (v1.02).bin'
destination: laseractive/[BIOS] LaserActive PAC-N10 (US) (v1.02).bin
required: true
md5: f0fb8a4605ac7eefbafd4f2d5a793cc8
- name: "[BIOS] LaserActive PCE-LP1 (Japan) (v1.02).bin"
destination: "laseractive/[BIOS] LaserActive PCE-LP1 (Japan) (v1.02).bin"
- name: '[BIOS] LaserActive PCE-LP1 (Japan) (v1.02).bin'
destination: laseractive/[BIOS] LaserActive PCE-LP1 (Japan) (v1.02).bin
required: true
md5: 761fea207d0eafd4cfd78da7c44cac88
- name: "Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer - Sega)(JP)(en-ja).bin"
destination: "laseractive/Pioneer LaserActive Sega PAC Boot ROM v1.02 (1993)(Pioneer\
\ - Sega)(JP)(en-ja).bin"
- name: '[BIOS] LaserActive PAC-S10 (US) (v1.04).bin'
destination: laseractive/[BIOS] LaserActive PAC-S10 (US) (v1.04).bin
required: true
md5: a5a2f9aae57d464bc66b80ee79c3da6e
- name: "Pioneer LaserActive Sega PAC Boot ROM v1.04 (1993)(Pioneer - Sega)(US).bin"
destination: "laseractive/Pioneer LaserActive Sega PAC Boot ROM v1.04 (1993)(Pioneer\
\ - Sega)(US).bin"
- name: '[BIOS] LaserActive PAC-S1 (Japan) (v1.05).bin'
destination: laseractive/[BIOS] LaserActive PAC-S1 (Japan) (v1.05).bin
required: true
md5: 0e7393cd0951d6dde818fcd4cd819466
lynx:
files:
- name: lynxboot.img
@@ -916,11 +786,11 @@ systems:
required: true
mastersystem:
files:
- name: "[BIOS] Sega Master System (USA, Europe) (v1.3).sms"
destination: "[BIOS] Sega Master System (USA, Europe) (v1.3).sms"
- name: '[BIOS] Sega Master System (USA, Europe) (v1.3).sms'
destination: '[BIOS] Sega Master System (USA, Europe) (v1.3).sms'
required: true
- name: "[BIOS] Sega Master System (Japan) (v2.1).sms"
destination: "[BIOS] Sega Master System (Japan) (v2.1).sms"
- name: '[BIOS] Sega Master System (Japan) (v2.1).sms'
destination: '[BIOS] Sega Master System (Japan) (v2.1).sms'
required: true
msx:
files:
@@ -1205,13 +1075,13 @@ systems:
md5: 64a95a4a884cf4cc15a566b856603193
ngp:
files:
- name: "[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp"
destination: "[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp"
- name: '[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp'
destination: '[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp'
required: true
ngpc:
files:
- name: "[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp"
destination: "[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp"
- name: '[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp'
destination: '[BIOS] SNK Neo Geo Pocket (Japan, Europe) (En,Ja).ngp'
required: true
odyssey2:
files:
@@ -1319,7 +1189,7 @@ systems:
destination: bios.min
required: true
md5: 1e4fb124a3a886865acb574f388c803d
ps2:
sony-playstation-2:
files:
- name: ps2-0230a-20080220.bin
destination: pcsx2/bios/ps2-0230a-20080220.bin

View File

@@ -191,6 +191,37 @@ def load_platform_config(platform_name: str, platforms_dir: str = "platforms") -
system.setdefault("files", []).append(gf)
existing.add(key)
# Merge metadata from _registry.yml. The registry is our curated source;
# the scraped YAML may be incomplete (missing cores, metadata fields).
# Registry fields supplement (not replace) the scraped config.
registry_path = os.path.join(platforms_dir, "_registry.yml")
if os.path.exists(registry_path):
reg_real = os.path.realpath(registry_path)
if reg_real not in _shared_yml_cache:
with open(registry_path) as f:
_shared_yml_cache[reg_real] = yaml.safe_load(f) or {}
reg = _shared_yml_cache[reg_real]
reg_entry = reg.get("platforms", {}).get(platform_name, {})
# Merge cores (union for lists, override for all_libretro)
reg_cores = reg_entry.get("cores")
if reg_cores is not None:
cfg_cores = config.get("cores")
if reg_cores == "all_libretro":
config["cores"] = "all_libretro"
elif isinstance(reg_cores, list) and isinstance(cfg_cores, list):
merged_set = {str(c) for c in cfg_cores} | {str(c) for c in reg_cores}
config["cores"] = sorted(merged_set)
elif isinstance(reg_cores, list) and cfg_cores is None:
config["cores"] = reg_cores
# Merge all registry fields absent from config (except cores,
# handled above with union logic). No hardcoded list — any field
# added to the registry is automatically available in the config.
for key, val in reg_entry.items():
if key != "cores" and key not in config:
config[key] = val
_platform_config_cache[cache_key] = config
return config
@@ -522,6 +553,25 @@ def resolve_local_file(
if fn.casefold() in basename_targets:
return os.path.join(root, fn), "data_dir"
# Agnostic fallback: for filename-agnostic files, find any DB file
# matching the system path prefix and size criteria
if file_entry.get("agnostic"):
agnostic_prefix = file_entry.get("agnostic_path_prefix", "")
min_size = file_entry.get("min_size", 0)
max_size = file_entry.get("max_size", float("inf"))
exact_size = file_entry.get("size")
if exact_size and not min_size:
min_size = exact_size
max_size = exact_size
if agnostic_prefix:
for _sha1, entry in files_db.items():
path = entry.get("path", "")
if not path.startswith(agnostic_prefix):
continue
size = entry.get("size", 0)
if min_size <= size <= max_size and os.path.exists(path):
return path, "agnostic_fallback"
return None, "not_found"
@@ -633,6 +683,8 @@ def load_emulator_profiles(
if not emu_path.exists():
return profiles
for f in sorted(emu_path.glob("*.yml")):
if f.name.endswith(".old.yml"):
continue
with open(f) as fh:
profile = yaml.safe_load(fh) or {}
if "emulator" not in profile:
@@ -735,6 +787,15 @@ def resolve_platform_cores(
for c in core_set
if c in core_to_profile
}
# Support "all_libretro" as a list element: combines all libretro
# profiles with explicitly listed standalone cores (e.g. RetroDECK
# ships RetroArch + standalone emulators)
if "all_libretro" in core_set or "retroarch" in core_set:
result |= {
name for name, p in profiles.items()
if "libretro" in p.get("type", "")
and p.get("type") != "alias"
}
else:
# Fallback: system ID intersection with normalization
norm_plat_systems = {_norm_system_id(s) for s in config.get("systems", {})}
@@ -882,6 +943,72 @@ def filter_systems_by_target(
return filtered
def expand_platform_declared_names(config: dict, db: dict) -> set[str]:
"""Build set of file names declared by a platform config.
Enriches the set with canonical names and aliases from the database
by resolving each platform file's MD5 through by_md5. This handles
cases where a platform declares a file under a different name than
the emulator profile (e.g. Batocera ROM1 vs gsplus ROM).
"""
declared: set[str] = set()
by_md5 = db.get("indexes", {}).get("by_md5", {})
files_db = db.get("files", {})
for system in config.get("systems", {}).values():
for fe in system.get("files", []):
name = fe.get("name", "")
if name:
declared.add(name)
md5 = fe.get("md5", "")
if not md5:
continue
# Skip multi-hash and zippedFile entries (inner ROM MD5, not file MD5)
if "," in md5 or fe.get("zippedFile"):
continue
sha1 = by_md5.get(md5.lower())
if not sha1:
continue
entry = files_db.get(sha1, {})
db_name = entry.get("name", "")
if db_name:
declared.add(db_name)
for alias in entry.get("aliases", []):
declared.add(alias)
return declared
import re
_TIMESTAMP_PATTERNS = [
re.compile(r'"generated_at":\s*"[^"]*"'), # database.json
re.compile(r'\*Auto-generated on [^*]*\*'), # README.md
re.compile(r'\*Generated on [^*]*\*'), # docs site pages
]
def write_if_changed(path: str, content: str) -> bool:
"""Write content to path only if the non-timestamp content differs.
Compares new and existing content after stripping timestamp lines.
Returns True if the file was written, False if skipped (unchanged).
"""
if os.path.exists(path):
with open(path) as f:
existing = f.read()
if _strip_timestamps(existing) == _strip_timestamps(content):
return False
with open(path, "w") as f:
f.write(content)
return True
def _strip_timestamps(text: str) -> str:
"""Remove known timestamp patterns for content comparison."""
result = text
for pattern in _TIMESTAMP_PATTERNS:
result = pattern.sub("", result)
return result
# Validation and mode filtering -extracted to validation.py for SoC.
# Re-exported below for backward compatibility.

View File

@@ -16,15 +16,27 @@ from exporter import discover_exporters
OUTPUT_FILENAMES: dict[str, str] = {
"retroarch": "System.dat",
"lakka": "System.dat",
"retropie": "System.dat",
"batocera": "batocera-systems",
"recalbox": "es_bios.xml",
"retrobat": "batocera-systems.json",
"emudeck": "checkBIOS.sh",
"retrodeck": "component_manifest.json",
"romm": "known_bios_files.json",
}
def output_filename(platform: str) -> str:
"""Return the native output filename for a platform."""
return OUTPUT_FILENAMES.get(platform, f"{platform}_bios.dat")
def output_path(platform: str, output_dir: str) -> str:
"""Return the full output path for a platform's native export.
Each platform gets its own subdirectory to avoid filename collisions
(e.g. retroarch, lakka, retropie all produce System.dat).
"""
filename = OUTPUT_FILENAMES.get(platform, f"{platform}_bios.dat")
plat_dir = Path(output_dir) / platform
plat_dir.mkdir(parents=True, exist_ok=True)
return str(plat_dir / filename)
def run(
@@ -35,8 +47,6 @@ def run(
) -> int:
"""Export truth to native formats, return exit code."""
exporters = discover_exporters()
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
errors = 0
@@ -60,7 +70,7 @@ def run(
except (FileNotFoundError, OSError):
pass
dest = str(output_path / output_filename(platform))
dest = output_path(platform, output_dir)
exporter = exporter_cls()
exporter.export(truth_data, dest, scraped_data=scraped)

View File

@@ -25,3 +25,37 @@ class BaseExporter(ABC):
@abstractmethod
def validate(self, truth_data: dict, output_path: str) -> list[str]:
"""Validate exported file against truth data, return list of issues."""
@staticmethod
def _is_pattern(name: str) -> bool:
"""Check if a filename is a placeholder pattern (not a real file)."""
return "<" in name or ">" in name or "*" in name
@staticmethod
def _dest(fe: dict) -> str:
"""Get destination path for a file entry, falling back to name."""
return fe.get("path") or fe.get("destination") or fe.get("name", "")
@staticmethod
def _display_name(
sys_id: str, scraped_sys: dict | None = None,
) -> str:
"""Get display name for a system from scraped data or slug."""
if scraped_sys:
name = scraped_sys.get("name")
if name:
return name
# Fallback: convert slug to display name with acronym handling
_UPPER = {
"3do", "cdi", "cpc", "cps1", "cps2", "cps3", "dos", "gba",
"gbc", "hle", "msx", "nes", "nds", "ngp", "psp", "psx",
"sms", "snes", "stv", "tvc", "vb", "zx",
}
parts = sys_id.replace("-", " ").split()
result = []
for p in parts:
if p.lower() in _UPPER:
result.append(p.upper())
else:
result.append(p.capitalize())
return " ".join(result)

View File

@@ -0,0 +1,111 @@
"""Exporter for Batocera batocera-systems format.
Produces a Python dict matching the exact format of
batocera-linux/batocera-scripts/scripts/batocera-systems.
"""
from __future__ import annotations
from pathlib import Path
from .base_exporter import BaseExporter
class Exporter(BaseExporter):
"""Export truth data to Batocera batocera-systems format."""
@staticmethod
def platform_name() -> str:
return "batocera"
def export(
self,
truth_data: dict,
output_path: str,
scraped_data: dict | None = None,
) -> None:
# Build native_id and display name maps from scraped data
native_map: dict[str, str] = {}
if scraped_data:
for sys_id, sys_data in scraped_data.get("systems", {}).items():
nid = sys_data.get("native_id")
if nid:
native_map[sys_id] = nid
lines: list[str] = ["systems = {", ""]
systems = truth_data.get("systems", {})
for sys_id in sorted(systems):
sys_data = systems[sys_id]
files = sys_data.get("files", [])
if not files:
continue
native_id = native_map.get(sys_id, sys_id)
scraped_sys = scraped_data.get("systems", {}).get(sys_id) if scraped_data else None
display_name = self._display_name(sys_id, scraped_sys)
# Build md5 lookup from scraped data for this system
scraped_md5: dict[str, str] = {}
if scraped_data:
s_sys = scraped_data.get("systems", {}).get(sys_id, {})
for sf in s_sys.get("files", []):
sname = sf.get("name", "").lower()
smd5 = sf.get("md5", "")
if sname and smd5:
scraped_md5[sname] = smd5
# Build biosFiles entries as compact single-line dicts
# Original format ALWAYS has md5 — use scraped md5 as fallback
bios_parts: list[str] = []
for fe in files:
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
dest = self._dest(fe)
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
if not md5:
md5 = scraped_md5.get(name.lower(), "")
# Original format requires md5 for every entry — skip without
if not md5:
continue
bios_parts.append(
f'{{ "md5": "{md5}", "file": "bios/{dest}" }}'
)
bios_str = ", ".join(bios_parts)
line = (
f' "{native_id}": '
f'{{ "name": "{display_name}", '
f'"biosFiles": [ {bios_str} ] }},'
)
lines.append(line)
lines.append("")
lines.append("}")
lines.append("")
Path(output_path).write_text("\n".join(lines), encoding="utf-8")
def validate(self, truth_data: dict, output_path: str) -> list[str]:
content = Path(output_path).read_text(encoding="utf-8")
issues: list[str] = []
for sys_data in truth_data.get("systems", {}).values():
for fe in sys_data.get("files", []):
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
# Skip entries without md5 (not exportable in this format)
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
if not md5:
continue
dest = self._dest(fe)
if dest not in content and name not in content:
issues.append(f"missing: {name}")
return issues

View File

@@ -0,0 +1,207 @@
"""Exporter for EmuDeck checkBIOS.sh format.
Produces a bash script matching the exact pattern of EmuDeck's
functions/checkBIOS.sh: per-system check functions with MD5 arrays
inside the function body, iterating over $biosPath/* files.
Two patterns:
- MD5 pattern: systems with known hashes, loop $biosPath/*, md5sum each, match
- File-exists pattern: systems with specific paths, check -f
"""
from __future__ import annotations
import re
from pathlib import Path
from .base_exporter import BaseExporter
# Map our system IDs to EmuDeck function naming conventions
_SYSTEM_CONFIG: dict[str, dict] = {
"sony-playstation": {
"func": "checkPS1BIOS",
"var": "PSXBIOS",
"array": "PSBios",
"pattern": "md5",
},
"sony-playstation-2": {
"func": "checkPS2BIOS",
"var": "PS2BIOS",
"array": "PS2Bios",
"pattern": "md5",
},
"sega-mega-cd": {
"func": "checkSegaCDBios",
"var": "SEGACDBIOS",
"array": "CDBios",
"pattern": "md5",
},
"sega-saturn": {
"func": "checkSaturnBios",
"var": "SATURNBIOS",
"array": "SaturnBios",
"pattern": "md5",
},
"sega-dreamcast": {
"func": "checkDreamcastBios",
"var": "BIOS",
"array": "hashes",
"pattern": "md5",
},
"nintendo-ds": {
"func": "checkDSBios",
"var": "BIOS",
"array": "hashes",
"pattern": "md5",
},
"nintendo-switch": {
"func": "checkCitronBios",
"pattern": "file-exists",
"firmware_path": "$biosPath/citron/firmware",
"keys_path": "$biosPath/citron/keys/prod.keys",
},
}
def _make_md5_function(cfg: dict, md5s: list[str]) -> list[str]:
"""Generate a MD5-checking function matching EmuDeck's exact pattern."""
func = cfg["func"]
var = cfg["var"]
array = cfg["array"]
md5_str = " ".join(md5s)
return [
f"{func}(){{",
"",
f'\t{var}="NULL"',
"",
'\tfor entry in "$biosPath/"*',
"\tdo",
'\t\tif [ -f "$entry" ]; then',
'\t\t\tmd5=($(md5sum "$entry"))',
f'\t\t\tif [[ "${var}" != true ]]; then',
f"\t\t\t\t{array}=({md5_str})",
f'\t\t\t\tfor i in "${{{array}[@]}}"',
"\t\t\t\tdo",
'\t\t\t\tif [[ "$md5" == *"${i}"* ]]; then',
f"\t\t\t\t\t{var}=true",
"\t\t\t\t\tbreak",
"\t\t\t\telse",
f"\t\t\t\t\t{var}=false",
"\t\t\t\tfi",
"\t\t\t\tdone",
"\t\t\tfi",
"\t\tfi",
"\tdone",
"",
"",
f"\tif [ ${var} == true ]; then",
'\t\techo "$entry true";',
"\telse",
'\t\techo "false";',
"\tfi",
"}",
]
def _make_file_exists_function(cfg: dict) -> list[str]:
"""Generate a file-exists function matching EmuDeck's pattern."""
func = cfg["func"]
firmware = cfg.get("firmware_path", "")
keys = cfg.get("keys_path", "")
return [
f"{func}(){{",
"",
f'\tlocal FIRMWARE="{firmware}"',
f'\tlocal KEYS="{keys}"',
'\tif [[ -f "$KEYS" ]] && [[ "$( ls -A "$FIRMWARE")" ]]; then',
'\t\t\techo "true";',
"\telse",
'\t\t\techo "false";',
"\tfi",
"}",
]
class Exporter(BaseExporter):
"""Export truth data to EmuDeck checkBIOS.sh format."""
@staticmethod
def platform_name() -> str:
return "emudeck"
def export(
self,
truth_data: dict,
output_path: str,
scraped_data: dict | None = None,
) -> None:
lines: list[str] = ["#!/bin/bash"]
systems = truth_data.get("systems", {})
for sys_id, cfg in sorted(_SYSTEM_CONFIG.items(), key=lambda x: x[1]["func"]):
sys_data = systems.get(sys_id)
if not sys_data:
continue
lines.append("")
if cfg["pattern"] == "md5":
md5s: list[str] = []
for fe in sys_data.get("files", []):
name = fe.get("name", "")
if self._is_pattern(name) or name.startswith("_"):
continue
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5s.extend(m for m in md5 if m and re.fullmatch(r"[a-f0-9]{32}", m))
elif md5 and re.fullmatch(r"[a-f0-9]{32}", md5):
md5s.append(md5)
if md5s:
lines.extend(_make_md5_function(cfg, md5s))
elif cfg["pattern"] == "file-exists":
lines.extend(_make_file_exists_function(cfg))
lines.append("")
Path(output_path).write_text("\n".join(lines), encoding="utf-8")
def validate(self, truth_data: dict, output_path: str) -> list[str]:
content = Path(output_path).read_text(encoding="utf-8")
issues: list[str] = []
systems = truth_data.get("systems", {})
for sys_id, cfg in _SYSTEM_CONFIG.items():
if cfg["pattern"] != "md5":
continue
sys_data = systems.get(sys_id)
if not sys_data:
continue
for fe in sys_data.get("files", []):
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
if md5 and re.fullmatch(r"[a-f0-9]{32}", md5) and md5 not in content:
issues.append(f"missing md5: {md5} ({fe.get('name', '')})")
for sys_id, cfg in _SYSTEM_CONFIG.items():
func = cfg["func"]
if func in content:
continue
sys_data = systems.get(sys_id)
if not sys_data or not sys_data.get("files"):
continue
# Only flag if the system has usable data for the function type
if cfg["pattern"] == "md5":
has_md5 = any(
fe.get("md5") and isinstance(fe.get("md5"), str)
and re.fullmatch(r"[a-f0-9]{32}", fe["md5"])
for fe in sys_data["files"]
)
if has_md5:
issues.append(f"missing function: {func}")
elif cfg["pattern"] == "file-exists":
issues.append(f"missing function: {func}")
return issues

View File

@@ -0,0 +1,17 @@
"""Exporter for Lakka (System.dat format, same as RetroArch).
Lakka inherits RetroArch cores and uses the same System.dat format.
Delegates to systemdat_exporter for export and validation.
"""
from __future__ import annotations
from .systemdat_exporter import Exporter as SystemDatExporter
class Exporter(SystemDatExporter):
"""Export truth data to Lakka System.dat format."""
@staticmethod
def platform_name() -> str:
return "lakka"

View File

@@ -0,0 +1,130 @@
"""Exporter for Recalbox es_bios.xml format.
Produces XML matching the exact format of recalbox's es_bios.xml:
- XML namespace declaration
- <system fullname="..." platform="...">
- <bios path="system/file" md5="..." core="..." /> with optional mandatory, hashMatchMandatory, note
- mandatory absent = true (only explicit when false)
- 2-space indentation
"""
from __future__ import annotations
from pathlib import Path
from .base_exporter import BaseExporter
class Exporter(BaseExporter):
"""Export truth data to Recalbox es_bios.xml format."""
@staticmethod
def platform_name() -> str:
return "recalbox"
def export(
self,
truth_data: dict,
output_path: str,
scraped_data: dict | None = None,
) -> None:
native_map: dict[str, str] = {}
if scraped_data:
for sys_id, sys_data in scraped_data.get("systems", {}).items():
nid = sys_data.get("native_id")
if nid:
native_map[sys_id] = nid
lines: list[str] = [
'<?xml version="1.0" encoding="UTF-8"?>',
'<biosList xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"'
' xsi:noNamespaceSchemaLocation="es_bios.xsd">',
]
systems = truth_data.get("systems", {})
for sys_id in sorted(systems):
sys_data = systems[sys_id]
files = sys_data.get("files", [])
if not files:
continue
native_id = native_map.get(sys_id, sys_id)
scraped_sys = scraped_data.get("systems", {}).get(sys_id) if scraped_data else None
display_name = self._display_name(sys_id, scraped_sys)
lines.append(f' <system fullname="{display_name}" platform="{native_id}">')
# Build path lookup from scraped data for this system
scraped_paths: dict[str, str] = {}
if scraped_data:
s_sys = scraped_data.get("systems", {}).get(sys_id, {})
for sf in s_sys.get("files", []):
sname = sf.get("name", "").lower()
spath = sf.get("destination", sf.get("name", ""))
if sname and spath:
scraped_paths[sname] = spath
for fe in files:
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
# Use scraped path when available (preserves original format)
path = scraped_paths.get(name.lower())
if not path:
dest = self._dest(fe)
path = f"{native_id}/{dest}" if "/" not in dest else dest
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = ",".join(md5)
required = fe.get("required", True)
# Build cores string from _cores
cores_list = fe.get("_cores", [])
core_str = ",".join(f"libretro/{c}" for c in cores_list) if cores_list else ""
attrs = [f'path="{path}"']
if md5:
attrs.append(f'md5="{md5}"')
if not required:
attrs.append('mandatory="false"')
if not required:
attrs.append('hashMatchMandatory="true"')
if core_str:
attrs.append(f'core="{core_str}"')
lines.append(f' <bios {" ".join(attrs)} />')
lines.append(" </system>")
lines.append("</biosList>")
lines.append("")
Path(output_path).write_text("\n".join(lines), encoding="utf-8")
def validate(self, truth_data: dict, output_path: str) -> list[str]:
from xml.etree.ElementTree import parse as xml_parse
tree = xml_parse(output_path)
root = tree.getroot()
exported_paths: set[str] = set()
for bios_el in root.iter("bios"):
path = bios_el.get("path", "")
if path:
exported_paths.add(path.lower())
exported_paths.add(path.split("/")[-1].lower())
issues: list[str] = []
for sys_data in truth_data.get("systems", {}).values():
for fe in sys_data.get("files", []):
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
dest = self._dest(fe)
if name.lower() not in exported_paths and dest.lower() not in exported_paths:
issues.append(f"missing: {name}")
return issues

View File

@@ -0,0 +1,114 @@
"""Exporter for RetroBat batocera-systems.json format.
Produces JSON matching the exact format of
RetroBat-Official/emulatorlauncher/batocera-systems/Resources/batocera-systems.json:
- System keys with "name" and "biosFiles" fields
- Each biosFile has "md5" before "file" (matching original key order)
"""
from __future__ import annotations
import json
from collections import OrderedDict
from pathlib import Path
from .base_exporter import BaseExporter
class Exporter(BaseExporter):
"""Export truth data to RetroBat batocera-systems.json format."""
@staticmethod
def platform_name() -> str:
return "retrobat"
def export(
self,
truth_data: dict,
output_path: str,
scraped_data: dict | None = None,
) -> None:
native_map: dict[str, str] = {}
if scraped_data:
for sys_id, sys_data in scraped_data.get("systems", {}).items():
nid = sys_data.get("native_id")
if nid:
native_map[sys_id] = nid
output: OrderedDict[str, dict] = OrderedDict()
systems = truth_data.get("systems", {})
for sys_id in sorted(systems):
sys_data = systems[sys_id]
files = sys_data.get("files", [])
if not files:
continue
native_id = native_map.get(sys_id, sys_id)
scraped_sys = scraped_data.get("systems", {}).get(sys_id) if scraped_data else None
display_name = self._display_name(sys_id, scraped_sys)
bios_files: list[OrderedDict] = []
for fe in files:
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
dest = self._dest(fe)
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
# Original format requires md5 for every entry
if not md5:
continue
entry: OrderedDict[str, str] = OrderedDict()
entry["md5"] = md5
entry["file"] = f"bios/{dest}"
bios_files.append(entry)
if bios_files:
if native_id in output:
existing_files = {e.get("file") for e in output[native_id]["biosFiles"]}
for entry in bios_files:
if entry.get("file") not in existing_files:
output[native_id]["biosFiles"].append(entry)
else:
sys_entry: OrderedDict[str, object] = OrderedDict()
sys_entry["name"] = display_name
sys_entry["biosFiles"] = bios_files
output[native_id] = sys_entry
Path(output_path).write_text(
json.dumps(output, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
def validate(self, truth_data: dict, output_path: str) -> list[str]:
data = json.loads(Path(output_path).read_text(encoding="utf-8"))
exported_files: set[str] = set()
for sys_data in data.values():
for bf in sys_data.get("biosFiles", []):
path = bf.get("file", "")
stripped = path.removeprefix("bios/")
exported_files.add(stripped)
basename = path.split("/")[-1] if "/" in path else path
exported_files.add(basename)
issues: list[str] = []
for sys_data in truth_data.get("systems", {}).values():
for fe in sys_data.get("files", []):
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
if not md5:
continue
dest = self._dest(fe)
if name not in exported_files and dest not in exported_files:
issues.append(f"missing: {name}")
return issues

View File

@@ -0,0 +1,208 @@
"""Exporter for RetroDECK component_manifest.json format.
Produces a JSON file compatible with RetroDECK's component manifests.
Each system maps to a component with BIOS entries containing filename,
md5 (comma-separated if multiple), paths ($bios_path default), and
required status.
Path tokens: $bios_path for bios/, $roms_path for roms/.
Entries without an explicit path default to $bios_path.
"""
from __future__ import annotations
import json
import re
from collections import OrderedDict
from pathlib import Path
from .base_exporter import BaseExporter
# retrobios slug -> RetroDECK system ID (reverse of scraper SYSTEM_SLUG_MAP)
_REVERSE_SLUG: dict[str, str] = {
"nintendo-nes": "nes",
"nintendo-snes": "snes",
"nintendo-64": "n64",
"nintendo-64dd": "n64dd",
"nintendo-gamecube": "gc",
"nintendo-wii": "wii",
"nintendo-wii-u": "wiiu",
"nintendo-switch": "switch",
"nintendo-gb": "gb",
"nintendo-gbc": "gbc",
"nintendo-gba": "gba",
"nintendo-ds": "nds",
"nintendo-3ds": "3ds",
"nintendo-fds": "fds",
"nintendo-sgb": "sgb",
"nintendo-virtual-boy": "virtualboy",
"nintendo-pokemon-mini": "pokemini",
"sony-playstation": "psx",
"sony-playstation-2": "ps2",
"sony-playstation-3": "ps3",
"sony-psp": "psp",
"sony-psvita": "psvita",
"sega-mega-drive": "megadrive",
"sega-mega-cd": "megacd",
"sega-saturn": "saturn",
"sega-dreamcast": "dreamcast",
"sega-dreamcast-arcade": "naomi",
"sega-game-gear": "gamegear",
"sega-master-system": "mastersystem",
"nec-pc-engine": "pcengine",
"nec-pc-fx": "pcfx",
"nec-pc-98": "pc98",
"nec-pc-88": "pc88",
"3do": "3do",
"amstrad-cpc": "amstradcpc",
"arcade": "arcade",
"atari-400-800": "atari800",
"atari-5200": "atari5200",
"atari-7800": "atari7800",
"atari-jaguar": "atarijaguar",
"atari-lynx": "atarilynx",
"atari-st": "atarist",
"commodore-c64": "c64",
"commodore-amiga": "amiga",
"philips-cdi": "cdimono1",
"fairchild-channel-f": "channelf",
"coleco-colecovision": "colecovision",
"mattel-intellivision": "intellivision",
"microsoft-msx": "msx",
"microsoft-xbox": "xbox",
"doom": "doom",
"j2me": "j2me",
"apple-macintosh-ii": "macintosh",
"apple-ii": "apple2",
"apple-iigs": "apple2gs",
"enterprise-64-128": "enterprise",
"tiger-game-com": "gamecom",
"hartung-game-master": "gmaster",
"epoch-scv": "scv",
"watara-supervision": "supervision",
"bandai-wonderswan": "wonderswan",
"snk-neogeo-cd": "neogeocd",
"tandy-coco": "coco",
"tandy-trs-80": "trs80",
"dragon-32-64": "dragon",
"pico8": "pico8",
"wolfenstein-3d": "wolfenstein",
"sinclair-zx-spectrum": "zxspectrum",
}
def _dest_to_path_token(destination: str) -> str:
"""Convert a truth destination path to a RetroDECK path token."""
if destination.startswith("roms/"):
return "$roms_path/" + destination.removeprefix("roms/")
if destination.startswith("bios/"):
return "$bios_path/" + destination.removeprefix("bios/")
# Default: bios path
return "$bios_path/" + destination
class Exporter(BaseExporter):
"""Export truth data to RetroDECK component_manifest.json format."""
@staticmethod
def platform_name() -> str:
return "retrodeck"
def export(
self,
truth_data: dict,
output_path: str,
scraped_data: dict | None = None,
) -> None:
native_map: dict[str, str] = {}
if scraped_data:
for sys_id, sys_data in scraped_data.get("systems", {}).items():
nid = sys_data.get("native_id")
if nid:
native_map[sys_id] = nid
manifest: OrderedDict[str, dict] = OrderedDict()
systems = truth_data.get("systems", {})
for sys_id in sorted(systems):
sys_data = systems[sys_id]
files = sys_data.get("files", [])
if not files:
continue
native_id = native_map.get(sys_id, _REVERSE_SLUG.get(sys_id, sys_id))
bios_entries: list[OrderedDict] = []
for fe in files:
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
dest = self._dest(fe)
path_token = _dest_to_path_token(dest)
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = ",".join(m for m in md5 if m)
required = fe.get("required", True)
entry: OrderedDict[str, object] = OrderedDict()
entry["filename"] = name
if md5:
# Validate MD5 entries
parts = [
m.strip().lower()
for m in str(md5).split(",")
if re.fullmatch(r"[0-9a-f]{32}", m.strip())
]
if parts:
entry["md5"] = ",".join(parts) if len(parts) > 1 else parts[0]
entry["paths"] = path_token
entry["required"] = required
system_val = native_id
entry["system"] = system_val
bios_entries.append(entry)
if bios_entries:
if native_id in manifest:
# Merge into existing component (multiple truth systems
# may map to the same native ID)
existing_names = {e["filename"] for e in manifest[native_id]["bios"]}
for entry in bios_entries:
if entry["filename"] not in existing_names:
manifest[native_id]["bios"].append(entry)
else:
component = OrderedDict()
component["system"] = native_id
component["bios"] = bios_entries
manifest[native_id] = component
Path(output_path).write_text(
json.dumps(manifest, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
def validate(self, truth_data: dict, output_path: str) -> list[str]:
data = json.loads(Path(output_path).read_text(encoding="utf-8"))
exported_names: set[str] = set()
for comp_data in data.values():
bios = comp_data.get("bios", [])
if isinstance(bios, list):
for entry in bios:
fn = entry.get("filename", "")
if fn:
exported_names.add(fn)
issues: list[str] = []
for sys_data in truth_data.get("systems", {}).values():
for fe in sys_data.get("files", []):
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
if name not in exported_names:
issues.append(f"missing: {name}")
return issues

View File

@@ -0,0 +1,17 @@
"""Exporter for RetroPie (System.dat format, same as RetroArch).
RetroPie inherits RetroArch cores and uses the same System.dat format.
Delegates to systemdat_exporter for export and validation.
"""
from __future__ import annotations
from .systemdat_exporter import Exporter as SystemDatExporter
class Exporter(SystemDatExporter):
"""Export truth data to RetroPie System.dat format."""
@staticmethod
def platform_name() -> str:
return "retropie"

View File

@@ -0,0 +1,160 @@
"""Exporter for RomM known_bios_files.json format.
Produces JSON matching the exact format of
rommapp/romm/backend/models/fixtures/known_bios_files.json:
- Keys are "igdb_slug:filename"
- Values contain size, crc, md5, sha1 (all optional but at least one hash)
- Hashes are lowercase hex strings
- Size is an integer
"""
from __future__ import annotations
import json
from collections import OrderedDict
from pathlib import Path
from .base_exporter import BaseExporter
# retrobios slug -> IGDB slug (reverse of scraper SLUG_MAP)
_REVERSE_SLUG: dict[str, str] = {
"3do": "3do",
"nintendo-64dd": "64dd",
"amstrad-cpc": "acpc",
"commodore-amiga": "amiga",
"arcade": "arcade",
"atari-st": "atari-st",
"atari-5200": "atari5200",
"atari-7800": "atari7800",
"atari-400-800": "atari8bit",
"coleco-colecovision": "colecovision",
"sega-dreamcast": "dc",
"doom": "doom",
"enterprise-64-128": "enterprise",
"fairchild-channel-f": "fairchild-channel-f",
"nintendo-fds": "fds",
"sega-game-gear": "gamegear",
"nintendo-gb": "gb",
"nintendo-gba": "gba",
"nintendo-gbc": "gbc",
"sega-mega-drive": "genesis",
"mattel-intellivision": "intellivision",
"j2me": "j2me",
"atari-lynx": "lynx",
"apple-macintosh-ii": "mac",
"microsoft-msx": "msx",
"nintendo-ds": "nds",
"snk-neogeo-cd": "neo-geo-cd",
"nintendo-nes": "nes",
"nintendo-gamecube": "ngc",
"magnavox-odyssey2": "odyssey-2-slash-videopac-g7000",
"nec-pc-98": "pc-9800-series",
"nec-pc-fx": "pc-fx",
"nintendo-pokemon-mini": "pokemon-mini",
"sony-playstation-2": "ps2",
"sony-psp": "psp",
"sony-playstation": "psx",
"nintendo-satellaview": "satellaview",
"sega-saturn": "saturn",
"scummvm": "scummvm",
"sega-mega-cd": "segacd",
"sharp-x68000": "sharp-x68000",
"sega-master-system": "sms",
"nintendo-snes": "snes",
"nintendo-sufami-turbo": "sufami-turbo",
"nintendo-sgb": "super-gb",
"nec-pc-engine": "tg16",
"videoton-tvc": "tvc",
"philips-videopac": "videopac-g7400",
"wolfenstein-3d": "wolfenstein",
"sharp-x1": "x1",
"microsoft-xbox": "xbox",
"sinclair-zx-spectrum": "zxs",
}
class Exporter(BaseExporter):
"""Export truth data to RomM known_bios_files.json format."""
@staticmethod
def platform_name() -> str:
return "romm"
def export(
self,
truth_data: dict,
output_path: str,
scraped_data: dict | None = None,
) -> None:
native_map: dict[str, str] = {}
if scraped_data:
for sys_id, sys_data in scraped_data.get("systems", {}).items():
nid = sys_data.get("native_id")
if nid:
native_map[sys_id] = nid
output: OrderedDict[str, dict] = OrderedDict()
systems = truth_data.get("systems", {})
for sys_id in sorted(systems):
sys_data = systems[sys_id]
files = sys_data.get("files", [])
if not files:
continue
igdb_slug = native_map.get(sys_id, _REVERSE_SLUG.get(sys_id, sys_id))
for fe in files:
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
key = f"{igdb_slug}:{name}"
entry: OrderedDict[str, object] = OrderedDict()
size = fe.get("size")
if size is not None:
entry["size"] = int(size)
crc = fe.get("crc32", "")
if crc:
entry["crc"] = str(crc).strip().lower()
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
if md5:
entry["md5"] = str(md5).strip().lower()
sha1 = fe.get("sha1", "")
if isinstance(sha1, list):
sha1 = sha1[0] if sha1 else ""
if sha1:
entry["sha1"] = str(sha1).strip().lower()
output[key] = entry
Path(output_path).write_text(
json.dumps(output, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
def validate(self, truth_data: dict, output_path: str) -> list[str]:
data = json.loads(Path(output_path).read_text(encoding="utf-8"))
exported_names: set[str] = set()
for key in data:
if ":" in key:
_, filename = key.split(":", 1)
exported_names.add(filename)
issues: list[str] = []
for sys_data in truth_data.get("systems", {}).values():
for fe in sys_data.get("files", []):
name = fe.get("name", "")
if name.startswith("_") or self._is_pattern(name):
continue
if name not in exported_names:
issues.append(f"missing: {name}")
return issues

View File

@@ -1,4 +1,8 @@
"""Exporter for libretro System.dat (clrmamepro DAT format)."""
"""Exporter for libretro System.dat (clrmamepro DAT format).
Produces a single 'game' block with all ROMs grouped by system,
matching the exact format of libretro-database/dat/System.dat.
"""
from __future__ import annotations
@@ -13,7 +17,7 @@ from .base_exporter import BaseExporter
def _slug_to_native(slug: str) -> str:
"""Convert a system slug to a native 'Manufacturer - Console' name."""
"""Convert a system slug to 'Manufacturer - Console' format."""
parts = slug.split("-", 1)
if len(parts) == 1:
return parts[0].title()
@@ -42,45 +46,69 @@ class Exporter(BaseExporter):
if nid:
native_map[sys_id] = nid
lines: list[str] = []
lines.append('clrmamepro (')
lines.append('\tname "System.dat"')
lines.append(')')
# Match exact header format of libretro-database/dat/System.dat
version = ""
if scraped_data:
version = scraped_data.get("dat_version", scraped_data.get("version", ""))
lines: list[str] = [
"clrmamepro (",
'\tname "System"',
'\tdescription "System"',
'\tcomment "System, firmware, and BIOS files used by libretro cores."',
]
if version:
lines.append(f"\tversion {version}")
lines.extend([
'\tauthor "libretro"',
'\thomepage "https://github.com/libretro/libretro-database/blob/master/dat/System.dat"',
'\turl "https://raw.githubusercontent.com/libretro/libretro-database/master/dat/System.dat"',
")",
"",
"game (",
'\tname "System"',
'\tcomment "System"',
])
systems = truth_data.get("systems", {})
for sys_id in sorted(systems):
sys_data = systems[sys_id]
native_name = native_map.get(sys_id, _slug_to_native(sys_id))
files = sys_data.get("files", [])
if not files:
continue
for fe in sys_data.get("files", []):
native_name = native_map.get(sys_id, _slug_to_native(sys_id))
lines.append("")
lines.append(f'\tcomment "{native_name}"')
for fe in files:
name = fe.get("name", "")
if name.startswith("_"):
if name.startswith("_") or self._is_pattern(name):
continue
dest = fe.get("path", name)
size = fe.get("size", 0)
# Quote names with spaces or special chars (matching original format)
needs_quote = " " in name or "(" in name or ")" in name
name_str = f'"{name}"' if needs_quote else name
rom_parts = [f"name {name_str}"]
size = fe.get("size")
if size:
rom_parts.append(f"size {size}")
crc = fe.get("crc32", "")
md5 = fe.get("md5", "")
sha1 = fe.get("sha1", "")
rom_parts = [f'name "{name}"']
rom_parts.append(f"size {size}")
if crc:
rom_parts.append(f"crc {crc}")
rom_parts.append(f"crc {crc.upper()}")
md5 = fe.get("md5", "")
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
if md5:
rom_parts.append(f"md5 {md5}")
sha1 = fe.get("sha1", "")
if isinstance(sha1, list):
sha1 = sha1[0] if sha1 else ""
if sha1:
rom_parts.append(f"sha1 {sha1}")
rom_str = " ".join(rom_parts)
game_name = f"{native_name}/{dest}"
lines.append("")
lines.append("game (")
lines.append(f'\tname "{game_name}"')
lines.append(f'\tdescription "{name}"')
lines.append(f"\trom ( {rom_str} )")
lines.append(")")
lines.append(f"\trom ( {' '.join(rom_parts)} )")
lines.append(")")
lines.append("")
Path(output_path).write_text("\n".join(lines), encoding="utf-8")
@@ -93,12 +121,11 @@ class Exporter(BaseExporter):
exported_names.add(rom.name)
issues: list[str] = []
for sys_id, sys_data in truth_data.get("systems", {}).items():
for sys_data in truth_data.get("systems", {}).values():
for fe in sys_data.get("files", []):
name = fe.get("name", "")
if name.startswith("_"):
if name.startswith("_") or self._is_pattern(name):
continue
if name not in exported_names:
issues.append(f"missing: {name} (system {sys_id})")
issues.append(f"missing: {name}")
return issues

View File

@@ -18,7 +18,7 @@ from datetime import datetime, timezone
from pathlib import Path
sys.path.insert(0, os.path.dirname(__file__))
from common import compute_hashes, list_registered_platforms
from common import compute_hashes, list_registered_platforms, write_if_changed
CACHE_DIR = ".cache"
CACHE_FILE = os.path.join(CACHE_DIR, "db_cache.json")
@@ -315,14 +315,15 @@ def main():
"indexes": indexes,
}
with open(args.output, "w") as f:
json.dump(database, f, indent=2)
new_content = json.dumps(database, indent=2)
written = write_if_changed(args.output, new_content)
save_cache(CACHE_FILE, new_cache)
alias_count = sum(len(v) for v in aliases.values())
name_count = len(indexes["by_name"])
print(f"Generated {args.output}: {len(files)} files, {total_size:,} bytes total")
status = "Generated" if written else "Unchanged"
print(f"{status} {args.output}: {len(files)} files, {total_size:,} bytes total")
print(f" Name index: {name_count} names ({alias_count} aliases)")
return 0
@@ -400,6 +401,8 @@ def _collect_all_aliases(files: dict) -> dict:
try:
import yaml
for emu_file in emulators_dir.glob("*.yml"):
if emu_file.name.endswith(".old.yml"):
continue
try:
with open(emu_file) as f:
emu_config = yaml.safe_load(f) or {}

View File

@@ -28,7 +28,7 @@ sys.path.insert(0, os.path.dirname(__file__))
from common import (
MANUFACTURER_PREFIXES,
build_target_cores_cache, build_zip_contents_index, check_inside_zip,
compute_hashes, fetch_large_file, group_identical_platforms,
compute_hashes, expand_platform_declared_names, fetch_large_file, group_identical_platforms,
list_emulator_profiles, list_platform_system_ids, list_registered_platforms,
filter_systems_by_target, list_system_ids, load_database,
load_data_dir_registry, load_emulator_profiles, load_platform_config,
@@ -371,12 +371,8 @@ def _collect_emulator_extras(
by_path_suffix = db.get("indexes", {}).get("by_path_suffix", {})
# Build set of filenames already covered (platform baseline + first pass extras)
covered_names: set[str] = set()
for sys_id, system in config.get("systems", {}).items():
for fe in system.get("files", []):
n = fe.get("name", "")
if n:
covered_names.add(n)
# Enriched with canonical names from DB via MD5 (handles platform renaming)
covered_names = expand_platform_declared_names(config, db)
for e in extras:
covered_names.add(e["name"])
@@ -426,6 +422,91 @@ def _collect_emulator_extras(
"source_emulator": profile.get("emulator", emu_name),
})
# Third pass: agnostic scan — for filename-agnostic cores, include all
# DB files matching the system path prefix and size criteria.
files_db = db.get("files", {})
for emu_name, profile in sorted(profiles.items()):
if profile.get("type") in ("launcher", "alias"):
continue
if emu_name not in relevant:
continue
is_profile_agnostic = profile.get("bios_mode") == "agnostic"
if not is_profile_agnostic:
if not any(f.get("agnostic") for f in profile.get("files", [])):
continue
for f in profile.get("files", []):
if not is_profile_agnostic and not f.get("agnostic"):
continue
fname = f.get("name", "")
if not fname:
continue
# Derive path prefix from the representative file in the DB
path_prefix = None
sha1_list = by_name.get(fname, [])
for sha1 in sha1_list:
entry = files_db.get(sha1, {})
path = entry.get("path", "")
if path:
parts = path.rsplit("/", 1)
if len(parts) == 2:
path_prefix = parts[0] + "/"
break
if not path_prefix:
# Fallback: try other files in the profile for the same system
for other_f in profile.get("files", []):
if other_f is f:
continue
other_name = other_f.get("name", "")
for sha1 in by_name.get(other_name, []):
entry = files_db.get(sha1, {})
path = entry.get("path", "")
if path:
parts = path.rsplit("/", 1)
if len(parts) == 2:
path_prefix = parts[0] + "/"
break
if path_prefix:
break
if not path_prefix:
continue
# Size criteria from the file entry
min_size = f.get("min_size", 0)
max_size = f.get("max_size", float("inf"))
exact_size = f.get("size")
if exact_size and not min_size:
min_size = exact_size
max_size = exact_size
# Scan DB for all files under this prefix matching size
for sha1, entry in files_db.items():
path = entry.get("path", "")
if not path.startswith(path_prefix):
continue
size = entry.get("size", 0)
if not (min_size <= size <= max_size):
continue
scan_name = entry.get("name", "")
if not scan_name:
continue
dest = scan_name
full_dest = f"{base_dest}/{dest}" if base_dest else dest
if full_dest in seen_dests:
continue
seen_dests.add(full_dest)
extras.append({
"name": scan_name,
"destination": dest,
"required": False,
"hle_fallback": False,
"source_emulator": profile.get("emulator", emu_name),
"agnostic_scan": True,
})
return extras
@@ -625,6 +706,24 @@ def _build_readme(platform_name: str, platform_display: str,
return header + guide + footer
def _build_agnostic_rename_readme(
destination: str, original: str, alternatives: list[str],
) -> str:
"""Build a README explaining an agnostic file rename."""
lines = [
"This file was renamed for compatibility:",
f" {destination} <- {original}",
"",
]
if alternatives:
lines.append("All variants included in this pack:")
for alt in sorted(alternatives):
lines.append(f" {alt}")
lines.append("")
lines.append(f"To use a different variant, rename it to: {destination}")
return "\n".join(lines) + "\n"
def generate_pack(
platform_name: str,
platforms_dir: str,
@@ -792,10 +891,71 @@ def generate_pack(
continue
if status == "not_found":
if not already_packed:
missing_files.append(file_entry["name"])
file_status[dedup_key] = "missing"
continue
# Agnostic fallback: if an agnostic core covers this system,
# find any matching file in the DB
by_name = db.get("indexes", {}).get("by_name", {})
files_db = db.get("files", {})
agnostic_path = None
agnostic_resolved = False
if emu_profiles:
for _emu_key, _emu_prof in emu_profiles.items():
if _emu_prof.get("bios_mode") != "agnostic":
continue
if sys_id not in set(_emu_prof.get("systems", [])):
continue
for _ef in _emu_prof.get("files", []):
ef_name = _ef.get("name", "")
for _sha1 in by_name.get(ef_name, []):
_entry = files_db.get(_sha1, {})
_path = _entry.get("path", "")
if _path:
_prefix = _path.rsplit("/", 1)[0] + "/"
_min = _ef.get("min_size", 0)
_max = _ef.get("max_size", float("inf"))
if _ef.get("size") and not _min:
_min = _ef["size"]
_max = _ef["size"]
for _s, _e in files_db.items():
if _e.get("path", "").startswith(_prefix):
if _min <= _e.get("size", 0) <= _max:
if os.path.exists(_e["path"]):
local_path = _e["path"]
agnostic_path = _prefix
agnostic_resolved = True
break
break
if agnostic_resolved:
break
if agnostic_resolved:
break
if agnostic_resolved and local_path:
# Write rename README
original_name = os.path.basename(local_path)
dest_name = file_entry.get("name", "")
if original_name != dest_name and agnostic_path:
alt_names = []
for _s, _e in files_db.items():
_p = _e.get("path", "")
if _p.startswith(agnostic_path):
_n = _e.get("name", "")
if _n and _n != original_name:
alt_names.append(_n)
readme_text = _build_agnostic_rename_readme(
dest_name, original_name, alt_names,
)
readme_name = f"RENAMED_{dest_name}.txt"
readme_full = f"{base_dest}/{readme_name}" if base_dest else readme_name
if readme_full not in seen_destinations:
zf.writestr(readme_full, readme_text)
seen_destinations.add(readme_full)
status = "agnostic_fallback"
# Fall through to normal packing below
else:
if not already_packed:
missing_files.append(file_entry["name"])
file_status[dedup_key] = "missing"
continue
if status == "hash_mismatch" and verification_mode != "existence":
zf_name = file_entry.get("zipped_file")

View File

@@ -18,7 +18,7 @@ from datetime import datetime, timezone
from pathlib import Path
sys.path.insert(0, os.path.dirname(__file__))
from common import list_registered_platforms, load_database, load_platform_config
from common import list_registered_platforms, load_database, load_platform_config, write_if_changed
from verify import verify_platform
def compute_coverage(platform_name: str, platforms_dir: str, db: dict) -> dict:
@@ -91,6 +91,7 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
emulator_count = sum(
1 for f in Path("emulators").glob("*.yml")
if not f.name.endswith(".old.yml")
) if Path("emulators").exists() else 0
# Count systems from emulator profiles
@@ -100,6 +101,8 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
try:
import yaml
for f in emu_dir.glob("*.yml"):
if f.name.endswith(".old.yml"):
continue
with open(f) as fh:
p = yaml.safe_load(fh) or {}
system_ids.update(p.get("systems", []))
@@ -316,14 +319,12 @@ def main():
db = load_database(args.db)
readme = generate_readme(db, args.platforms_dir)
with open("README.md", "w") as f:
f.write(readme)
print(f"Generated ./README.md")
status = "Generated" if write_if_changed("README.md", readme) else "Unchanged"
print(f"{status} ./README.md")
contributing = generate_contributing()
with open("CONTRIBUTING.md", "w") as f:
f.write(contributing)
print(f"Generated ./CONTRIBUTING.md")
status = "Generated" if write_if_changed("CONTRIBUTING.md", contributing) else "Unchanged"
print(f"{status} ./CONTRIBUTING.md")
if __name__ == "__main__":

View File

@@ -20,7 +20,7 @@ from datetime import datetime, timezone
from pathlib import Path
sys.path.insert(0, os.path.dirname(__file__))
from common import list_registered_platforms, load_database, load_emulator_profiles, load_platform_config, require_yaml
from common import list_registered_platforms, load_database, load_emulator_profiles, load_platform_config, require_yaml, write_if_changed
yaml = require_yaml()
from generate_readme import compute_coverage
@@ -1358,6 +1358,8 @@ The CI automatically:
# Wiki pages
# index, architecture, tools, profiling are maintained as wiki/ sources
# and copied verbatim by main(). Only data-model is generated dynamically.
def generate_wiki_index() -> str:
"""Generate wiki landing page."""
@@ -1994,10 +1996,19 @@ def generate_mkdocs_nav(
wiki_nav = [
{"Overview": "wiki/index.md"},
{"Getting started": "wiki/getting-started.md"},
{"FAQ": "wiki/faq.md"},
{"Troubleshooting": "wiki/troubleshooting.md"},
{"Architecture": "wiki/architecture.md"},
{"Tools": "wiki/tools.md"},
{"Profiling guide": "wiki/profiling.md"},
{"Advanced usage": "wiki/advanced-usage.md"},
{"Verification modes": "wiki/verification-modes.md"},
{"Data model": "wiki/data-model.md"},
{"Profiling guide": "wiki/profiling.md"},
{"Adding a platform": "wiki/adding-a-platform.md"},
{"Adding a scraper": "wiki/adding-a-scraper.md"},
{"Testing guide": "wiki/testing-guide.md"},
{"Release process": "wiki/release-process.md"},
]
return [
@@ -2064,7 +2075,7 @@ def main():
# Generate home
print("Generating home page...")
(docs / "index.md").write_text(generate_home(db, coverages, profiles, registry))
write_if_changed(str(docs / "index.md"), generate_home(db, coverages, profiles, registry))
# Build system_id -> manufacturer page map (needed by all generators)
print("Building system cross-reference map...")
@@ -2074,37 +2085,35 @@ def main():
# Generate platform pages
print("Generating platform pages...")
(docs / "platforms" / "index.md").write_text(generate_platform_index(coverages))
write_if_changed(str(docs / "platforms" / "index.md"), generate_platform_index(coverages))
for name, cov in coverages.items():
(docs / "platforms" / f"{name}.md").write_text(generate_platform_page(name, cov, registry, emulator_files))
write_if_changed(str(docs / "platforms" / f"{name}.md"), generate_platform_page(name, cov, registry, emulator_files))
# Generate system pages
print("Generating system pages...")
(docs / "systems" / "index.md").write_text(generate_systems_index(manufacturers))
write_if_changed(str(docs / "systems" / "index.md"), generate_systems_index(manufacturers))
for mfr, consoles in manufacturers.items():
slug = mfr.lower().replace(" ", "-")
page = generate_system_page(mfr, consoles, platform_files, emulator_files)
(docs / "systems" / f"{slug}.md").write_text(page)
write_if_changed(str(docs / "systems" / f"{slug}.md"), page)
# Generate emulator pages
print("Generating emulator pages...")
(docs / "emulators" / "index.md").write_text(generate_emulators_index(profiles))
write_if_changed(str(docs / "emulators" / "index.md"), generate_emulators_index(profiles))
for name, profile in profiles.items():
page = generate_emulator_page(name, profile, db, platform_files)
(docs / "emulators" / f"{name}.md").write_text(page)
write_if_changed(str(docs / "emulators" / f"{name}.md"), page)
# Generate cross-reference page
print("Generating cross-reference page...")
(docs / "cross-reference.md").write_text(
generate_cross_reference(coverages, profiles)
)
write_if_changed(str(docs / "cross-reference.md"),
generate_cross_reference(coverages, profiles))
# Generate gap analysis page
print("Generating gap analysis page...")
(docs / "gaps.md").write_text(
generate_gap_analysis(profiles, coverages, db)
)
write_if_changed(str(docs / "gaps.md"),
generate_gap_analysis(profiles, coverages, db))
# Wiki pages: copy manually maintained sources + generate dynamic ones
print("Generating wiki pages...")
@@ -2115,11 +2124,11 @@ def main():
for src_file in wiki_src.glob("*.md"):
shutil.copy2(src_file, wiki_dest / src_file.name)
# data-model.md is generated (contains live DB stats)
(wiki_dest / "data-model.md").write_text(generate_wiki_data_model(db, profiles))
write_if_changed(str(wiki_dest / "data-model.md"), generate_wiki_data_model(db, profiles))
# Generate contributing
print("Generating contributing page...")
(docs / "contributing.md").write_text(generate_contributing())
write_if_changed(str(docs / "contributing.md"), generate_contributing())
# Update mkdocs.yml nav section only (avoid yaml.dump round-trip mangling quotes)
print("Updating mkdocs.yml nav...")
@@ -2167,15 +2176,17 @@ markdown_extensions:
- toc:
permalink: true
- pymdownx.details
- pymdownx.superfences
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_code_format
- pymdownx.tabbed:
alternate_style: true
plugins:
- search
"""
with open("mkdocs.yml", "w") as f:
f.write(mkdocs_static)
f.write(nav_yaml)
write_if_changed("mkdocs.yml", mkdocs_static + nav_yaml)
total_pages = (
1 # home
@@ -2184,7 +2195,7 @@ plugins:
+ 1 # cross-reference
+ 1 + len(profiles) # emulator index + detail
+ 1 # gap analysis
+ 5 # wiki (index, architecture, tools, profiling, data model)
+ 14 # wiki pages (copied from wiki/ + generated data-model)
+ 1 # contributing
)
print(f"\nGenerated {total_pages} pages in {args.docs_dir}/")

View File

@@ -177,6 +177,28 @@ def main():
print("\n--- 2/9 refresh data directories: SKIPPED (--offline) ---")
results["refresh_data"] = True
# Step 2a: Refresh MAME BIOS hashes
if not args.offline:
ok, _ = run(
[sys.executable, "-m", "scripts.scraper.mame_hash_scraper"],
"2a refresh MAME hashes",
)
results["mame_hashes"] = ok
else:
print("\n--- 2a refresh MAME hashes: SKIPPED (--offline) ---")
results["mame_hashes"] = True
# Step 2a2: Refresh FBNeo BIOS hashes
if not args.offline:
ok, _ = run(
[sys.executable, "-m", "scripts.scraper.fbneo_hash_scraper"],
"2a2 refresh FBNeo hashes",
)
results["fbneo_hashes"] = ok
else:
print("\n--- 2a2 refresh FBNeo hashes: SKIPPED (--offline) ---")
results["fbneo_hashes"] = True
# Step 2b: Check buildbot system directory (non-blocking)
if args.check_buildbot and not args.offline:
ok, _ = run(

View File

@@ -0,0 +1,565 @@
"""Merge fetched hash data into emulator YAML profiles.
Supports two strategies:
- MAME: bios_zip entries with contents lists
- FBNeo: individual ROM entries grouped by archive field
"""
from __future__ import annotations
import json
import shutil
from pathlib import Path
from typing import Any
import yaml
def merge_mame_profile(
profile_path: str,
hashes_path: str,
write: bool = False,
add_new: bool = True,
) -> dict[str, Any]:
"""Merge MAME bios_zip entries from upstream hash data.
Preserves system, note, required per entry. Updates contents and
source_ref from the hashes JSON. New sets are only added when
add_new=True (main profile). Entries not in the hash data are
left untouched (the scraper only covers MACHINE_IS_BIOS_ROOT sets,
not all machine ROM sets).
If write=True, backs up existing profile to .old.yml before writing.
"""
profile = _load_yaml(profile_path)
hashes = _load_json(hashes_path)
profile['core_version'] = hashes.get('version', profile.get('core_version'))
files = profile.get('files', [])
bios_zip, non_bios = _split_files(files, lambda f: f.get('category') == 'bios_zip')
existing_by_name: dict[str, dict] = {}
for entry in bios_zip:
key = _zip_name_to_set(entry['name'])
existing_by_name[key] = entry
updated_bios: list[dict] = []
matched_names: set[str] = set()
for set_name, set_data in hashes.get('bios_sets', {}).items():
contents = _build_contents(set_data.get('roms', []))
source_ref = _build_source_ref(set_data)
if set_name in existing_by_name:
# Update existing entry: preserve manual fields, update contents
entry = existing_by_name[set_name].copy()
entry['contents'] = contents
if source_ref:
entry['source_ref'] = source_ref
updated_bios.append(entry)
matched_names.add(set_name)
elif add_new:
# New BIOS set — only added to the main profile
entry = {
'name': f'{set_name}.zip',
'required': True,
'category': 'bios_zip',
'system': None,
'source_ref': source_ref,
'contents': contents,
}
updated_bios.append(entry)
# Entries not matched by the scraper stay untouched
# (computer ROMs, device ROMs, etc. — outside BIOS root set scope)
for set_name, entry in existing_by_name.items():
if set_name not in matched_names:
updated_bios.append(entry)
profile['files'] = non_bios + updated_bios
if write:
_backup_and_write(profile_path, profile)
return profile
def merge_fbneo_profile(
profile_path: str,
hashes_path: str,
write: bool = False,
add_new: bool = True,
) -> dict[str, Any]:
"""Merge FBNeo individual ROM entries from upstream hash data.
Preserves system, required per entry. Updates crc32, size, and
source_ref. New ROMs are only added when add_new=True (main profile).
Entries not in the hash data are left untouched.
If write=True, backs up existing profile to .old.yml before writing.
"""
profile = _load_yaml(profile_path)
hashes = _load_json(hashes_path)
profile['core_version'] = hashes.get('version', profile.get('core_version'))
files = profile.get('files', [])
archive_files, non_archive = _split_files(files, lambda f: 'archive' in f)
existing_by_key: dict[tuple[str, str], dict] = {}
for entry in archive_files:
key = (entry['archive'], entry['name'])
existing_by_key[key] = entry
merged: list[dict] = []
matched_keys: set[tuple[str, str]] = set()
for set_name, set_data in hashes.get('bios_sets', {}).items():
archive_name = f'{set_name}.zip'
source_ref = _build_source_ref(set_data)
for rom in set_data.get('roms', []):
rom_name = rom['name']
key = (archive_name, rom_name)
if key in existing_by_key:
entry = existing_by_key[key].copy()
entry['size'] = rom['size']
entry['crc32'] = rom['crc32']
if rom.get('sha1'):
entry['sha1'] = rom['sha1']
if source_ref:
entry['source_ref'] = source_ref
merged.append(entry)
matched_keys.add(key)
elif add_new:
entry = {
'name': rom_name,
'archive': archive_name,
'required': True,
'size': rom['size'],
'crc32': rom['crc32'],
}
if rom.get('sha1'):
entry['sha1'] = rom['sha1']
if source_ref:
entry['source_ref'] = source_ref
merged.append(entry)
# Entries not matched stay untouched
for key, entry in existing_by_key.items():
if key not in matched_keys:
merged.append(entry)
profile['files'] = non_archive + merged
if write:
_backup_and_write_fbneo(profile_path, profile, hashes)
return profile
def compute_diff(
profile_path: str,
hashes_path: str,
mode: str = 'mame',
) -> dict[str, Any]:
"""Compute diff between profile and hashes without writing.
Returns counts of added, updated, removed, and unchanged entries.
"""
profile = _load_yaml(profile_path)
hashes = _load_json(hashes_path)
if mode == 'mame':
return _diff_mame(profile, hashes)
return _diff_fbneo(profile, hashes)
def _diff_mame(
profile: dict[str, Any],
hashes: dict[str, Any],
) -> dict[str, Any]:
files = profile.get('files', [])
bios_zip, _ = _split_files(files, lambda f: f.get('category') == 'bios_zip')
existing_by_name: dict[str, dict] = {}
for entry in bios_zip:
existing_by_name[_zip_name_to_set(entry['name'])] = entry
added: list[str] = []
updated: list[str] = []
unchanged = 0
bios_sets = hashes.get('bios_sets', {})
for set_name, set_data in bios_sets.items():
if set_name not in existing_by_name:
added.append(set_name)
continue
old_entry = existing_by_name[set_name]
new_contents = _build_contents(set_data.get('roms', []))
old_contents = old_entry.get('contents', [])
if _contents_differ(old_contents, new_contents):
updated.append(set_name)
else:
unchanged += 1
# Items in profile but not in scraper output = out of scope (not removed)
out_of_scope = len(existing_by_name) - sum(
1 for s in existing_by_name if s in bios_sets
)
return {
'added': added,
'updated': updated,
'removed': [],
'unchanged': unchanged,
'out_of_scope': out_of_scope,
}
def _diff_fbneo(
profile: dict[str, Any],
hashes: dict[str, Any],
) -> dict[str, Any]:
files = profile.get('files', [])
archive_files, _ = _split_files(files, lambda f: 'archive' in f)
existing_by_key: dict[tuple[str, str], dict] = {}
for entry in archive_files:
existing_by_key[(entry['archive'], entry['name'])] = entry
added: list[str] = []
updated: list[str] = []
unchanged = 0
seen_keys: set[tuple[str, str]] = set()
bios_sets = hashes.get('bios_sets', {})
for set_name, set_data in bios_sets.items():
archive_name = f'{set_name}.zip'
for rom in set_data.get('roms', []):
key = (archive_name, rom['name'])
seen_keys.add(key)
label = f"{archive_name}:{rom['name']}"
if key not in existing_by_key:
added.append(label)
continue
old = existing_by_key[key]
if old.get('crc32') != rom.get('crc32') or old.get('size') != rom.get('size'):
updated.append(label)
else:
unchanged += 1
out_of_scope = sum(1 for k in existing_by_key if k not in seen_keys)
return {
'added': added,
'updated': updated,
'removed': [],
'unchanged': unchanged,
'out_of_scope': out_of_scope,
}
# ── Helpers ──────────────────────────────────────────────────────────
def _load_yaml(path: str) -> dict[str, Any]:
with open(path, encoding='utf-8') as f:
return yaml.safe_load(f) or {}
def _load_json(path: str) -> dict[str, Any]:
with open(path, encoding='utf-8') as f:
return json.load(f)
def _split_files(
files: list[dict],
predicate: Any,
) -> tuple[list[dict], list[dict]]:
matching: list[dict] = []
rest: list[dict] = []
for f in files:
if predicate(f):
matching.append(f)
else:
rest.append(f)
return matching, rest
def _zip_name_to_set(name: str) -> str:
if name.endswith('.zip'):
return name[:-4]
return name
def _build_contents(roms: list[dict]) -> list[dict]:
contents: list[dict] = []
for rom in roms:
entry: dict[str, Any] = {
'name': rom['name'],
'size': rom['size'],
'crc32': rom['crc32'],
}
if rom.get('sha1'):
entry['sha1'] = rom['sha1']
desc = rom.get('bios_description') or rom.get('bios_label') or ''
if desc:
entry['description'] = desc
if rom.get('bad_dump'):
entry['bad_dump'] = True
contents.append(entry)
return contents
def _build_source_ref(set_data: dict) -> str:
source_file = set_data.get('source_file', '')
source_line = set_data.get('source_line')
if source_file and source_line is not None:
return f'{source_file}:{source_line}'
return source_file
def _contents_differ(old: list[dict], new: list[dict]) -> bool:
if len(old) != len(new):
return True
old_by_name = {c['name']: c for c in old}
for entry in new:
prev = old_by_name.get(entry['name'])
if prev is None:
return True
if prev.get('crc32') != entry.get('crc32'):
return True
if prev.get('size') != entry.get('size'):
return True
if prev.get('sha1') != entry.get('sha1'):
return True
return False
def _backup_and_write(path: str, data: dict) -> None:
"""Write merged profile using text-based patching to preserve formatting.
Instead of yaml.dump (which destroys comments, quoting, indentation),
this reads the original file as text, patches specific fields
(core_version, contents, source_ref), and appends new entries.
"""
p = Path(path)
backup = p.with_suffix('.old.yml')
shutil.copy2(p, backup)
original = p.read_text(encoding='utf-8')
patched = _patch_core_version(original, data.get('core_version', ''))
patched = _patch_bios_entries(patched, data.get('files', []))
patched = _append_new_entries(patched, data.get('files', []), original)
p.write_text(patched, encoding='utf-8')
def _patch_core_version(text: str, version: str) -> str:
"""Replace core_version value in-place."""
if not version:
return text
import re
return re.sub(
r'^(core_version:\s*).*$',
rf'\g<1>"{version}"',
text,
count=1,
flags=re.MULTILINE,
)
def _patch_bios_entries(text: str, files: list[dict]) -> str:
"""Patch contents and source_ref for existing bios_zip entries in-place.
Processes entries in reverse order to preserve line offsets.
Each entry's "owned" lines are: the `- name:` line plus all indented
lines that follow (4+ spaces), stopping at blank lines, comments,
or the next `- name:`.
"""
import re
# Build a lookup of what to patch
patches: dict[str, dict] = {}
for fe in files:
if fe.get('category') != 'bios_zip':
continue
patches[fe['name']] = fe
if not patches:
return text
lines = text.split('\n')
# Find all entry start positions (line indices)
entry_starts: list[tuple[int, str]] = []
for i, line in enumerate(lines):
m = re.match(r'^ - name:\s*(.+?)\s*$', line)
if m:
entry_starts.append((i, m.group(1).strip('"').strip("'")))
# Process in reverse so line insertions don't shift indices
for idx in range(len(entry_starts) - 1, -1, -1):
start_line, entry_name = entry_starts[idx]
if entry_name not in patches:
continue
fe = patches[entry_name]
contents = fe.get('contents', [])
source_ref = fe.get('source_ref', '')
# Find the last "owned" line of this entry
# Owned = indented with 4+ spaces (field lines of this entry)
last_owned = start_line
for j in range(start_line + 1, len(lines)):
stripped = lines[j].strip()
if not stripped:
break # blank line = end of entry
if stripped.startswith('#'):
break # comment = belongs to next entry
if re.match(r'^ - ', lines[j]):
break # next list item
if re.match(r'^ ', lines[j]) or re.match(r'^ \w', lines[j]):
last_owned = j
else:
break
# Patch source_ref in-place
if source_ref:
found_sr = False
for j in range(start_line + 1, last_owned + 1):
if re.match(r'^ source_ref:', lines[j]):
lines[j] = f' source_ref: "{source_ref}"'
found_sr = True
break
if not found_sr:
lines.insert(last_owned + 1, f' source_ref: "{source_ref}"')
last_owned += 1
# Remove existing contents block if present
contents_start = None
contents_end = None
for j in range(start_line + 1, last_owned + 1):
if re.match(r'^ contents:', lines[j]):
contents_start = j
elif contents_start is not None:
if re.match(r'^ ', lines[j]):
contents_end = j
else:
break
if contents_end is None and contents_start is not None:
contents_end = contents_start
if contents_start is not None:
del lines[contents_start:contents_end + 1]
last_owned -= (contents_end - contents_start + 1)
# Insert new contents after last owned line
if contents:
new_lines = _format_contents(contents).split('\n')
for k, cl in enumerate(new_lines):
lines.insert(last_owned + 1 + k, cl)
return '\n'.join(lines)
def _append_new_entries(text: str, files: list[dict], original: str) -> str:
"""Append new bios_zip entries (system=None) that aren't in the original."""
# Parse original to get existing entry names (more reliable than text search)
existing_data = yaml.safe_load(original) or {}
existing_names = {f['name'] for f in existing_data.get('files', [])}
new_entries = []
for fe in files:
if fe.get('category') != 'bios_zip' or fe.get('system') is not None:
continue
if fe['name'] in existing_names:
continue
new_entries.append(fe)
if not new_entries:
return text
lines = []
for fe in new_entries:
lines.append(f'\n - name: {fe["name"]}')
lines.append(f' required: {str(fe["required"]).lower()}')
lines.append(f' category: bios_zip')
if fe.get('source_ref'):
lines.append(f' source_ref: "{fe["source_ref"]}"')
if fe.get('contents'):
lines.append(_format_contents(fe['contents']))
if lines:
text = text.rstrip('\n') + '\n' + '\n'.join(lines) + '\n'
return text
def _format_contents(contents: list[dict]) -> str:
"""Format a contents list as YAML text."""
lines = [' contents:']
for rom in contents:
lines.append(f' - name: {rom["name"]}')
if rom.get('description'):
lines.append(f' description: {rom["description"]}')
if rom.get('size'):
lines.append(f' size: {rom["size"]}')
if rom.get('crc32'):
lines.append(f' crc32: "{rom["crc32"]}"')
if rom.get('sha1'):
lines.append(f' sha1: "{rom["sha1"]}"')
if rom.get('bad_dump'):
lines.append(f' bad_dump: true')
return '\n'.join(lines)
def _backup_and_write_fbneo(path: str, data: dict, hashes: dict) -> None:
"""Write merged FBNeo profile using text-based patching.
FBNeo profiles have individual ROM entries with archive: field.
Only patches core_version and appends new ROM entries.
Existing entries are left untouched (CRC32 changes are rare).
"""
p = Path(path)
backup = p.with_suffix('.old.yml')
shutil.copy2(p, backup)
original = p.read_text(encoding='utf-8')
patched = _patch_core_version(original, data.get('core_version', ''))
# Identify new ROM entries by comparing parsed data keys, not text search
existing_data = yaml.safe_load(original) or {}
existing_keys = {
(f['archive'], f['name'])
for f in existing_data.get('files', [])
if f.get('archive')
}
new_roms = [
f for f in data.get('files', [])
if f.get('archive') and (f['archive'], f['name']) not in existing_keys
]
if new_roms:
lines = []
for fe in new_roms:
lines.append(f' - name: "{fe["name"]}"')
lines.append(f' archive: {fe["archive"]}')
lines.append(f' required: {str(fe.get("required", True)).lower()}')
if fe.get('size'):
lines.append(f' size: {fe["size"]}')
if fe.get('crc32'):
lines.append(f' crc32: "{fe["crc32"]}"')
if fe.get('source_ref'):
lines.append(f' source_ref: "{fe["source_ref"]}"')
lines.append('')
patched = patched.rstrip('\n') + '\n\n' + '\n'.join(lines)
p.write_text(patched, encoding='utf-8')

View File

@@ -8,6 +8,7 @@ import urllib.request
import urllib.error
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from pathlib import Path
@dataclass
@@ -231,7 +232,28 @@ def scraper_cli(scraper_class: type, description: str = "Scrape BIOS requirement
if req.zipped_file:
entry["zipped_file"] = req.zipped_file
config["systems"][sys_id]["files"].append(entry)
with open(args.output, "w") as f:
# Merge into existing YAML: preserve fields the scraper doesn't generate
# (data_directories, case_insensitive_fs, manually added metadata).
# The scraper replaces systems + files; everything else is preserved.
output_path = Path(args.output)
if output_path.exists():
with open(output_path) as f:
existing = yaml.safe_load(f) or {}
# Preserve existing keys not generated by the scraper.
# Only keys present in the NEW config are considered scraper-generated.
# Everything else in the existing file is preserved.
for key, val in existing.items():
if key not in config:
config[key] = val
# Preserve per-system fields not generated by the scraper
# (data_directories, native_id from manual additions, etc.)
existing_systems = existing.get("systems", {})
for sys_id, sys_data in config.get("systems", {}).items():
old_sys = existing_systems.get(sys_id, {})
for field in ("data_directories",):
if field in old_sys and field not in sys_data:
sys_data[field] = old_sys[field]
with open(output_path, "w") as f:
yaml.dump(config, f, default_flow_style=False, sort_keys=False)
print(f"Written {len(reqs)} entries to {args.output}")
return

View File

@@ -92,6 +92,27 @@ SYSTEM_SLUG_MAP = {
"dos": "dos",
"videopac": "philips-videopac",
"pokemini": "nintendo-pokemon-mini",
"gsplus": "apple-iigs",
"apple2": "apple-ii",
"apple2gs": "apple-iigs",
"ps3": "sony-playstation-3",
"psvita": "sony-playstation-vita",
"coco": "coco",
"dragon32": "dragon32",
"dragon64": "dragon64",
"mc10": "mc10",
"msx2+": "microsoft-msx",
"msxturbor": "microsoft-msx",
"spectravideo": "spectravideo",
"tvc": "videoton-tvc",
"enterprise": "enterprise-64-128",
"vis": "tandy-vis",
"supracan": "supracan",
"jaguar": "atari-jaguar",
"jaguarcd": "atari-jaguar",
"switch": "nintendo-switch",
"wii": "nintendo-wii",
"xbox360": "microsoft-xbox-360",
}
@@ -282,12 +303,25 @@ class Scraper(BaseScraper):
"""Generate a platform YAML config dict from scraped data."""
requirements = self.fetch_requirements()
# Parse source to extract display names per system
raw = self._fetch_raw()
source_dict = self._extract_systems_dict(raw)
display_names: dict[str, str] = {}
for sys_key, sys_data in source_dict.items():
dname = sys_data.get("name", "")
if dname:
slug = SYSTEM_SLUG_MAP.get(sys_key, sys_key)
display_names[slug] = dname
systems = {}
for req in requirements:
if req.system not in systems:
sys_entry: dict = {"files": []}
if req.native_id:
sys_entry["native_id"] = req.native_id
dname = display_names.get(req.system)
if dname:
sys_entry["name"] = dname
systems[req.system] = sys_entry
entry = {
@@ -308,6 +342,13 @@ class Scraper(BaseScraper):
num = tag.removeprefix("batocera-")
if num.isdigit():
batocera_version = num
if not batocera_version:
# Preserve existing version when fetch fails (offline mode)
existing = Path(__file__).resolve().parents[2] / "platforms" / "batocera.yml"
if existing.exists():
with open(existing) as f:
old = yaml.safe_load(f) or {}
batocera_version = str(old.get("version", ""))
cores, standalone = self._fetch_cores()
result = {

View File

@@ -0,0 +1,315 @@
"""Scrape FBNeo BIOS set hashes from upstream source via sparse clone.
Does NOT inherit BaseScraper (uses git sparse clone, not URL fetch).
Parses BDF_BOARDROM drivers from src/burn/drv/ to extract CRC32/size
for all BIOS ROM sets, then optionally merges into emulator profiles.
"""
from __future__ import annotations
import argparse
import json
import logging
import shutil
import subprocess
import sys
from datetime import datetime, timezone, timedelta
from pathlib import Path
from typing import Any
import yaml
from scripts.scraper.fbneo_parser import parse_fbneo_source_tree
from scripts.scraper._hash_merge import compute_diff, merge_fbneo_profile
log = logging.getLogger(__name__)
REPO_URL = 'https://github.com/finalburnneo/FBNeo.git'
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
CLONE_DIR = REPO_ROOT / 'tmp' / 'fbneo'
CACHE_PATH = REPO_ROOT / 'data' / 'fbneo-hashes.json'
EMULATORS_DIR = REPO_ROOT / 'emulators'
STALE_HOURS = 24
def _is_cache_fresh() -> bool:
"""Check if the JSON cache exists and is less than 24 hours old."""
if not CACHE_PATH.exists():
return False
try:
data = json.loads(CACHE_PATH.read_text(encoding='utf-8'))
fetched_at = datetime.fromisoformat(data['fetched_at'])
return datetime.now(timezone.utc) - fetched_at < timedelta(hours=STALE_HOURS)
except (json.JSONDecodeError, KeyError, ValueError):
return False
def _sparse_clone() -> None:
"""Sparse clone FBNeo repo, checking out only src/burn/drv."""
if CLONE_DIR.exists():
shutil.rmtree(CLONE_DIR)
CLONE_DIR.parent.mkdir(parents=True, exist_ok=True)
subprocess.run(
[
'git', 'clone', '--depth', '1', '--filter=blob:none',
'--sparse', REPO_URL, str(CLONE_DIR),
],
check=True,
capture_output=True,
text=True,
)
subprocess.run(
['git', 'sparse-checkout', 'set', 'src/burn/drv', 'src/burner/resource.h'],
cwd=CLONE_DIR,
check=True,
capture_output=True,
text=True,
)
def _extract_version() -> tuple[str, str]:
"""Extract version tag and commit SHA from the cloned repo.
Returns (version, commit_sha). Falls back to resource.h if no tag.
"""
result = subprocess.run(
['git', 'describe', '--tags', '--abbrev=0'],
cwd=CLONE_DIR,
capture_output=True,
text=True,
)
# Prefer real version tags over pseudo-tags like "latest"
version = 'unknown'
if result.returncode == 0:
tag = result.stdout.strip()
if tag and tag != 'latest':
version = tag
# Fallback: resource.h
if version == 'unknown':
version = _version_from_resource_h()
# Last resort: use GitHub API for latest real release tag
if version == 'unknown':
try:
import urllib.request
import urllib.error
req = urllib.request.Request(
'https://api.github.com/repos/finalburnneo/FBNeo/tags?per_page=10',
headers={'User-Agent': 'retrobios-scraper/1.0'},
)
with urllib.request.urlopen(req, timeout=10) as resp:
import json as json_mod
tags = json_mod.loads(resp.read())
for t in tags:
if t['name'] != 'latest' and t['name'].startswith('v'):
version = t['name']
break
except (urllib.error.URLError, OSError):
pass
sha_result = subprocess.run(
['git', 'rev-parse', 'HEAD'],
cwd=CLONE_DIR,
capture_output=True,
text=True,
check=True,
)
commit = sha_result.stdout.strip()
return version, commit
def _version_from_resource_h() -> str:
"""Fallback: parse VER_FULL_VERSION_STR from resource.h."""
resource_h = CLONE_DIR / 'src' / 'burner' / 'resource.h'
if not resource_h.exists():
return 'unknown'
text = resource_h.read_text(encoding='utf-8', errors='replace')
for line in text.splitlines():
if 'VER_FULL_VERSION_STR' in line:
parts = line.split('"')
if len(parts) >= 2:
return parts[1]
return 'unknown'
def _cleanup() -> None:
"""Remove the sparse clone directory."""
if CLONE_DIR.exists():
shutil.rmtree(CLONE_DIR)
def fetch_and_cache(force: bool = False) -> dict[str, Any]:
"""Clone, parse, and write JSON cache. Returns the cache dict."""
if not force and _is_cache_fresh():
log.info('cache fresh, skipping clone (use --force to override)')
return json.loads(CACHE_PATH.read_text(encoding='utf-8'))
try:
log.info('sparse cloning %s', REPO_URL)
_sparse_clone()
log.info('extracting version')
version, commit = _extract_version()
log.info('parsing source tree')
bios_sets = parse_fbneo_source_tree(str(CLONE_DIR))
cache: dict[str, Any] = {
'source': 'finalburnneo/FBNeo',
'version': version,
'commit': commit,
'fetched_at': datetime.now(timezone.utc).isoformat(),
'bios_sets': bios_sets,
}
CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
CACHE_PATH.write_text(
json.dumps(cache, indent=2, ensure_ascii=False) + '\n',
encoding='utf-8',
)
log.info('wrote %d BIOS sets to %s', len(bios_sets), CACHE_PATH)
return cache
finally:
_cleanup()
def _find_fbneo_profiles() -> list[Path]:
"""Find emulator profiles whose upstream references finalburnneo/FBNeo."""
profiles: list[Path] = []
for path in sorted(EMULATORS_DIR.glob('*.yml')):
if path.name.endswith('.old.yml'):
continue
try:
data = yaml.safe_load(path.read_text(encoding='utf-8'))
except (yaml.YAMLError, OSError):
continue
if not data or not isinstance(data, dict):
continue
upstream = data.get('upstream', '')
if isinstance(upstream, str) and 'finalburnneo/fbneo' in upstream.lower():
profiles.append(path)
return profiles
def _format_diff(profile_name: str, diff: dict[str, Any], show_added: bool = True) -> str:
"""Format diff for a single profile."""
lines: list[str] = []
lines.append(f' {profile_name}:')
added = diff.get('added', [])
updated = diff.get('updated', [])
oos = diff.get('out_of_scope', 0)
if not added and not updated:
lines.append(' no changes')
if oos:
lines.append(f' . {oos} out of scope')
return '\n'.join(lines)
if show_added:
for label in added:
lines.append(f' + {label}')
elif added:
lines.append(f' + {len(added)} new ROMs available (main profile only)')
for label in updated:
lines.append(f' ~ {label}')
lines.append(f' = {diff["unchanged"]} unchanged')
if oos:
lines.append(f' . {oos} out of scope')
return '\n'.join(lines)
def run(
dry_run: bool = False,
force: bool = False,
json_output: bool = False,
) -> int:
"""Main entry point for the scraper."""
cache = fetch_and_cache(force=force)
version = cache.get('version', 'unknown')
commit = cache.get('commit', '?')[:12]
bios_sets = cache.get('bios_sets', {})
profiles = _find_fbneo_profiles()
if json_output:
result: dict[str, Any] = {
'source': cache.get('source'),
'version': version,
'commit': cache.get('commit'),
'bios_set_count': len(bios_sets),
'profiles': {},
}
for path in profiles:
diff = compute_diff(str(path), str(CACHE_PATH), mode='fbneo')
result['profiles'][path.stem] = diff
print(json.dumps(result, indent=2))
return 0
header = (
f'fbneo-hashes: {len(bios_sets)} BIOS sets '
f'from finalburnneo/FBNeo @ {version} ({commit})'
)
print(header)
print()
if not profiles:
print(' no matching emulator profiles found')
return 0
for path in profiles:
is_main = path.name == 'fbneo.yml'
diff = compute_diff(str(path), str(CACHE_PATH), mode='fbneo')
print(_format_diff(path.stem, diff, show_added=is_main))
effective_added = diff['added'] if is_main else []
if not dry_run and (effective_added or diff['updated']):
merge_fbneo_profile(str(path), str(CACHE_PATH), write=True, add_new=is_main)
log.info('merged changes into %s', path.name)
return 0
def main() -> None:
parser = argparse.ArgumentParser(
description='Scrape FBNeo BIOS set hashes from upstream source',
)
parser.add_argument(
'--dry-run',
action='store_true',
help='show diff without writing changes',
)
parser.add_argument(
'--force',
action='store_true',
help='force re-clone even if cache is fresh',
)
parser.add_argument(
'--json',
action='store_true',
dest='json_output',
help='output diff as JSON',
)
args = parser.parse_args()
logging.basicConfig(
level=logging.INFO,
format='%(name)s: %(message)s',
)
sys.exit(run(
dry_run=args.dry_run,
force=args.force,
json_output=args.json_output,
))
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,127 @@
"""Parser for FBNeo source files to extract BIOS sets and ROM definitions.
Parses BurnRomInfo structs (static ROM arrays) and BurnDriver structs
(driver registration) from FBNeo C source files. BIOS sets are identified
by the BDF_BOARDROM flag in BurnDriver definitions.
"""
from __future__ import annotations
import os
import re
from pathlib import Path
_ROM_ENTRY_RE = re.compile(
r'\{\s*"([^"]+)"\s*,\s*(0x[\da-fA-F]+)\s*,\s*(0x[\da-fA-F]+)\s*,\s*([^}]+)\}',
)
_BURN_DRIVER_RE = re.compile(
r'struct\s+BurnDriver\s+BurnDrv(\w+)\s*=\s*\{(.*?)\};',
re.DOTALL,
)
_ROM_DESC_RE = re.compile(
r'static\s+struct\s+BurnRomInfo\s+(\w+)RomDesc\s*\[\s*\]\s*=\s*\{(.*?)\};',
re.DOTALL,
)
def find_bios_sets(source: str, filename: str) -> dict[str, dict]:
"""Find BDF_BOARDROM drivers in source code.
Returns a dict mapping set name to metadata:
{set_name: {"source_file": str, "source_line": int}}
"""
results: dict[str, dict] = {}
for match in _BURN_DRIVER_RE.finditer(source):
body = match.group(2)
if 'BDF_BOARDROM' not in body:
continue
# Set name is the first quoted string in the struct body
name_match = re.search(r'"([^"]+)"', body)
if not name_match:
continue
set_name = name_match.group(1)
line_num = source[:match.start()].count('\n') + 1
results[set_name] = {
'source_file': filename,
'source_line': line_num,
}
return results
def parse_rom_info(source: str, set_name: str) -> list[dict]:
"""Parse a BurnRomInfo array for the given set name.
Returns a list of dicts with keys: name, size, crc32.
Sentinel entries (empty name) are skipped.
"""
pattern = re.compile(
r'static\s+struct\s+BurnRomInfo\s+'
+ re.escape(set_name)
+ r'RomDesc\s*\[\s*\]\s*=\s*\{(.*?)\};',
re.DOTALL,
)
match = pattern.search(source)
if not match:
return []
body = match.group(1)
roms: list[dict] = []
for entry in _ROM_ENTRY_RE.finditer(body):
name = entry.group(1)
if not name:
continue
size = int(entry.group(2), 16)
crc32 = format(int(entry.group(3), 16), '08x')
roms.append({
'name': name,
'size': size,
'crc32': crc32,
})
return roms
def parse_fbneo_source_tree(base_path: str) -> dict[str, dict]:
"""Walk the FBNeo driver source tree and extract all BIOS sets.
Scans .cpp files under src/burn/drv/ for BDF_BOARDROM drivers,
then parses their associated BurnRomInfo arrays.
Returns a dict mapping set name to:
{source_file, source_line, roms: [{name, size, crc32}, ...]}
"""
drv_path = Path(base_path) / 'src' / 'burn' / 'drv'
if not drv_path.is_dir():
return {}
results: dict[str, dict] = {}
for root, _dirs, files in os.walk(drv_path):
for fname in files:
if not fname.endswith('.cpp'):
continue
filepath = Path(root) / fname
source = filepath.read_text(encoding='utf-8', errors='replace')
rel_path = str(filepath.relative_to(base_path))
bios_sets = find_bios_sets(source, rel_path)
for set_name, meta in bios_sets.items():
roms = parse_rom_info(source, set_name)
results[set_name] = {
'source_file': meta['source_file'],
'source_line': meta['source_line'],
'roms': roms,
}
return results

View File

@@ -0,0 +1,322 @@
"""Fetch MAME BIOS hashes from mamedev/mame source and merge into profiles.
Sparse clones the MAME repo, parses the source tree for BIOS root sets,
caches results to data/mame-hashes.json, and optionally merges into
emulator profiles that reference mamedev/mame upstream.
"""
from __future__ import annotations
import argparse
import json
import logging
import shutil
import subprocess
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import yaml
from .mame_parser import parse_mame_source_tree
from ._hash_merge import compute_diff, merge_mame_profile
log = logging.getLogger(__name__)
_ROOT = Path(__file__).resolve().parents[2]
_CACHE_PATH = _ROOT / 'data' / 'mame-hashes.json'
_CLONE_DIR = _ROOT / 'tmp' / 'mame'
_EMULATORS_DIR = _ROOT / 'emulators'
_REPO_URL = 'https://github.com/mamedev/mame.git'
_STALE_HOURS = 24
# ── Cache ────────────────────────────────────────────────────────────
def _load_cache() -> dict[str, Any] | None:
if not _CACHE_PATH.exists():
return None
try:
with open(_CACHE_PATH, encoding='utf-8') as f:
return json.load(f)
except (json.JSONDecodeError, OSError):
return None
def _is_stale(cache: dict[str, Any] | None) -> bool:
if cache is None:
return True
fetched_at = cache.get('fetched_at')
if not fetched_at:
return True
try:
ts = datetime.fromisoformat(fetched_at)
age = datetime.now(timezone.utc) - ts
return age.total_seconds() > _STALE_HOURS * 3600
except (ValueError, TypeError):
return True
def _write_cache(data: dict[str, Any]) -> None:
_CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
with open(_CACHE_PATH, 'w', encoding='utf-8') as f:
json.dump(data, f, indent=2, ensure_ascii=False)
log.info('cache written to %s', _CACHE_PATH)
# ── Git operations ───────────────────────────────────────────────────
def _run_git(args: list[str], cwd: Path | None = None) -> subprocess.CompletedProcess[str]:
return subprocess.run(
['git', *args],
cwd=cwd,
check=True,
capture_output=True,
text=True,
)
def _sparse_clone() -> None:
if _CLONE_DIR.exists():
shutil.rmtree(_CLONE_DIR)
_CLONE_DIR.parent.mkdir(parents=True, exist_ok=True)
log.info('sparse cloning mamedev/mame into %s', _CLONE_DIR)
_run_git([
'clone',
'--depth', '1',
'--filter=blob:none',
'--sparse',
_REPO_URL,
str(_CLONE_DIR),
])
_run_git(
['sparse-checkout', 'set', 'src/mame', 'src/devices'],
cwd=_CLONE_DIR,
)
def _get_version() -> str:
# version.cpp is generated at build time, not in the repo.
# Use GitHub API to get the latest release tag.
try:
req = urllib.request.Request(
'https://api.github.com/repos/mamedev/mame/releases/latest',
headers={'User-Agent': 'retrobios-scraper/1.0',
'Accept': 'application/vnd.github.v3+json'},
)
with urllib.request.urlopen(req, timeout=10) as resp:
data = json.loads(resp.read())
tag = data.get('tag_name', '')
if tag:
return _parse_version_tag(tag)
except (urllib.error.URLError, json.JSONDecodeError, OSError):
pass
return 'unknown'
def _parse_version_tag(tag: str) -> str:
prefix = 'mame'
raw = tag.removeprefix(prefix) if tag.startswith(prefix) else tag
if raw.isdigit() and len(raw) >= 4:
return f'{raw[0]}.{raw[1:]}'
return raw
def _get_commit() -> str:
try:
result = _run_git(['rev-parse', 'HEAD'], cwd=_CLONE_DIR)
return result.stdout.strip()
except subprocess.CalledProcessError:
return ''
def _cleanup() -> None:
if _CLONE_DIR.exists():
log.info('cleaning up %s', _CLONE_DIR)
shutil.rmtree(_CLONE_DIR)
# ── Profile discovery ────────────────────────────────────────────────
def _find_mame_profiles() -> list[Path]:
profiles: list[Path] = []
for path in sorted(_EMULATORS_DIR.glob('*.yml')):
if path.name.endswith('.old.yml'):
continue
try:
with open(path, encoding='utf-8') as f:
data = yaml.safe_load(f)
if not isinstance(data, dict):
continue
upstream = data.get('upstream', '')
# Only match profiles tracking current MAME (not frozen snapshots
# which have upstream like "mamedev/mame/tree/mame0139")
if isinstance(upstream, str) and upstream.rstrip('/') == 'https://github.com/mamedev/mame':
profiles.append(path)
except (yaml.YAMLError, OSError):
continue
return profiles
# ── Diff formatting ──────────────────────────────────────────────────
def _format_diff(
profile_path: Path,
diff: dict[str, Any],
hashes: dict[str, Any],
show_added: bool = True,
) -> list[str]:
lines: list[str] = []
name = profile_path.stem
added = diff.get('added', [])
updated = diff.get('updated', [])
removed = diff.get('removed', [])
unchanged = diff.get('unchanged', 0)
if not added and not updated and not removed:
lines.append(f' {name}:')
lines.append(' no changes')
return lines
lines.append(f' {name}:')
if show_added:
bios_sets = hashes.get('bios_sets', {})
for set_name in added:
rom_count = len(bios_sets.get(set_name, {}).get('roms', []))
source_file = bios_sets.get(set_name, {}).get('source_file', '')
source_line = bios_sets.get(set_name, {}).get('source_line', '')
ref = f'{source_file}:{source_line}' if source_file else ''
lines.append(f' + {set_name}.zip ({ref}, {rom_count} ROMs)')
elif added:
lines.append(f' + {len(added)} new sets available (main profile only)')
for set_name in updated:
lines.append(f' ~ {set_name}.zip (contents changed)')
oos = diff.get('out_of_scope', 0)
lines.append(f' = {unchanged} unchanged')
if oos:
lines.append(f' . {oos} out of scope (not BIOS root sets)')
return lines
# ── Main ─────────────────────────────────────────────────────────────
def _fetch_hashes(force: bool) -> dict[str, Any]:
cache = _load_cache()
if not force and not _is_stale(cache):
log.info('using cached data from %s', cache.get('fetched_at', ''))
return cache # type: ignore[return-value]
try:
_sparse_clone()
bios_sets = parse_mame_source_tree(str(_CLONE_DIR))
version = _get_version()
commit = _get_commit()
data: dict[str, Any] = {
'source': 'mamedev/mame',
'version': version,
'commit': commit,
'fetched_at': datetime.now(timezone.utc).isoformat(),
'bios_sets': bios_sets,
}
_write_cache(data)
return data
finally:
_cleanup()
def _run(args: argparse.Namespace) -> None:
hashes = _fetch_hashes(args.force)
total_sets = len(hashes.get('bios_sets', {}))
version = hashes.get('version', 'unknown')
commit = hashes.get('commit', '')[:12]
if args.json:
json.dump(hashes, sys.stdout, indent=2, ensure_ascii=False)
sys.stdout.write('\n')
return
print(f'mame-hashes: {total_sets} BIOS root sets from mamedev/mame'
f' @ {version} ({commit})')
print()
profiles = _find_mame_profiles()
if not profiles:
print(' no profiles with mamedev/mame upstream found')
return
for profile_path in profiles:
is_main = profile_path.name == 'mame.yml'
diff = compute_diff(str(profile_path), str(_CACHE_PATH), mode='mame')
lines = _format_diff(profile_path, diff, hashes, show_added=is_main)
for line in lines:
print(line)
if not args.dry_run:
updated = diff.get('updated', [])
added = diff.get('added', []) if is_main else []
if added or updated:
merge_mame_profile(
str(profile_path),
str(_CACHE_PATH),
write=True,
add_new=is_main,
)
log.info('merged into %s', profile_path.name)
print()
if args.dry_run:
print('(dry run, no files modified)')
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog='mame_hash_scraper',
description='Fetch MAME BIOS hashes from source and merge into profiles.',
)
parser.add_argument(
'--dry-run',
action='store_true',
help='show diff only, do not modify profiles',
)
parser.add_argument(
'--json',
action='store_true',
help='output raw JSON to stdout',
)
parser.add_argument(
'--force',
action='store_true',
help='re-fetch even if cache is fresh',
)
return parser
def main() -> None:
logging.basicConfig(
level=logging.INFO,
format='%(levelname)s: %(message)s',
)
parser = build_parser()
args = parser.parse_args()
_run(args)
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,389 @@
"""Parser for MAME C source files.
Extracts BIOS root sets and ROM definitions from MAME driver sources.
Handles GAME/SYST/COMP/CONS macros with MACHINE_IS_BIOS_ROOT flag,
ROM_START/ROM_END blocks, ROM_LOAD variants, ROM_REGION, ROM_SYSTEM_BIOS,
NO_DUMP filtering, and BAD_DUMP flagging.
"""
from __future__ import annotations
import os
import re
from pathlib import Path
# Macros that declare a machine entry
_MACHINE_MACROS = re.compile(
r'\b(GAME|SYST|COMP|CONS)\s*\(',
re.MULTILINE,
)
# ROM block boundaries
_ROM_START = re.compile(r'ROM_START\s*\(\s*(\w+)\s*\)')
_ROM_END = re.compile(r'ROM_END')
# ROM_REGION variants: ROM_REGION, ROM_REGION16_BE, ROM_REGION16_LE, ROM_REGION32_LE, etc.
_ROM_REGION = re.compile(
r'ROM_REGION\w*\s*\('
r'\s*(0x[\da-fA-F]+|\d+)\s*,' # size
r'\s*"([^"]+)"\s*,', # tag
)
# ROM_SYSTEM_BIOS( index, label, description )
_ROM_SYSTEM_BIOS = re.compile(
r'ROM_SYSTEM_BIOS\s*\('
r'\s*(\d+)\s*,' # index
r'\s*"([^"]+)"\s*,' # label
r'\s*"([^"]+)"\s*\)', # description
)
# All ROM_LOAD variants including custom BIOS macros.
# Standard: ROM_LOAD("name", offset, size, hash)
# BIOS variant: ROM_LOAD_BIOS(biosidx, "name", offset, size, hash)
# ROM_LOAD16_WORD_SWAP_BIOS(biosidx, "name", offset, size, hash)
# The key pattern: any macro containing "ROM_LOAD" or "ROMX_LOAD" in its name,
# with the first quoted string being the ROM filename.
_ROM_LOAD = re.compile(
r'\b\w*ROMX?_LOAD\w*\s*\('
r'[^"]*' # skip any args before the filename (e.g., bios index)
r'"([^"]+)"\s*,' # name (first quoted string)
r'\s*(0x[\da-fA-F]+|\d+)\s*,' # offset
r'\s*(0x[\da-fA-F]+|\d+)\s*,', # size
)
# CRC32 and SHA1 within a ROM_LOAD line
_CRC_SHA = re.compile(
r'CRC\s*\(\s*([0-9a-fA-F]+)\s*\)'
r'\s+'
r'SHA1\s*\(\s*([0-9a-fA-F]+)\s*\)',
)
_NO_DUMP = re.compile(r'\bNO_DUMP\b')
_BAD_DUMP = re.compile(r'\bBAD_DUMP\b')
_ROM_BIOS = re.compile(r'ROM_BIOS\s*\(\s*(\d+)\s*\)')
def find_bios_root_sets(source: str, filename: str) -> dict[str, dict]:
"""Find machine entries flagged as BIOS root sets.
Scans for GAME/SYST/COMP/CONS macros where the args include
MACHINE_IS_BIOS_ROOT, returns set names with source location.
"""
results: dict[str, dict] = {}
for match in _MACHINE_MACROS.finditer(source):
start = match.end() - 1 # position of opening paren
block_end = _find_closing_paren(source, start)
if block_end == -1:
continue
block = source[start:block_end + 1]
if 'MACHINE_IS_BIOS_ROOT' not in block:
continue
# Extract set name: first arg after the opening paren
inner = block[1:] # skip opening paren
args = _split_macro_args(inner)
if not args:
continue
# The set name position varies by macro type
# GAME(year, setname, parent, machine, input, init, monitor, company, fullname, flags)
# CONS(year, setname, parent, compat, machine, input, init, company, fullname, flags)
# COMP(year, setname, parent, compat, machine, input, init, company, fullname, flags)
# SYST(year, setname, parent, compat, machine, input, init, company, fullname, flags)
# In all cases, setname is the second arg (index 1)
if len(args) < 2:
continue
set_name = args[1].strip()
line_no = source[:match.start()].count('\n') + 1
results[set_name] = {
'source_file': filename,
'source_line': line_no,
}
return results
def parse_rom_block(source: str, set_name: str) -> list[dict]:
"""Parse ROM definitions for a given set name.
Finds the ROM_START(set_name)...ROM_END block, expands local
#define macros that contain ROM_LOAD/ROM_REGION calls, then
extracts all ROM entries. Skips NO_DUMP, flags BAD_DUMP.
"""
pattern = re.compile(
r'ROM_START\s*\(\s*' + re.escape(set_name) + r'\s*\)',
)
start_match = pattern.search(source)
if not start_match:
return []
end_match = _ROM_END.search(source, start_match.end())
if not end_match:
return []
block = source[start_match.end():end_match.start()]
# Pre-expand macros: find #define macros in the file that contain
# ROM_LOAD/ROM_REGION/ROM_SYSTEM_BIOS calls, then expand their
# invocations within the ROM block.
macros = _collect_rom_macros(source)
block = _expand_macros(block, macros, depth=5)
return _parse_rom_entries(block)
def parse_mame_source_tree(base_path: str) -> dict[str, dict]:
"""Walk MAME source tree and extract all BIOS root sets with ROMs.
Scans src/mame/ and src/devices/ for C/C++ source files.
"""
results: dict[str, dict] = {}
root = Path(base_path)
search_dirs = [root / 'src' / 'mame', root / 'src' / 'devices']
for search_dir in search_dirs:
if not search_dir.is_dir():
continue
for dirpath, _dirnames, filenames in os.walk(search_dir):
for fname in filenames:
if not fname.endswith(('.cpp', '.c', '.h', '.hxx')):
continue
filepath = Path(dirpath) / fname
rel_path = str(filepath.relative_to(root))
content = filepath.read_text(encoding='utf-8', errors='replace')
bios_sets = find_bios_root_sets(content, rel_path)
for set_name, info in bios_sets.items():
roms = parse_rom_block(content, set_name)
results[set_name] = {
'source_file': info['source_file'],
'source_line': info['source_line'],
'roms': roms,
}
return results
# Regex for #define macros that span multiple lines (backslash continuation)
_DEFINE_RE = re.compile(
r'^\s*#\s*define\s+(\w+)(?:\([^)]*\))?\s*((?:.*\\\n)*.*)',
re.MULTILINE,
)
# ROM-related tokens that indicate a macro is relevant for expansion
_ROM_TOKENS = {'ROM_LOAD', 'ROMX_LOAD', 'ROM_REGION', 'ROM_SYSTEM_BIOS',
'ROM_FILL', 'ROM_COPY', 'ROM_RELOAD'}
def _collect_rom_macros(source: str) -> dict[str, str]:
"""Collect #define macros that contain ROM-related calls.
Returns {macro_name: expanded_body} with backslash continuations joined.
Only collects macros that contain actual ROM data (quoted filenames),
not wrapper macros like ROM_LOAD16_WORD_SWAP_BIOS that just redirect
to ROMX_LOAD with formal parameters.
"""
macros: dict[str, str] = {}
for m in _DEFINE_RE.finditer(source):
name = m.group(1)
body = m.group(2)
# Join backslash-continued lines
body = body.replace('\\\n', ' ')
# Only keep macros that contain ROM-related tokens
if not any(tok in body for tok in _ROM_TOKENS):
continue
# Skip wrapper macros: if the body contains ROMX_LOAD/ROM_LOAD
# with unquoted args (formal parameters), it's a wrapper.
# These are already recognized by the _ROM_LOAD regex directly.
if re.search(r'ROMX?_LOAD\s*\(\s*\w+\s*,\s*\w+\s*,', body):
continue
macros[name] = body
return macros
def _expand_macros(block: str, macros: dict[str, str], depth: int = 5) -> str:
"""Expand macro invocations in a ROM block.
Handles both simple macros (NEOGEO_BIOS) and parameterized ones
(NEOGEO_UNIBIOS_2_2_AND_NEWER(16)). Recurses up to `depth` levels
for nested macros.
"""
if depth <= 0 or not macros:
return block
changed = True
iterations = 0
while changed and iterations < depth:
changed = False
iterations += 1
for name, body in macros.items():
# Match macro invocation: NAME or NAME(args)
pattern = re.compile(r'\b' + re.escape(name) + r'(?:\s*\([^)]*\))?')
if pattern.search(block):
block = pattern.sub(body, block)
changed = True
return block
def _find_closing_paren(source: str, start: int) -> int:
"""Find the matching closing paren for source[start] which must be '('."""
depth = 0
i = start
while i < len(source):
ch = source[i]
if ch == '(':
depth += 1
elif ch == ')':
depth -= 1
if depth == 0:
return i
elif ch == '"':
i += 1
while i < len(source) and source[i] != '"':
i += 1
i += 1
return -1
def _split_macro_args(inner: str) -> list[str]:
"""Split macro arguments respecting nested parens and strings."""
args: list[str] = []
depth = 0
current: list[str] = []
i = 0
while i < len(inner):
ch = inner[i]
if ch == '"':
current.append(ch)
i += 1
while i < len(inner) and inner[i] != '"':
current.append(inner[i])
i += 1
if i < len(inner):
current.append(inner[i])
elif ch == '(':
depth += 1
current.append(ch)
elif ch == ')':
if depth == 0:
args.append(''.join(current))
break
depth -= 1
current.append(ch)
elif ch == ',' and depth == 0:
args.append(''.join(current))
current = []
else:
current.append(ch)
i += 1
if current:
remaining = ''.join(current).strip()
if remaining:
args.append(remaining)
return args
def _parse_rom_entries(block: str) -> list[dict]:
"""Parse ROM entries from a ROM block (content between ROM_START and ROM_END).
Uses regex scanning over the entire block (not line-by-line) to handle
macro-expanded content where multiple statements may be on one line.
Processes matches in order of appearance to track region and BIOS context.
"""
roms: list[dict] = []
current_region = ''
bios_labels: dict[int, tuple[str, str]] = {}
# Build a combined pattern that matches all interesting tokens
# and process them in order of occurrence
token_patterns = [
('region', _ROM_REGION),
('bios_label', _ROM_SYSTEM_BIOS),
('rom_load', _ROM_LOAD),
]
# Collect all matches with their positions
events: list[tuple[int, str, re.Match]] = []
for tag, pat in token_patterns:
for m in pat.finditer(block):
events.append((m.start(), tag, m))
# Sort by position in block
events.sort(key=lambda e: e[0])
for _pos, tag, m in events:
if tag == 'region':
current_region = m.group(2)
elif tag == 'bios_label':
idx = int(m.group(1))
bios_labels[idx] = (m.group(2), m.group(3))
elif tag == 'rom_load':
# Get the full macro call as context (find closing paren)
context_start = m.start()
# Find the opening paren of the ROM_LOAD macro
paren_pos = block.find('(', context_start)
if paren_pos != -1:
close_pos = _find_closing_paren(block, paren_pos)
context_end = close_pos + 1 if close_pos != -1 else m.end() + 200
else:
context_end = m.end() + 200
context = block[context_start:min(context_end, len(block))]
if _NO_DUMP.search(context):
continue
rom_name = m.group(1)
rom_size = _parse_int(m.group(3))
crc_sha_match = _CRC_SHA.search(context)
crc32 = ''
sha1 = ''
if crc_sha_match:
crc32 = crc_sha_match.group(1).lower()
sha1 = crc_sha_match.group(2).lower()
bad_dump = bool(_BAD_DUMP.search(context))
bios_index = None
bios_label = ''
bios_description = ''
bios_ref = _ROM_BIOS.search(context)
if bios_ref:
bios_index = int(bios_ref.group(1))
if bios_index in bios_labels:
bios_label, bios_description = bios_labels[bios_index]
entry: dict = {
'name': rom_name,
'size': rom_size,
'crc32': crc32,
'sha1': sha1,
'region': current_region,
'bad_dump': bad_dump,
}
if bios_index is not None:
entry['bios_index'] = bios_index
entry['bios_label'] = bios_label
entry['bios_description'] = bios_description
roms.append(entry)
return roms
def _parse_int(value: str) -> int:
"""Parse an integer that may be hex (0x...) or decimal."""
value = value.strip()
if value.startswith('0x') or value.startswith('0X'):
return int(value, 16)
return int(value)

View File

@@ -27,6 +27,14 @@ SOURCE_URL = (
GITHUB_REPO = "RetroBat-Official/retrobat"
# Map RetroBat system keys to our normalized system IDs
SYSTEM_SLUG_MAP = {
"ps2": "sony-playstation-2",
"ps3": "sony-playstation-3",
"psvita": "sony-playstation-vita",
"gsplus": "apple-iigs",
}
class Scraper(BaseScraper):
"""Scraper for RetroBat batocera-systems.json."""
@@ -83,7 +91,7 @@ class Scraper(BaseScraper):
requirements.append(BiosRequirement(
name=name,
system=sys_key,
system=SYSTEM_SLUG_MAP.get(sys_key, sys_key),
md5=md5 or None,
destination=file_path,
required=True,
@@ -113,10 +121,25 @@ class Scraper(BaseScraper):
"""Generate a platform YAML config dict from scraped data."""
requirements = self.fetch_requirements()
# Parse source to extract display names per system
raw = self._fetch_raw()
source_data = json.loads(raw)
display_names: dict[str, str] = {}
for sys_key, sys_data in source_data.items():
if isinstance(sys_data, dict):
dname = sys_data.get("name", "")
if dname:
slug = SYSTEM_SLUG_MAP.get(sys_key, sys_key)
display_names[slug] = dname
systems = {}
for req in requirements:
if req.system not in systems:
systems[req.system] = {"files": []}
sys_entry: dict = {"files": []}
dname = display_names.get(req.system)
if dname:
sys_entry["name"] = dname
systems[req.system] = sys_entry
entry = {
"name": req.name,

View File

@@ -38,8 +38,14 @@ def _enrich_hashes(entry: dict, db: dict) -> None:
sha1 = entry.get("sha1", "")
md5 = entry.get("md5", "")
# Hashes can be lists (multi-hash) — use first string value
if isinstance(sha1, list):
sha1 = sha1[0] if sha1 else ""
if isinstance(md5, list):
md5 = md5[0] if md5 else ""
record = None
if sha1 and db.get("files"):
if sha1 and isinstance(sha1, str) and db.get("files"):
record = db["files"].get(sha1)
if record is None and md5:
by_md5 = db.get("by_md5", {})
@@ -323,9 +329,46 @@ def _diff_system(truth_sys: dict, scraped_sys: dict) -> dict:
"scraped_required": s_req,
})
# Truth files not matched -> missing
for fe in truth_sys.get("files", []):
if fe["name"].lower() not in matched_truth_names:
# Collect unmatched files from both sides
unmatched_truth = [
fe for fe in truth_sys.get("files", [])
if fe["name"].lower() not in matched_truth_names
]
unmatched_scraped = {
s_key: s_entry for s_key, s_entry in scraped_index.items()
if s_key not in truth_index
}
# Hash-based fallback: detect platform renames (e.g. Batocera ROM → ROM1)
# If an unmatched scraped file shares a hash with an unmatched truth file,
# it's the same file under a different name — a platform rename, not a gap.
rename_matched_truth: set[str] = set()
rename_matched_scraped: set[str] = set()
if unmatched_truth and unmatched_scraped:
# Build hash → truth file index for unmatched truth files
truth_hash_index: dict[str, dict] = {}
for fe in unmatched_truth:
for h in ("sha1", "md5", "crc32"):
val = fe.get(h)
if val and isinstance(val, str):
truth_hash_index[val.lower()] = fe
for s_key, s_entry in unmatched_scraped.items():
for h in ("sha1", "md5", "crc32"):
s_val = s_entry.get(h)
if not s_val or not isinstance(s_val, str):
continue
t_entry = truth_hash_index.get(s_val.lower())
if t_entry is not None:
# Rename detected — count as matched
rename_matched_truth.add(t_entry["name"].lower())
rename_matched_scraped.add(s_key)
break
# Truth files not matched (by name, alias, or hash) -> missing
for fe in unmatched_truth:
if fe["name"].lower() not in rename_matched_truth:
missing.append({
"name": fe["name"],
"cores": list(fe.get("_cores", [])),
@@ -335,13 +378,14 @@ def _diff_system(truth_sys: dict, scraped_sys: dict) -> dict:
# Scraped files not in truth -> extra
coverage = truth_sys.get("_coverage", {})
has_unprofiled = bool(coverage.get("cores_unprofiled"))
for s_key, s_entry in scraped_index.items():
if s_key not in truth_index:
entry = {"name": s_entry["name"]}
if has_unprofiled:
extra_unprofiled.append(entry)
else:
extra_phantom.append(entry)
for s_key, s_entry in unmatched_scraped.items():
if s_key in rename_matched_scraped:
continue
entry = {"name": s_entry["name"]}
if has_unprofiled:
extra_unprofiled.append(entry)
else:
extra_phantom.append(entry)
result: dict = {}
if missing:

View File

@@ -31,10 +31,11 @@ from pathlib import Path
sys.path.insert(0, os.path.dirname(__file__))
from common import (
build_target_cores_cache, build_zip_contents_index, check_inside_zip,
compute_hashes, filter_systems_by_target, group_identical_platforms,
list_emulator_profiles, list_system_ids, load_data_dir_registry,
load_emulator_profiles, load_platform_config, md5sum, md5_composite,
require_yaml, resolve_local_file, resolve_platform_cores,
compute_hashes, expand_platform_declared_names, filter_systems_by_target,
group_identical_platforms, list_emulator_profiles, list_system_ids,
load_data_dir_registry, load_emulator_profiles, load_platform_config,
md5sum, md5_composite, require_yaml, resolve_local_file,
resolve_platform_cores,
)
yaml = require_yaml()
@@ -261,13 +262,9 @@ def find_undeclared_files(
data_names: set[str] | None = None,
) -> list[dict]:
"""Find files needed by cores but not declared in platform config."""
# Collect all filenames declared by this platform
declared_names: set[str] = set()
for sys_id, system in config.get("systems", {}).items():
for fe in system.get("files", []):
name = fe.get("name", "")
if name:
declared_names.add(name)
# Collect all filenames declared by this platform, enriched with
# canonical names from DB via MD5 (handles platform renaming)
declared_names = expand_platform_declared_names(config, db)
# Collect data_directory refs
declared_dd: set[str] = set()
@@ -294,6 +291,10 @@ def find_undeclared_files(
if emu_name not in relevant:
continue
# Skip agnostic profiles entirely (filename-agnostic BIOS detection)
if profile.get("bios_mode") == "agnostic":
continue
# Check if this profile is standalone: match profile name or any cores: alias
is_standalone = emu_name in standalone_set or bool(
standalone_set & {str(c) for c in profile.get("cores", [])}
@@ -320,6 +321,10 @@ def find_undeclared_files(
if load_from and load_from != "system_dir":
continue
# Skip agnostic files (filename-agnostic, handled by agnostic scan)
if f.get("agnostic"):
continue
archive = f.get("archive")
# Skip files declared by the platform (by name or archive)

View File

@@ -31,6 +31,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts"))
import yaml
from common import (
build_zip_contents_index, check_inside_zip, compute_hashes,
expand_platform_declared_names,
group_identical_platforms, load_emulator_profiles, load_platform_config,
md5_composite, md5sum, parse_md5_list, resolve_local_file,
resolve_platform_cores, safe_extract_zip,
@@ -169,6 +170,7 @@ class TestE2E(unittest.TestCase):
"md5": info["md5"],
"name": name,
"crc32": info.get("crc32", ""),
"size": len(info["data"]),
}
by_md5[info["md5"]] = sha1
by_name.setdefault(name, []).append(sha1)
@@ -269,6 +271,12 @@ class TestE2E(unittest.TestCase):
{"ref": "test-data-dir", "destination": "TestData"},
],
},
"sys-renamed": {
"files": [
{"name": "renamed_file.bin", "destination": "renamed_file.bin",
"md5": f["correct_hash.bin"]["md5"], "required": True},
],
},
},
}
with open(os.path.join(self.platforms_dir, "test_md5.yml"), "w") as fh:
@@ -490,6 +498,46 @@ class TestE2E(unittest.TestCase):
with open(os.path.join(self.emulators_dir, "test_subdir_core.yml"), "w") as fh:
yaml.dump(emu_subdir, fh)
# Emulator whose file is declared by platform under a different name
# (e.g. gsplus ROM vs Batocera ROM1) — hash-based matching should resolve
emu_renamed = {
"emulator": "TestRenamed",
"type": "standalone",
"systems": ["sys-renamed"],
"files": [
{"name": "correct_hash.bin", "required": True},
],
}
with open(os.path.join(self.emulators_dir, "test_renamed.yml"), "w") as fh:
yaml.dump(emu_renamed, fh)
# Agnostic profile (bios_mode: agnostic) — skipped by find_undeclared_files
emu_agnostic = {
"emulator": "TestAgnostic",
"type": "standalone",
"bios_mode": "agnostic",
"systems": ["console-a"],
"files": [
{"name": "correct_hash.bin", "required": True,
"min_size": 1, "max_size": 999999},
],
}
with open(os.path.join(self.emulators_dir, "test_agnostic.yml"), "w") as fh:
yaml.dump(emu_agnostic, fh)
# Mixed profile with per-file agnostic
emu_mixed_agnostic = {
"emulator": "TestMixedAgnostic",
"type": "libretro",
"systems": ["console-a"],
"files": [
{"name": "undeclared_req.bin", "required": True},
{"name": "agnostic_file.bin", "required": True, "agnostic": True},
],
}
with open(os.path.join(self.emulators_dir, "test_mixed_agnostic.yml"), "w") as fh:
yaml.dump(emu_mixed_agnostic, fh)
# ---------------------------------------------------------------
# THE TEST -one method per feature area, all using same fixtures
# ---------------------------------------------------------------
@@ -3122,7 +3170,8 @@ class TestE2E(unittest.TestCase):
self.assertIn("Sony - PlayStation", content)
self.assertIn("scph5501.bin", content)
self.assertIn("b056ee5a4d65937e1a3a17e1e78f3258ea49c38e", content)
self.assertIn('name "System.dat"', content)
self.assertIn('name "System"', content)
self.assertIn("71AF80B4", content) # CRC uppercase
issues = exporter.validate(truth, out_path)
self.assertEqual(issues, [])
@@ -3251,6 +3300,433 @@ class TestE2E(unittest.TestCase):
self.assertEqual(div["extra_unprofiled"][0]["name"], "phantom.bin")
self.assertNotIn("extra_phantom", div)
def test_173_cross_ref_hash_matching(self):
"""Platform file under different name matched by MD5 is not undeclared."""
config = load_platform_config("test_md5", self.platforms_dir)
profiles = load_emulator_profiles(self.emulators_dir)
undeclared = find_undeclared_files(config, self.emulators_dir, self.db, profiles)
names = {u["name"] for u in undeclared}
# correct_hash.bin is declared by platform as renamed_file.bin with same MD5
# hash-based matching should suppress it from undeclared
self.assertNotIn("correct_hash.bin", names)
def test_174_expand_platform_declared_names(self):
"""expand_platform_declared_names enriches with DB canonical names."""
config = load_platform_config("test_md5", self.platforms_dir)
result = expand_platform_declared_names(config, self.db)
# renamed_file.bin is declared directly
self.assertIn("renamed_file.bin", result)
# correct_hash.bin is the DB canonical name for the same MD5
self.assertIn("correct_hash.bin", result)
# ---------------------------------------------------------------
# Registry merge + all_libretro expansion + diff hash fallback
# ---------------------------------------------------------------
def test_175_registry_merge_cores(self):
"""load_platform_config merges cores from _registry.yml."""
from common import _platform_config_cache
# Platform YAML with 1 core
config = {
"platform": "TestMerge",
"cores": ["core_a"],
"systems": {"test-system": {"files": []}},
}
with open(os.path.join(self.platforms_dir, "testmerge.yml"), "w") as f:
yaml.dump(config, f)
# Registry with 2 cores (superset)
registry = {
"platforms": {
"testmerge": {
"config": "testmerge.yml",
"status": "active",
"cores": ["core_a", "core_b"],
}
}
}
with open(os.path.join(self.platforms_dir, "_registry.yml"), "w") as f:
yaml.dump(registry, f)
_platform_config_cache.clear()
loaded = load_platform_config("testmerge", self.platforms_dir)
cores = [str(c) for c in loaded["cores"]]
self.assertIn("core_a", cores)
self.assertIn("core_b", cores)
def test_176_all_libretro_in_list(self):
"""resolve_platform_cores expands all_libretro/retroarch in a list."""
from common import resolve_platform_cores, load_emulator_profiles
# Create a libretro profile and a standalone profile
for name, ptype in [("lr_core", "libretro"), ("sa_core", "standalone")]:
profile = {
"emulator": name,
"type": ptype,
"cores": [name],
"systems": ["test-system"],
"files": [],
}
with open(os.path.join(self.emulators_dir, f"{name}.yml"), "w") as f:
yaml.dump(profile, f)
profiles = load_emulator_profiles(self.emulators_dir)
# Config with retroarch + sa_core in cores list
config = {"cores": ["retroarch", "sa_core"]}
resolved = resolve_platform_cores(config, profiles)
self.assertIn("lr_core", resolved) # expanded via retroarch
self.assertIn("sa_core", resolved) # explicit
def test_177_diff_hash_fallback_rename(self):
"""Diff detects platform renames via hash fallback."""
from truth import diff_platform_truth
truth = {
"systems": {
"test-system": {
"_coverage": {"cores_profiled": ["c"], "cores_unprofiled": []},
"files": [
{"name": "ROM", "required": True, "md5": "abcd1234" * 4,
"_cores": ["c"], "_source_refs": []},
],
}
}
}
scraped = {
"systems": {
"test-system": {
"files": [
{"name": "ROM1", "required": True, "md5": "abcd1234" * 4},
],
}
}
}
result = diff_platform_truth(truth, scraped)
# ROM and ROM1 share the same hash — rename, not missing+phantom
self.assertEqual(result["summary"]["total_missing"], 0)
self.assertEqual(result["summary"]["total_extra_phantom"], 0)
def test_178_diff_system_normalization(self):
"""Diff matches systems with different IDs via normalization."""
from truth import diff_platform_truth
truth = {
"systems": {
"sega-gamegear": {
"_coverage": {"cores_profiled": ["c"], "cores_unprofiled": []},
"files": [
{"name": "bios.gg", "required": True, "md5": "a" * 32,
"_cores": ["c"], "_source_refs": []},
],
},
}
}
scraped = {
"systems": {
"sega-game-gear": {
"files": [
{"name": "bios.gg", "required": True, "md5": "a" * 32},
],
},
}
}
result = diff_platform_truth(truth, scraped)
self.assertEqual(result["summary"]["systems_uncovered"], 0)
self.assertEqual(result["summary"]["total_missing"], 0)
self.assertEqual(result["summary"]["systems_compared"], 1)
def test_179_agnostic_profile_skipped_in_undeclared(self):
"""bios_mode: agnostic profiles are skipped entirely by find_undeclared_files."""
config = load_platform_config("test_existence", self.platforms_dir)
profiles = load_emulator_profiles(self.emulators_dir)
undeclared = find_undeclared_files(config, self.emulators_dir, self.db, profiles)
emulators = {u["emulator"] for u in undeclared}
# TestAgnostic should NOT appear in undeclared (bios_mode: agnostic)
self.assertNotIn("TestAgnostic", emulators)
def test_180_agnostic_file_skipped_in_undeclared(self):
"""Files with agnostic: true are skipped, others in same profile are not."""
config = load_platform_config("test_existence", self.platforms_dir)
profiles = load_emulator_profiles(self.emulators_dir)
undeclared = find_undeclared_files(config, self.emulators_dir, self.db, profiles)
names = {u["name"] for u in undeclared}
# agnostic_file.bin should NOT be in undeclared (agnostic: true)
self.assertNotIn("agnostic_file.bin", names)
# undeclared_req.bin should still be in undeclared (not agnostic)
self.assertIn("undeclared_req.bin", names)
def test_181_agnostic_extras_scan(self):
"""Agnostic profiles add all matching DB files as extras."""
from generate_pack import _collect_emulator_extras
config = load_platform_config("test_existence", self.platforms_dir)
profiles = load_emulator_profiles(self.emulators_dir)
extras = _collect_emulator_extras(
config, self.emulators_dir, self.db, set(), "system", profiles,
)
agnostic_extras = [e for e in extras if e.get("source_emulator") == "TestAgnostic"]
# Agnostic scan should find files in the same directory as correct_hash.bin
self.assertTrue(len(agnostic_extras) > 0, "Agnostic scan should produce extras")
# All agnostic extras should have agnostic_scan flag
for e in agnostic_extras:
self.assertTrue(e.get("agnostic_scan", False))
def test_182_agnostic_rename_readme(self):
"""_build_agnostic_rename_readme generates correct text."""
from generate_pack import _build_agnostic_rename_readme
result = _build_agnostic_rename_readme(
"dsi_nand.bin", "DSi_Nand_AUS.bin",
["DSi_Nand_EUR.bin", "DSi_Nand_USA.bin"],
)
self.assertIn("dsi_nand.bin <- DSi_Nand_AUS.bin", result)
self.assertIn("DSi_Nand_EUR.bin", result)
self.assertIn("DSi_Nand_USA.bin", result)
self.assertIn("rename it to: dsi_nand.bin", result)
def test_183_agnostic_resolve_fallback(self):
"""resolve_local_file with agnostic fallback finds a system file."""
file_entry = {
"name": "nonexistent_agnostic.bin",
"agnostic": True,
"min_size": 1,
"max_size": 999999,
"agnostic_path_prefix": self.bios_dir + "/",
}
path, status = resolve_local_file(file_entry, self.db)
self.assertIsNotNone(path)
self.assertEqual(status, "agnostic_fallback")
def test_179_batocera_exporter_round_trip(self):
"""Batocera exporter produces valid Python dict format."""
from exporter.batocera_exporter import Exporter
truth = {
"systems": {
"sony-playstation": {
"_coverage": {"cores_profiled": ["c"]},
"files": [
{"name": "scph5501.bin", "destination": "scph5501.bin",
"required": True, "md5": "b" * 32,
"_cores": ["c"], "_source_refs": []},
],
}
}
}
scraped = {
"systems": {
"sony-playstation": {"native_id": "psx", "files": []},
}
}
out = os.path.join(self.root, "batocera-systems")
exp = Exporter()
exp.export(truth, out, scraped_data=scraped)
content = open(out).read()
self.assertIn('"psx"', content)
self.assertIn("scph5501.bin", content)
self.assertIn("b" * 32, content)
self.assertEqual(exp.validate(truth, out), [])
def test_180_recalbox_exporter_round_trip(self):
"""Recalbox exporter produces valid es_bios.xml."""
from exporter.recalbox_exporter import Exporter
truth = {
"systems": {
"sony-playstation": {
"_coverage": {"cores_profiled": ["c"]},
"files": [
{"name": "scph5501.bin", "destination": "scph5501.bin",
"required": True, "md5": "b" * 32,
"_cores": ["c"], "_source_refs": []},
],
}
}
}
scraped = {
"systems": {
"sony-playstation": {"native_id": "psx", "files": []},
}
}
out = os.path.join(self.root, "es_bios.xml")
exp = Exporter()
exp.export(truth, out, scraped_data=scraped)
content = open(out).read()
self.assertIn("<biosList", content)
self.assertIn('platform="psx"', content)
self.assertIn('fullname=', content)
self.assertIn("scph5501.bin", content)
# mandatory="true" is the default, not emitted (matching Recalbox format)
self.assertNotIn('mandatory="false"', content)
self.assertIn('core="libretro/c"', content)
self.assertEqual(exp.validate(truth, out), [])
def test_181_retrobat_exporter_round_trip(self):
"""RetroBat exporter produces valid JSON."""
import json as _json
from exporter.retrobat_exporter import Exporter
truth = {
"systems": {
"sony-playstation": {
"_coverage": {"cores_profiled": ["c"]},
"files": [
{"name": "scph5501.bin", "destination": "scph5501.bin",
"required": True, "md5": "b" * 32,
"_cores": ["c"], "_source_refs": []},
],
}
}
}
scraped = {
"systems": {
"sony-playstation": {"native_id": "psx", "files": []},
}
}
out = os.path.join(self.root, "batocera-systems.json")
exp = Exporter()
exp.export(truth, out, scraped_data=scraped)
data = _json.loads(open(out).read())
self.assertIn("psx", data)
self.assertTrue(any("scph5501" in bf["file"] for bf in data["psx"]["biosFiles"]))
self.assertEqual(exp.validate(truth, out), [])
def test_182_exporter_discovery(self):
"""All exporters are discovered by the plugin system."""
from exporter import discover_exporters
exporters = discover_exporters()
self.assertIn("retroarch", exporters)
self.assertIn("batocera", exporters)
self.assertIn("recalbox", exporters)
self.assertIn("retrobat", exporters)
# ---------------------------------------------------------------
# Hash scraper: parsers + merge
# ---------------------------------------------------------------
def test_mame_parser_finds_bios_root_sets(self):
from scripts.scraper.mame_parser import find_bios_root_sets, parse_rom_block
source = '''
ROM_START( neogeo )
ROM_REGION( 0x020000, "mainbios", 0 )
ROM_LOAD( "sp-s2.sp1", 0x00000, 0x020000, CRC(9036d879) SHA1(4f834c580f3471ce40c3210ef5e7491df38d8851) )
ROM_END
GAME( 1990, neogeo, 0, ng, neogeo, ng_state, empty_init, ROT0, "SNK", "Neo Geo", MACHINE_IS_BIOS_ROOT )
ROM_START( pacman )
ROM_REGION( 0x10000, "maincpu", 0 )
ROM_LOAD( "pacman.6e", 0x0000, 0x1000, CRC(c1e6ab10) SHA1(e87e059c5be45753f7e9f33dff851f16d6751181) )
ROM_END
GAME( 1980, pacman, 0, pacman, pacman, pacman_state, empty_init, ROT90, "Namco", "Pac-Man", 0 )
'''
sets = find_bios_root_sets(source, "neogeo.cpp")
self.assertIn("neogeo", sets)
self.assertNotIn("pacman", sets)
roms = parse_rom_block(source, "neogeo")
self.assertEqual(len(roms), 1)
self.assertEqual(roms[0]["crc32"], "9036d879")
def test_fbneo_parser_finds_bios_sets(self):
from scripts.scraper.fbneo_parser import find_bios_sets, parse_rom_info
source = '''
static struct BurnRomInfo neogeoRomDesc[] = {
{ "sp-s2.sp1", 0x020000, 0x9036d879, BRF_ESS | BRF_BIOS },
{ "", 0, 0, 0 }
};
STD_ROM_PICK(neogeo)
STD_ROM_FN(neogeo)
struct BurnDriver BurnDrvneogeo = {
"neogeo", NULL, NULL, NULL, "1990",
"Neo Geo\\0", "BIOS only", "SNK", "Neo Geo MVS",
NULL, NULL, NULL, NULL, BDF_BOARDROM, 0, 0,
0, 0, 0, NULL, neogeoRomInfo, neogeoRomName, NULL, NULL,
NULL, NULL, NULL, NULL, 0
};
'''
sets = find_bios_sets(source, "d_neogeo.cpp")
self.assertIn("neogeo", sets)
roms = parse_rom_info(source, "neogeo")
self.assertEqual(len(roms), 1)
self.assertEqual(roms[0]["crc32"], "9036d879")
def test_mame_merge_preserves_manual_fields(self):
import json as json_mod
from scripts.scraper._hash_merge import merge_mame_profile
merge_dir = os.path.join(self.root, "merge_mame")
os.makedirs(merge_dir)
profile = {
"emulator": "Test", "type": "libretro",
"upstream": "https://github.com/mamedev/mame",
"core_version": "0.285",
"files": [{
"name": "neogeo.zip", "required": True, "category": "bios_zip",
"system": "snk-neogeo-mvs", "note": "MVS BIOS",
"source_ref": "old.cpp:1",
"contents": [{"name": "sp-s2.sp1", "size": 131072, "crc32": "oldcrc"}],
}],
}
profile_path = os.path.join(merge_dir, "test.yml")
with open(profile_path, "w") as f:
yaml.dump(profile, f, sort_keys=False)
hashes = {
"source": "mamedev/mame", "version": "0.286", "commit": "abc",
"fetched_at": "2026-03-30T00:00:00Z",
"bios_sets": {"neogeo": {
"source_file": "neo.cpp", "source_line": 42,
"roms": [{"name": "sp-s2.sp1", "size": 131072, "crc32": "newcrc", "sha1": "abc123"}],
}},
}
hashes_path = os.path.join(merge_dir, "hashes.json")
with open(hashes_path, "w") as f:
json_mod.dump(hashes, f)
result = merge_mame_profile(profile_path, hashes_path)
neo = next(f for f in result["files"] if f["name"] == "neogeo.zip")
self.assertEqual(neo["contents"][0]["crc32"], "newcrc")
self.assertEqual(neo["system"], "snk-neogeo-mvs")
self.assertEqual(neo["note"], "MVS BIOS")
self.assertEqual(neo["source_ref"], "neo.cpp:42")
self.assertEqual(result["core_version"], "0.286")
def test_fbneo_merge_updates_individual_roms(self):
import json as json_mod
from scripts.scraper._hash_merge import merge_fbneo_profile
merge_dir = os.path.join(self.root, "merge_fbneo")
os.makedirs(merge_dir)
profile = {
"emulator": "FBNeo", "type": "libretro",
"upstream": "https://github.com/finalburnneo/FBNeo",
"core_version": "v1.0.0.02",
"files": [{"name": "sp-s2.sp1", "archive": "neogeo.zip",
"system": "snk-neogeo-mvs", "required": True,
"size": 131072, "crc32": "oldcrc"}],
}
profile_path = os.path.join(merge_dir, "fbneo.yml")
with open(profile_path, "w") as f:
yaml.dump(profile, f, sort_keys=False)
hashes = {
"source": "finalburnneo/FBNeo", "version": "v1.0.0.03", "commit": "def",
"fetched_at": "2026-03-30T00:00:00Z",
"bios_sets": {"neogeo": {
"source_file": "neo.cpp", "source_line": 10,
"roms": [{"name": "sp-s2.sp1", "size": 131072, "crc32": "newcrc"}],
}},
}
hashes_path = os.path.join(merge_dir, "hashes.json")
with open(hashes_path, "w") as f:
json_mod.dump(hashes, f)
result = merge_fbneo_profile(profile_path, hashes_path)
rom = next(f for f in result["files"] if f["name"] == "sp-s2.sp1")
self.assertEqual(rom["crc32"], "newcrc")
self.assertEqual(rom["system"], "snk-neogeo-mvs")
self.assertEqual(result["core_version"], "v1.0.0.03")
if __name__ == "__main__":
unittest.main()

190
tests/test_fbneo_parser.py Normal file
View File

@@ -0,0 +1,190 @@
"""Tests for the FBNeo source parser."""
from __future__ import annotations
import os
import tempfile
import unittest
from pathlib import Path
from scripts.scraper.fbneo_parser import (
find_bios_sets,
parse_fbneo_source_tree,
parse_rom_info,
)
NEOGEO_FIXTURE = """\
static struct BurnRomInfo neogeoRomDesc[] = {
{ "sp-s2.sp1", 0x020000, 0x9036d879, BRF_ESS | BRF_BIOS },
{ "sp-s.sp1", 0x020000, 0xc7f2fa45, BRF_ESS | BRF_BIOS },
{ "asia-s3.rom", 0x020000, 0x91b64be3, BRF_ESS | BRF_BIOS },
{ "vs-bios.rom", 0x020000, 0xf0e8f27d, BRF_ESS | BRF_BIOS },
{ "uni-bios.rom", 0x020000, 0x2d50996a, BRF_ESS | BRF_BIOS },
{ "", 0, 0, 0 }
};
STD_ROM_FN(neogeo)
struct BurnDriver BurnDrvneogeo = {
"neogeo", NULL, NULL, NULL, "1990",
"Neo Geo\\0", "BIOS only", "SNK", "Neo Geo MVS",
NULL, NULL, NULL, NULL,
BDF_BOARDROM, 0, HARDWARE_PREFIX_CARTRIDGE | HARDWARE_SNK_NEOGEO,
GBF_BIOS, 0,
NULL, neogeoRomInfo, neogeoRomName, NULL, NULL, NULL, NULL,
neogeoInputInfo, neogeoDIPInfo,
NULL, NULL, NULL, NULL, 0x1000,
304, 224, 4, 3
};
"""
PGM_FIXTURE = """\
static struct BurnRomInfo pgmRomDesc[] = {
{ "pgm_t01s.rom", 0x200000, 0x1a7123a0, BRF_GRA },
{ "pgm_m01s.rom", 0x200000, 0x45ae7159, BRF_SND },
{ "pgm_p01s.rom", 0x020000, 0xe42b166e, BRF_ESS | BRF_BIOS },
{ "", 0, 0, 0 }
};
STD_ROM_FN(pgm)
struct BurnDriver BurnDrvpgm = {
"pgm", NULL, NULL, NULL, "1997",
"PGM (Polygame Master)\\0", "BIOS only", "IGS", "PGM",
NULL, NULL, NULL, NULL,
BDF_BOARDROM, 0, HARDWARE_IGS_PGM,
GBF_BIOS, 0,
NULL, pgmRomInfo, pgmRomName, NULL, NULL, NULL, NULL,
pgmInputInfo, pgmDIPInfo,
NULL, NULL, NULL, NULL, 0x900,
448, 224, 4, 3
};
"""
NON_BIOS_FIXTURE = """\
static struct BurnRomInfo mslugRomDesc[] = {
{ "201-p1.p1", 0x100000, 0x08d8daa5, BRF_ESS | BRF_PRG },
{ "", 0, 0, 0 }
};
STD_ROM_FN(mslug)
struct BurnDriver BurnDrvmslug = {
"mslug", NULL, "neogeo", NULL, "1996",
"Metal Slug\\0", NULL, "Nazca", "Neo Geo MVS",
NULL, NULL, NULL, NULL,
BDF_GAME_WORKING, 2, HARDWARE_PREFIX_CARTRIDGE | HARDWARE_SNK_NEOGEO,
GBF_PLATFORM | GBF_HORSHOOT, 0,
NULL, mslugRomInfo, mslugRomName, NULL, NULL, NULL, NULL,
neogeoInputInfo, neogeoDIPInfo,
NULL, NULL, NULL, NULL, 0x1000,
304, 224, 4, 3
};
"""
class TestFindBiosSets(unittest.TestCase):
def test_detects_neogeo(self) -> None:
result = find_bios_sets(NEOGEO_FIXTURE, 'd_neogeo.cpp')
self.assertIn('neogeo', result)
self.assertEqual(result['neogeo']['source_file'], 'd_neogeo.cpp')
def test_detects_pgm(self) -> None:
result = find_bios_sets(PGM_FIXTURE, 'd_pgm.cpp')
self.assertIn('pgm', result)
self.assertEqual(result['pgm']['source_file'], 'd_pgm.cpp')
def test_ignores_non_bios(self) -> None:
result = find_bios_sets(NON_BIOS_FIXTURE, 'd_neogeo.cpp')
self.assertEqual(result, {})
def test_source_line_positive(self) -> None:
result = find_bios_sets(NEOGEO_FIXTURE, 'd_neogeo.cpp')
self.assertGreater(result['neogeo']['source_line'], 0)
class TestParseRomInfo(unittest.TestCase):
def test_neogeo_rom_count(self) -> None:
roms = parse_rom_info(NEOGEO_FIXTURE, 'neogeo')
self.assertEqual(len(roms), 5)
def test_sentinel_skipped(self) -> None:
roms = parse_rom_info(NEOGEO_FIXTURE, 'neogeo')
names = [r['name'] for r in roms]
self.assertNotIn('', names)
def test_crc32_lowercase_hex(self) -> None:
roms = parse_rom_info(NEOGEO_FIXTURE, 'neogeo')
first = roms[0]
self.assertEqual(first['crc32'], '9036d879')
self.assertRegex(first['crc32'], r'^[0-9a-f]{8}$')
def test_no_sha1(self) -> None:
roms = parse_rom_info(NEOGEO_FIXTURE, 'neogeo')
for rom in roms:
self.assertNotIn('sha1', rom)
def test_neogeo_first_rom(self) -> None:
roms = parse_rom_info(NEOGEO_FIXTURE, 'neogeo')
first = roms[0]
self.assertEqual(first['name'], 'sp-s2.sp1')
self.assertEqual(first['size'], 0x020000)
self.assertEqual(first['crc32'], '9036d879')
def test_pgm_rom_count(self) -> None:
roms = parse_rom_info(PGM_FIXTURE, 'pgm')
self.assertEqual(len(roms), 3)
def test_pgm_bios_entry(self) -> None:
roms = parse_rom_info(PGM_FIXTURE, 'pgm')
bios = roms[2]
self.assertEqual(bios['name'], 'pgm_p01s.rom')
self.assertEqual(bios['crc32'], 'e42b166e')
def test_unknown_set_returns_empty(self) -> None:
roms = parse_rom_info(NEOGEO_FIXTURE, 'nonexistent')
self.assertEqual(roms, [])
class TestParseSourceTree(unittest.TestCase):
def test_walks_drv_directory(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
drv_dir = Path(tmpdir) / 'src' / 'burn' / 'drv' / 'neogeo'
drv_dir.mkdir(parents=True)
(drv_dir / 'd_neogeo.cpp').write_text(NEOGEO_FIXTURE)
result = parse_fbneo_source_tree(tmpdir)
self.assertIn('neogeo', result)
self.assertEqual(len(result['neogeo']['roms']), 5)
def test_skips_non_cpp(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
drv_dir = Path(tmpdir) / 'src' / 'burn' / 'drv'
drv_dir.mkdir(parents=True)
(drv_dir / 'd_neogeo.h').write_text(NEOGEO_FIXTURE)
result = parse_fbneo_source_tree(tmpdir)
self.assertEqual(result, {})
def test_missing_directory_returns_empty(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
result = parse_fbneo_source_tree(tmpdir)
self.assertEqual(result, {})
def test_multiple_sets(self) -> None:
combined = NEOGEO_FIXTURE + '\n' + PGM_FIXTURE
with tempfile.TemporaryDirectory() as tmpdir:
drv_dir = Path(tmpdir) / 'src' / 'burn' / 'drv'
drv_dir.mkdir(parents=True)
(drv_dir / 'd_combined.cpp').write_text(combined)
result = parse_fbneo_source_tree(tmpdir)
self.assertIn('neogeo', result)
self.assertIn('pgm', result)
if __name__ == '__main__':
unittest.main()

423
tests/test_hash_merge.py Normal file
View File

@@ -0,0 +1,423 @@
"""Tests for the hash merge module."""
from __future__ import annotations
import json
import tempfile
import unittest
from pathlib import Path
import yaml
from scripts.scraper._hash_merge import (
compute_diff,
merge_fbneo_profile,
merge_mame_profile,
)
def _write_yaml(path: Path, data: dict) -> str:
p = str(path)
with open(p, 'w', encoding='utf-8') as f:
yaml.dump(data, f, default_flow_style=False, sort_keys=False)
return p
def _write_json(path: Path, data: dict) -> str:
p = str(path)
with open(p, 'w', encoding='utf-8') as f:
json.dump(data, f)
return p
def _make_mame_profile(**overrides: object) -> dict:
base = {
'emulator': 'MAME',
'core_version': '0.285',
'files': [
{
'name': 'neogeo.zip',
'required': True,
'category': 'bios_zip',
'system': 'snk-neogeo-mvs',
'source_ref': 'src/mame/neogeo/neogeo.cpp:2400',
'contents': [
{
'name': 'sp-s2.sp1',
'size': 131072,
'crc32': 'oldcrc32',
'description': 'Europe MVS (Ver. 2)',
},
],
},
],
}
base.update(overrides)
return base
def _make_mame_hashes(**overrides: object) -> dict:
base = {
'source': 'mamedev/mame',
'version': '0.286',
'commit': 'abc123',
'fetched_at': '2026-03-30T12:00:00Z',
'bios_sets': {
'neogeo': {
'source_file': 'src/mame/neogeo/neogeo.cpp',
'source_line': 2432,
'roms': [
{
'name': 'sp-s2.sp1',
'size': 131072,
'crc32': '9036d879',
'sha1': '4f834c55',
'region': 'mainbios',
'bios_label': 'euro',
'bios_description': 'Europe MVS (Ver. 2)',
},
],
},
},
}
base.update(overrides)
return base
def _make_fbneo_profile(**overrides: object) -> dict:
base = {
'emulator': 'FinalBurn Neo',
'core_version': 'v1.0.0.02',
'files': [
{
'name': 'sp-s2.sp1',
'archive': 'neogeo.zip',
'system': 'snk-neogeo-mvs',
'required': True,
'size': 131072,
'crc32': 'oldcrc32',
'source_ref': 'src/burn/drv/neogeo/d_neogeo.cpp:1605',
},
{
'name': 'hiscore.dat',
'required': False,
},
],
}
base.update(overrides)
return base
def _make_fbneo_hashes(**overrides: object) -> dict:
base = {
'source': 'finalburnneo/FBNeo',
'version': 'v1.0.0.03',
'commit': 'def456',
'fetched_at': '2026-03-30T12:00:00Z',
'bios_sets': {
'neogeo': {
'source_file': 'src/burn/drv/neogeo/d_neogeo.cpp',
'source_line': 1604,
'roms': [
{
'name': 'sp-s2.sp1',
'size': 131072,
'crc32': '9036d879',
'sha1': 'aabbccdd',
},
],
},
},
}
base.update(overrides)
return base
class TestMameMerge(unittest.TestCase):
"""Tests for merge_mame_profile."""
def test_merge_updates_contents(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', _make_mame_hashes())
result = merge_mame_profile(profile_path, hashes_path)
bios_files = [f for f in result['files'] if f.get('category') == 'bios_zip']
self.assertEqual(len(bios_files), 1)
contents = bios_files[0]['contents']
self.assertEqual(contents[0]['crc32'], '9036d879')
self.assertEqual(contents[0]['sha1'], '4f834c55')
self.assertEqual(contents[0]['description'], 'Europe MVS (Ver. 2)')
def test_merge_preserves_manual_fields(self) -> None:
profile = _make_mame_profile()
profile['files'][0]['note'] = 'manually curated note'
profile['files'][0]['system'] = 'snk-neogeo-mvs'
profile['files'][0]['required'] = False
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', profile)
hashes_path = _write_json(p / 'hashes.json', _make_mame_hashes())
result = merge_mame_profile(profile_path, hashes_path)
entry = [f for f in result['files'] if f.get('category') == 'bios_zip'][0]
self.assertEqual(entry['note'], 'manually curated note')
self.assertEqual(entry['system'], 'snk-neogeo-mvs')
self.assertFalse(entry['required'])
def test_merge_adds_new_bios_set(self) -> None:
hashes = _make_mame_hashes()
hashes['bios_sets']['pgm'] = {
'source_file': 'src/mame/igs/pgm.cpp',
'source_line': 5515,
'roms': [
{'name': 'pgm_t01s.rom', 'size': 2097152, 'crc32': '1a7123a0'},
],
}
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', hashes)
result = merge_mame_profile(profile_path, hashes_path)
bios_files = [f for f in result['files'] if f.get('category') == 'bios_zip']
names = {f['name'] for f in bios_files}
self.assertIn('pgm.zip', names)
pgm = next(f for f in bios_files if f['name'] == 'pgm.zip')
self.assertIsNone(pgm['system'])
self.assertTrue(pgm['required'])
self.assertEqual(pgm['category'], 'bios_zip')
def test_merge_preserves_non_bios_files(self) -> None:
profile = _make_mame_profile()
profile['files'].append({'name': 'hiscore.dat', 'required': False})
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', profile)
hashes_path = _write_json(p / 'hashes.json', _make_mame_hashes())
result = merge_mame_profile(profile_path, hashes_path)
non_bios = [f for f in result['files'] if f.get('category') != 'bios_zip']
self.assertEqual(len(non_bios), 1)
self.assertEqual(non_bios[0]['name'], 'hiscore.dat')
def test_merge_keeps_unmatched_bios_set(self) -> None:
"""Entries not in scraper scope stay untouched (no _upstream_removed)."""
hashes = _make_mame_hashes()
hashes['bios_sets'] = {} # nothing from scraper
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', hashes)
result = merge_mame_profile(profile_path, hashes_path)
bios_files = [f for f in result['files'] if f.get('category') == 'bios_zip']
self.assertEqual(len(bios_files), 1)
self.assertNotIn('_upstream_removed', bios_files[0])
self.assertEqual(bios_files[0]['name'], 'neogeo.zip')
def test_merge_updates_core_version(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', _make_mame_hashes())
result = merge_mame_profile(profile_path, hashes_path)
self.assertEqual(result['core_version'], '0.286')
def test_merge_backup_created(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', _make_mame_hashes())
merge_mame_profile(profile_path, hashes_path, write=True)
backup = p / 'mame.old.yml'
self.assertTrue(backup.exists())
with open(backup, encoding='utf-8') as f:
old = yaml.safe_load(f)
self.assertEqual(old['core_version'], '0.285')
def test_merge_updates_source_ref(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', _make_mame_hashes())
result = merge_mame_profile(profile_path, hashes_path)
entry = [f for f in result['files'] if f.get('category') == 'bios_zip'][0]
self.assertEqual(entry['source_ref'], 'src/mame/neogeo/neogeo.cpp:2432')
class TestFbneoMerge(unittest.TestCase):
"""Tests for merge_fbneo_profile."""
def test_merge_updates_rom_entries(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'fbneo.yml', _make_fbneo_profile())
hashes_path = _write_json(p / 'hashes.json', _make_fbneo_hashes())
result = merge_fbneo_profile(profile_path, hashes_path)
archive_files = [f for f in result['files'] if 'archive' in f]
self.assertEqual(len(archive_files), 1)
self.assertEqual(archive_files[0]['crc32'], '9036d879')
self.assertEqual(archive_files[0]['system'], 'snk-neogeo-mvs')
def test_merge_adds_new_roms(self) -> None:
hashes = _make_fbneo_hashes()
hashes['bios_sets']['neogeo']['roms'].append({
'name': 'sp-s3.sp1',
'size': 131072,
'crc32': '91b64be3',
})
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'fbneo.yml', _make_fbneo_profile())
hashes_path = _write_json(p / 'hashes.json', hashes)
result = merge_fbneo_profile(profile_path, hashes_path)
archive_files = [f for f in result['files'] if 'archive' in f]
self.assertEqual(len(archive_files), 2)
new_rom = next(f for f in archive_files if f['name'] == 'sp-s3.sp1')
self.assertEqual(new_rom['archive'], 'neogeo.zip')
self.assertTrue(new_rom['required'])
def test_merge_preserves_non_archive_files(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'fbneo.yml', _make_fbneo_profile())
hashes_path = _write_json(p / 'hashes.json', _make_fbneo_hashes())
result = merge_fbneo_profile(profile_path, hashes_path)
non_archive = [f for f in result['files'] if 'archive' not in f]
self.assertEqual(len(non_archive), 1)
self.assertEqual(non_archive[0]['name'], 'hiscore.dat')
def test_merge_keeps_unmatched_roms(self) -> None:
"""Entries not in scraper scope stay untouched (no _upstream_removed)."""
hashes = _make_fbneo_hashes()
hashes['bios_sets'] = {}
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'fbneo.yml', _make_fbneo_profile())
hashes_path = _write_json(p / 'hashes.json', hashes)
result = merge_fbneo_profile(profile_path, hashes_path)
archive_files = [f for f in result['files'] if 'archive' in f]
self.assertEqual(len(archive_files), 1)
self.assertNotIn('_upstream_removed', archive_files[0])
def test_merge_updates_core_version(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'fbneo.yml', _make_fbneo_profile())
hashes_path = _write_json(p / 'hashes.json', _make_fbneo_hashes())
result = merge_fbneo_profile(profile_path, hashes_path)
self.assertEqual(result['core_version'], 'v1.0.0.03')
class TestDiff(unittest.TestCase):
"""Tests for compute_diff."""
def test_diff_mame_detects_changes(self) -> None:
hashes = _make_mame_hashes()
hashes['bios_sets']['pgm'] = {
'source_file': 'src/mame/igs/pgm.cpp',
'source_line': 5515,
'roms': [
{'name': 'pgm_t01s.rom', 'size': 2097152, 'crc32': '1a7123a0'},
],
}
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', hashes)
diff = compute_diff(profile_path, hashes_path, mode='mame')
self.assertIn('pgm', diff['added'])
self.assertIn('neogeo', diff['updated'])
self.assertEqual(len(diff['removed']), 0)
self.assertEqual(diff['unchanged'], 0)
def test_diff_mame_out_of_scope(self) -> None:
"""Items in profile but not in scraper output = out of scope, not removed."""
hashes = _make_mame_hashes()
hashes['bios_sets'] = {}
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'mame.yml', _make_mame_profile())
hashes_path = _write_json(p / 'hashes.json', hashes)
diff = compute_diff(profile_path, hashes_path, mode='mame')
self.assertEqual(diff['removed'], [])
self.assertEqual(diff['out_of_scope'], 1)
self.assertEqual(len(diff['added']), 0)
def test_diff_fbneo_detects_changes(self) -> None:
hashes = _make_fbneo_hashes()
hashes['bios_sets']['neogeo']['roms'].append({
'name': 'sp-s3.sp1',
'size': 131072,
'crc32': '91b64be3',
})
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'fbneo.yml', _make_fbneo_profile())
hashes_path = _write_json(p / 'hashes.json', hashes)
diff = compute_diff(profile_path, hashes_path, mode='fbneo')
self.assertIn('neogeo.zip:sp-s3.sp1', diff['added'])
self.assertIn('neogeo.zip:sp-s2.sp1', diff['updated'])
self.assertEqual(len(diff['removed']), 0)
def test_diff_fbneo_unchanged(self) -> None:
profile = _make_fbneo_profile()
profile['files'][0]['crc32'] = '9036d879'
profile['files'][0]['size'] = 131072
hashes = _make_fbneo_hashes()
with tempfile.TemporaryDirectory() as td:
p = Path(td)
profile_path = _write_yaml(p / 'fbneo.yml', profile)
hashes_path = _write_json(p / 'hashes.json', hashes)
diff = compute_diff(profile_path, hashes_path, mode='fbneo')
self.assertEqual(diff['unchanged'], 1)
self.assertEqual(len(diff['added']), 0)
self.assertEqual(len(diff['updated']), 0)
if __name__ == '__main__':
unittest.main()

244
tests/test_mame_parser.py Normal file
View File

@@ -0,0 +1,244 @@
"""Tests for MAME source code parser."""
from __future__ import annotations
import os
import tempfile
import unittest
from scripts.scraper.mame_parser import (
find_bios_root_sets,
parse_mame_source_tree,
parse_rom_block,
)
# Standard GAME macro with MACHINE_IS_BIOS_ROOT, multiple ROM entries, BIOS variants
NEOGEO_FIXTURE = """\
ROM_START( neogeo )
ROM_REGION( 0x100000, "mainbios", 0 )
ROM_SYSTEM_BIOS( 0, "euro", "Europe MVS (Ver. 2)" )
ROMX_LOAD( "sp-s2.sp1", 0x00000, 0x020000, CRC(9036d879) SHA1(4f5ed7105b7128794654ce82b51723e16e389543), ROM_BIOS(0) )
ROM_SYSTEM_BIOS( 1, "japan", "Japan MVS (Ver. 3)" )
ROMX_LOAD( "vs-bios.rom", 0x00000, 0x020000, CRC(f0e8f27d) SHA1(ecf01bf6b3d6c7e4e0aae01e51e3ed4c0e1d5c2e), ROM_BIOS(1) )
ROM_REGION( 0x10000, "audiocpu", 0 )
ROM_LOAD( "sm1.sm1", 0x00000, 0x20000, CRC(94416d67) SHA1(42f9d7ddd6c0931fd64226a60dc73602b2819571) )
ROM_END
GAME( 1990, neogeo, 0, neogeo_noslot, neogeo, neogeo_state, init_neogeo, ROT0, "SNK", "Neo Geo", MACHINE_IS_BIOS_ROOT )
"""
# COMP macro with MACHINE_IS_BIOS_ROOT
DEVICE_FIXTURE = """\
ROM_START( bbcb )
ROM_REGION( 0x40000, "maincpu", 0 )
ROM_LOAD( "basic2.rom", 0x00000, 0x4000, CRC(a1b6a0e9) SHA1(6a0b9b8b7c3b3b9e6b7e8d0f2e7a6e7b8c9a0b1c) )
ROM_END
COMP( 1981, bbcb, 0, 0, bbcb, bbcb, bbc_state, init_bbc, "Acorn", "BBC Micro Model B", MACHINE_IS_BIOS_ROOT )
"""
# ROM_LOAD with NO_DUMP (should be skipped)
NODUMP_FIXTURE = """\
ROM_START( testnd )
ROM_REGION( 0x10000, "maincpu", 0 )
ROM_LOAD( "good.rom", 0x00000, 0x4000, CRC(aabbccdd) SHA1(1122334455667788990011223344556677889900) )
ROM_LOAD( "missing.rom", 0x04000, 0x4000, NO_DUMP )
ROM_END
GAME( 2000, testnd, 0, testnd, testnd, test_state, init_test, ROT0, "Test", "Test ND", MACHINE_IS_BIOS_ROOT )
"""
# ROM_LOAD with BAD_DUMP
BADDUMP_FIXTURE = """\
ROM_START( testbd )
ROM_REGION( 0x10000, "maincpu", 0 )
ROM_LOAD( "badrom.bin", 0x00000, 0x4000, BAD_DUMP CRC(deadbeef) SHA1(0123456789abcdef0123456789abcdef01234567) )
ROM_END
GAME( 2000, testbd, 0, testbd, testbd, test_state, init_test, ROT0, "Test", "Test BD", MACHINE_IS_BIOS_ROOT )
"""
# CONS macro with ROM_LOAD16_WORD
CONS_FIXTURE = """\
ROM_START( megadriv )
ROM_REGION( 0x400000, "maincpu", 0 )
ROM_LOAD16_WORD( "epr-6209.ic7", 0x000000, 0x004000, CRC(cafebabe) SHA1(abcdef0123456789abcdef0123456789abcdef01) )
ROM_END
CONS( 1988, megadriv, 0, 0, megadriv, megadriv, md_state, init_megadriv, "Sega", "Mega Drive", MACHINE_IS_BIOS_ROOT )
"""
# GAME macro WITHOUT MACHINE_IS_BIOS_ROOT (should NOT be detected)
NON_BIOS_FIXTURE = """\
ROM_START( pacman )
ROM_REGION( 0x10000, "maincpu", 0 )
ROM_LOAD( "pacman.6e", 0x0000, 0x1000, CRC(c1e6ab10) SHA1(e87e059c5be45753f7e9f33dff851f16d6751181) )
ROM_END
GAME( 1980, pacman, 0, pacman, pacman, pacman_state, init_pacman, ROT90, "Namco", "Pac-Man", MACHINE_SUPPORTS_SAVE )
"""
class TestFindBiosRootSets(unittest.TestCase):
"""Tests for find_bios_root_sets."""
def test_detects_neogeo_from_game_macro(self) -> None:
result = find_bios_root_sets(NEOGEO_FIXTURE, 'src/mame/snk/neogeo.cpp')
self.assertIn('neogeo', result)
self.assertEqual(result['neogeo']['source_file'], 'src/mame/snk/neogeo.cpp')
self.assertIsInstance(result['neogeo']['source_line'], int)
def test_detects_from_comp_macro(self) -> None:
result = find_bios_root_sets(DEVICE_FIXTURE, 'src/mame/acorn/bbc.cpp')
self.assertIn('bbcb', result)
def test_detects_from_cons_macro(self) -> None:
result = find_bios_root_sets(CONS_FIXTURE, 'src/mame/sega/megadriv.cpp')
self.assertIn('megadriv', result)
def test_ignores_non_bios_games(self) -> None:
result = find_bios_root_sets(NON_BIOS_FIXTURE, 'src/mame/pacman/pacman.cpp')
self.assertEqual(result, {})
def test_detects_from_nodump_fixture(self) -> None:
result = find_bios_root_sets(NODUMP_FIXTURE, 'test.cpp')
self.assertIn('testnd', result)
def test_detects_from_baddump_fixture(self) -> None:
result = find_bios_root_sets(BADDUMP_FIXTURE, 'test.cpp')
self.assertIn('testbd', result)
class TestParseRomBlock(unittest.TestCase):
"""Tests for parse_rom_block."""
def test_extracts_rom_names(self) -> None:
roms = parse_rom_block(NEOGEO_FIXTURE, 'neogeo')
names = [r['name'] for r in roms]
self.assertIn('sp-s2.sp1', names)
self.assertIn('vs-bios.rom', names)
self.assertIn('sm1.sm1', names)
def test_extracts_crc32_and_sha1(self) -> None:
roms = parse_rom_block(NEOGEO_FIXTURE, 'neogeo')
sp_s2 = next(r for r in roms if r['name'] == 'sp-s2.sp1')
self.assertEqual(sp_s2['crc32'], '9036d879')
self.assertEqual(sp_s2['sha1'], '4f5ed7105b7128794654ce82b51723e16e389543')
def test_extracts_size(self) -> None:
roms = parse_rom_block(NEOGEO_FIXTURE, 'neogeo')
sp_s2 = next(r for r in roms if r['name'] == 'sp-s2.sp1')
self.assertEqual(sp_s2['size'], 0x020000)
def test_extracts_bios_metadata(self) -> None:
roms = parse_rom_block(NEOGEO_FIXTURE, 'neogeo')
sp_s2 = next(r for r in roms if r['name'] == 'sp-s2.sp1')
self.assertEqual(sp_s2['bios_index'], 0)
self.assertEqual(sp_s2['bios_label'], 'euro')
self.assertEqual(sp_s2['bios_description'], 'Europe MVS (Ver. 2)')
def test_non_bios_rom_has_no_bios_fields(self) -> None:
roms = parse_rom_block(NEOGEO_FIXTURE, 'neogeo')
sm1 = next(r for r in roms if r['name'] == 'sm1.sm1')
self.assertNotIn('bios_index', sm1)
self.assertNotIn('bios_label', sm1)
def test_skips_no_dump(self) -> None:
roms = parse_rom_block(NODUMP_FIXTURE, 'testnd')
names = [r['name'] for r in roms]
self.assertIn('good.rom', names)
self.assertNotIn('missing.rom', names)
def test_includes_bad_dump_with_flag(self) -> None:
roms = parse_rom_block(BADDUMP_FIXTURE, 'testbd')
self.assertEqual(len(roms), 1)
self.assertEqual(roms[0]['name'], 'badrom.bin')
self.assertTrue(roms[0]['bad_dump'])
self.assertEqual(roms[0]['crc32'], 'deadbeef')
self.assertEqual(roms[0]['sha1'], '0123456789abcdef0123456789abcdef01234567')
def test_handles_rom_load16_word(self) -> None:
roms = parse_rom_block(CONS_FIXTURE, 'megadriv')
self.assertEqual(len(roms), 1)
self.assertEqual(roms[0]['name'], 'epr-6209.ic7')
self.assertEqual(roms[0]['crc32'], 'cafebabe')
def test_tracks_rom_region(self) -> None:
roms = parse_rom_block(NEOGEO_FIXTURE, 'neogeo')
sp_s2 = next(r for r in roms if r['name'] == 'sp-s2.sp1')
sm1 = next(r for r in roms if r['name'] == 'sm1.sm1')
self.assertEqual(sp_s2['region'], 'mainbios')
self.assertEqual(sm1['region'], 'audiocpu')
def test_returns_empty_for_unknown_set(self) -> None:
roms = parse_rom_block(NEOGEO_FIXTURE, 'nonexistent')
self.assertEqual(roms, [])
def test_good_rom_not_flagged_bad_dump(self) -> None:
roms = parse_rom_block(NODUMP_FIXTURE, 'testnd')
good = next(r for r in roms if r['name'] == 'good.rom')
self.assertFalse(good['bad_dump'])
def test_crc32_sha1_lowercase(self) -> None:
fixture = """\
ROM_START( upper )
ROM_REGION( 0x10000, "maincpu", 0 )
ROM_LOAD( "test.rom", 0x00000, 0x4000, CRC(AABBCCDD) SHA1(AABBCCDDEEFF00112233AABBCCDDEEFF00112233) )
ROM_END
"""
roms = parse_rom_block(fixture, 'upper')
self.assertEqual(roms[0]['crc32'], 'aabbccdd')
self.assertEqual(roms[0]['sha1'], 'aabbccddeeff00112233aabbccddeeff00112233')
class TestParseMameSourceTree(unittest.TestCase):
"""Tests for parse_mame_source_tree."""
def test_walks_source_tree(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
mame_dir = os.path.join(tmpdir, 'src', 'mame', 'snk')
os.makedirs(mame_dir)
filepath = os.path.join(mame_dir, 'neogeo.cpp')
with open(filepath, 'w') as f:
f.write(NEOGEO_FIXTURE)
results = parse_mame_source_tree(tmpdir)
self.assertIn('neogeo', results)
self.assertEqual(len(results['neogeo']['roms']), 3)
self.assertEqual(
results['neogeo']['source_file'],
'src/mame/snk/neogeo.cpp',
)
def test_ignores_non_source_files(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
mame_dir = os.path.join(tmpdir, 'src', 'mame')
os.makedirs(mame_dir)
# Write a .txt file that should be ignored
with open(os.path.join(mame_dir, 'notes.txt'), 'w') as f:
f.write(NEOGEO_FIXTURE)
results = parse_mame_source_tree(tmpdir)
self.assertEqual(results, {})
def test_scans_devices_dir(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
dev_dir = os.path.join(tmpdir, 'src', 'devices', 'bus')
os.makedirs(dev_dir)
with open(os.path.join(dev_dir, 'test.cpp'), 'w') as f:
f.write(DEVICE_FIXTURE)
results = parse_mame_source_tree(tmpdir)
self.assertIn('bbcb', results)
def test_empty_tree(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
results = parse_mame_source_tree(tmpdir)
self.assertEqual(results, {})
if __name__ == '__main__':
unittest.main()

352
wiki/adding-a-platform.md Normal file
View File

@@ -0,0 +1,352 @@
# Adding a platform
How to add support for a new retrogaming platform (e.g. a frontend like Batocera,
a manager like EmuDeck, or a firmware database like BizHawk).
## Prerequisites
Before starting, gather the following from the upstream project:
- **Where does it define BIOS requirements?** Each platform has a canonical source:
a DAT file, a JSON fixture, an XML manifest, a Bash script, a C# database, etc.
- **What verification mode does it use?** Read the platform source code to determine
how it checks BIOS files at runtime: file existence only (`existence`), MD5 hash
matching (`md5`), SHA1 matching (`sha1`), or a combination of size and hash.
- **What is the base destination?** The directory name where BIOS files are placed
on disk (e.g. `system` for RetroArch, `bios` for Batocera, `Firmware` for BizHawk).
- **What hash type does it store?** The primary hash format used in the platform's
own data files (SHA1 for RetroArch/BizHawk, MD5 for Batocera/Recalbox/EmuDeck).
## Step 1: Create the scraper
Scrapers live in `scripts/scraper/` and are auto-discovered by the plugin system.
Any file matching `*_scraper.py` in that directory is loaded at import time via
`pkgutil.iter_modules`. No registration step is needed beyond placing the file.
### Module contract
The module must export two names:
```python
PLATFORM_NAME = "myplatform" # matches the key in _registry.yml
class Scraper(BaseScraper):
...
```
### Inheriting BaseScraper
`BaseScraper` provides:
- `_fetch_raw() -> str` - HTTP GET with 50 MB response limit, cached after first call.
Uses `urllib.request` with a `retrobios-scraper/1.0` user-agent and 30s timeout.
- `compare_with_config(config) -> ChangeSet` - diffs scraped requirements against
an existing platform YAML, returning added/removed/modified entries.
- `test_connection() -> bool` - checks if the source URL is reachable.
Two abstract methods must be implemented:
```python
def fetch_requirements(self) -> list[BiosRequirement]:
"""Parse the upstream source and return one BiosRequirement per file."""
def validate_format(self, raw_data: str) -> bool:
"""Return False if the upstream format has changed unexpectedly."""
```
### BiosRequirement fields
| Field | Type | Description |
|-------|------|-------------|
| `name` | `str` | Filename as the platform expects it |
| `system` | `str` | Retrobios system ID (e.g. `sony-playstation`) |
| `sha1` | `str \| None` | SHA1 hash if available |
| `md5` | `str \| None` | MD5 hash if available |
| `crc32` | `str \| None` | CRC32 if available |
| `size` | `int \| None` | Expected file size in bytes |
| `destination` | `str` | Relative path within the BIOS directory |
| `required` | `bool` | Whether the platform considers this file mandatory |
| `zipped_file` | `str \| None` | If set, the hash refers to a ROM inside a ZIP |
| `native_id` | `str \| None` | Original system name before normalization |
### System ID mapping
Every scraper needs a mapping from the platform's native system identifiers to
retrobios system IDs. Define this as a module-level dict:
```python
SLUG_MAP: dict[str, str] = {
"psx": "sony-playstation",
"saturn": "sega-saturn",
...
}
```
Warn on unmapped slugs so new systems are surfaced during scraping.
### generate_platform_yaml (optional)
If the scraper defines a `generate_platform_yaml() -> dict` method, the shared
CLI will use it instead of the generic YAML builder. This allows the scraper to
include platform metadata (homepage, version, inherits, cores list) in the output.
### CLI entry point
Add a `main()` function and `__main__` guard:
```python
def main():
from scripts.scraper.base_scraper import scraper_cli
scraper_cli(Scraper, "Scrape MyPlatform BIOS requirements")
if __name__ == "__main__":
main()
```
`scraper_cli` provides `--dry-run`, `--json`, and `--output` flags automatically.
### Test the scraper
```bash
python -m scripts.scraper.myplatform_scraper --dry-run
```
This fetches from upstream and prints a summary without writing anything.
## Step 2: Register the platform
Add an entry to `platforms/_registry.yml` under the `platforms:` key.
### Required fields
```yaml
platforms:
myplatform:
config: myplatform.yml # platform YAML filename in platforms/
status: active # active or archived
scraper: myplatform # matches PLATFORM_NAME in the scraper
source_url: https://... # upstream data URL
source_format: json # json, xml, clrmamepro_dat, python_dict, bash_script+csv, csharp_firmware_database, github_component_manifests
hash_type: md5 # primary hash in the upstream data
verification_mode: md5 # how the platform checks files: existence, md5, sha1
base_destination: bios # where files go on disk
cores: # which emulator profiles apply
- core_a
- core_b
```
The `cores` field determines which emulator profiles are resolved for this platform.
Three strategies exist:
- **Explicit list**: `cores: [beetle_psx, dolphin, ...]` - match by profile key name.
Used by Batocera, Recalbox, RetroBat, RomM.
- **all_libretro**: `cores: all_libretro` - include every profile with `type: libretro`
or `type: standalone + libretro`. Used by RetroArch, Lakka, RetroPie.
- **Omitted**: fallback to system ID intersection. Used by EmuDeck.
### Optional fields
```yaml
logo: https://... # SVG or PNG for UI/docs
schedule: weekly # scrape frequency: weekly, monthly, or null
inherits_from: retroarch # inherit systems/cores from another platform
case_insensitive_fs: true # if the platform runs on case-insensitive filesystems
target_scraper: myplatform_targets # hardware target scraper name
target_source: https://... # target data source URL
install:
detect: # auto-detection for install.py
- os: linux
method: config_file
config: $HOME/.config/myplatform/config.ini
parse_key: bios_directory
```
### Inheritance
If the new platform inherits from an existing one (e.g. Lakka inherits RetroArch),
set `inherits_from` in the registry AND add `inherits: retroarch` in the platform
YAML itself. `load_platform_config()` reads the `inherits:` field from the YAML to
merge parent systems and shared groups into the child. The child YAML only needs to
declare overrides.
## Step 3: Generate the platform YAML
Run the scraper with `--output` to produce the initial platform configuration:
```bash
python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
```
If a file already exists at the output path, the CLI preserves fields that the
scraper does not generate (e.g. `data_directories`, manually added metadata).
Only the `systems` section is replaced.
Verify the result:
```bash
python scripts/verify.py --platform myplatform
python scripts/verify.py --platform myplatform --verbose
```
## Step 4: Add verification logic
Check how the platform verifies BIOS files by reading its source code.
The `verification_mode` in the registry tells `verify.py` which strategy to use:
| Mode | Behavior | Example platforms |
|------|----------|-------------------|
| `existence` | File must exist, no hash check | RetroArch, Lakka, RetroPie |
| `md5` | MD5 must match the declared hash | Batocera, Recalbox, RetroBat, EmuDeck, RetroDECK |
| `sha1` | SHA1 must match | BizHawk |
If the platform has unique verification behavior (e.g. Batocera's `checkInsideZip`,
Recalbox's multi-hash comma-separated MD5, RomM's size + any-hash), add the logic
to `verify.py` in the platform-specific verification path.
Read the platform's source code to understand its exact verification behavior before writing any logic. Batocera's `checkInsideZip` uses `casefold()` for case-insensitive matching. Recalbox supports comma-separated MD5 lists. RomM checks file size before hashing. These details matter: the project replicates native behavior, not an approximation of it.
## Step 5: Create an exporter (optional)
Exporters convert truth data back to the platform's native format. They live in
`scripts/exporter/` and follow the same auto-discovery pattern (`*_exporter.py`).
### Module contract
The module must export an `Exporter` class inheriting `BaseExporter`:
```python
from scripts.exporter.base_exporter import BaseExporter
class Exporter(BaseExporter):
@staticmethod
def platform_name() -> str:
return "myplatform"
def export(self, truth_data: dict, output_path: str, scraped_data: dict | None = None) -> None:
# Write truth_data in the platform's native format to output_path
...
def validate(self, truth_data: dict, output_path: str) -> list[str]:
# Return a list of issues (empty = valid)
...
```
`BaseExporter` provides helper methods:
- `_is_pattern(name)` - True if the filename contains wildcards or placeholders.
- `_dest(fe)` - resolve destination path from a file entry dict.
- `_display_name(sys_id, scraped_sys)` - convert a system slug to a display name.
### Round-trip validation
The exporter enables a scrape-export-compare workflow:
```bash
# Scrape upstream
python -m scripts.scraper.myplatform_scraper --output /tmp/scraped.yml
# Export truth data
python scripts/export_native.py --platform myplatform --output /tmp/exported.json
# Compare exported file with upstream
diff /tmp/scraped.yml /tmp/exported.json
```
## Step 6: Create a target scraper (optional)
Target scrapers determine which emulator cores are available on each hardware
target (e.g. which RetroArch cores exist for Switch, RPi4, or x86_64).
They live in `scripts/scraper/targets/` and are auto-discovered by filename
(`*_targets_scraper.py`).
### Module contract
```python
from scripts.scraper.targets import BaseTargetScraper
PLATFORM_NAME = "myplatform_targets"
class Scraper(BaseTargetScraper):
def fetch_targets(self) -> dict:
return {
"platform": "myplatform",
"source": "https://...",
"scraped_at": "2026-03-30T00:00:00Z",
"targets": {
"x86_64": {
"architecture": "x86_64",
"cores": ["beetle_psx", "dolphin", "..."],
},
"rpi4": {
"architecture": "aarch64",
"cores": ["pcsx_rearmed", "mgba", "..."],
},
},
}
```
Add `target_scraper` and `target_source` to the platform's registry entry.
### Overrides
Hardware-specific overrides go in `platforms/targets/_overrides.yml`. This file
defines aliases (e.g. `arm64` maps to `aarch64`) and per-platform core
additions/removals that the scraper cannot determine automatically.
### Single-target platforms
For platforms that only run on one target (e.g. RetroBat on Windows, RomM in the
browser), create a static YAML file in `platforms/targets/` instead of a scraper.
Set `target_scraper: null` in the registry.
## Step 7: Add install detection (optional)
The `install` section in `_registry.yml` tells `install.py` how to detect
the platform on the user's machine and locate its BIOS directory.
Three detection methods are available:
| Method | Description | Fields |
|--------|-------------|--------|
| `config_file` | Parse a key from a config file | `config`, `parse_key`, optionally `bios_subdir` |
| `path_exists` | Check if a directory exists | `path`, optionally `bios_path` |
| `file_exists` | Check if a file exists | `file`, optionally `bios_path` |
Each entry is scoped to an OS (`linux`, `darwin`, `windows`). Multiple entries
per OS are tried in order.
## Step 8: Validate the full pipeline
After all pieces are in place, run the full pipeline:
```bash
python scripts/pipeline.py --offline
```
This executes in sequence:
1. `generate_db.py` - rebuild `database.json` from `bios/`
2. `refresh_data_dirs.py` - update data directories
3. `verify.py --all` - verify all platforms including the new one
4. `generate_pack.py --all` - build ZIP packs
5. Consistency check - verify counts match between verify and pack
Check the output for:
- The new platform appears in verify results
- No unexpected CRITICAL or WARNING entries
- Pack generation succeeds and includes the expected files
- Consistency check passes (verify file counts match pack file counts)
Verification is not optional. A platform that passes `pipeline.py` today may break tomorrow if upstream changes its data format. Run the full pipeline on every change, even if the modification seems trivial. The consistency check (verify counts must match pack counts) catches subtle issues where files resolve during verification but fail during pack generation, or vice versa.
## Checklist
- [ ] Scraper file in `scripts/scraper/<name>_scraper.py`
- [ ] `PLATFORM_NAME` and `Scraper` class exported
- [ ] `fetch_requirements()` and `validate_format()` implemented
- [ ] System ID mapping covers all upstream systems
- [ ] Entry added to `platforms/_registry.yml`
- [ ] Platform YAML generated and verified
- [ ] `python scripts/pipeline.py --offline` passes
- [ ] Exporter in `scripts/exporter/<name>_exporter.py` (if applicable)
- [ ] Target scraper in `scripts/scraper/targets/<name>_targets_scraper.py` (if applicable)
- [ ] Install detection entries in `_registry.yml` (if applicable)

423
wiki/adding-a-scraper.md Normal file
View File

@@ -0,0 +1,423 @@
# Adding a scraper
How to create or modify a scraper for fetching BIOS requirements from upstream
platform sources.
## Scraper architecture
### Plugin discovery
Scrapers are discovered automatically at import time. The `scripts/scraper/__init__.py`
module uses `pkgutil.iter_modules` to scan for files matching `*_scraper.py` in
the scraper directory. Each module must export:
- `PLATFORM_NAME: str` - the platform identifier (matches `_registry.yml`)
- `Scraper: class` - a subclass of `BaseScraper`
No registration code is needed. Drop a file, export the two names, and it works.
```python
# scripts/scraper/__init__.py (simplified)
for finder, name, ispkg in pkgutil.iter_modules([package_dir]):
if not name.endswith("_scraper"):
continue
module = importlib.import_module(f".{name}", package=__package__)
# looks for PLATFORM_NAME and Scraper attributes
```
### BaseScraper ABC
`BaseScraper` (`scripts/scraper/base_scraper.py`) provides the foundation:
```
BaseScraper
__init__(url: str)
_fetch_raw() -> str # HTTP GET, cached, 50 MB limit
fetch_requirements() -> list # abstract: parse upstream data
validate_format(raw_data) -> bool # abstract: detect format changes
compare_with_config(config) -> ChangeSet # diff against existing YAML
test_connection() -> bool # reachability check
```
`_fetch_raw()` handles HTTP with `urllib.request`, sets a `retrobios-scraper/1.0`
user-agent, enforces a 30-second timeout, and reads the response in 64 KB chunks
with a 50 MB hard limit to prevent memory exhaustion. The result is cached on the
instance after the first call.
### BiosRequirement
A dataclass representing a single BIOS file entry:
```python
@dataclass
class BiosRequirement:
name: str # filename
system: str # retrobios system ID
sha1: str | None = None
md5: str | None = None
crc32: str | None = None
size: int | None = None
destination: str = "" # relative path in BIOS dir
required: bool = True
zipped_file: str | None = None # ROM name inside a ZIP
native_id: str | None = None # original system name
```
### ChangeSet
Returned by `compare_with_config()`. Contains:
- `added: list[BiosRequirement]` - new files not in the existing config
- `removed: list[BiosRequirement]` - files present in config but gone upstream
- `modified: list[tuple[BiosRequirement, BiosRequirement]]` - hash changes
- `has_changes: bool` - True if any of the above are non-empty
- `summary() -> str` - human-readable summary (e.g. `+3 added, ~1 modified`)
### scraper_cli
`scraper_cli(scraper_class, description)` provides a shared CLI with three modes:
| Flag | Behavior |
|------|----------|
| `--dry-run` | Fetch and print a summary grouped by system |
| `--json` | Output all requirements as JSON |
| `--output FILE` | Write platform YAML to FILE |
When `--output` targets an existing file, the CLI preserves keys not generated by
the scraper (e.g. `data_directories`, manual additions). Only the `systems` section
is replaced. If the scraper defines `generate_platform_yaml()`, that method is used
instead of the generic YAML builder.
### Helper functions
Two additional functions in `base_scraper.py`:
- `fetch_github_latest_version(repo)` - fetches the latest release tag via GitHub API.
- `fetch_github_latest_tag(repo, prefix)` - fetches the most recent tag matching
an optional prefix.
## Creating a BIOS scraper
### Minimal example
Based on the RomM scraper pattern (JSON source, flat structure):
```python
"""Scraper for MyPlatform BIOS requirements."""
from __future__ import annotations
import json
import sys
try:
from .base_scraper import BaseScraper, BiosRequirement
except ImportError:
from base_scraper import BaseScraper, BiosRequirement
PLATFORM_NAME = "myplatform"
SOURCE_URL = "https://raw.githubusercontent.com/org/repo/main/bios_list.json"
SLUG_MAP: dict[str, str] = {
"psx": "sony-playstation",
"saturn": "sega-saturn",
}
class Scraper(BaseScraper):
def __init__(self, url: str = SOURCE_URL):
super().__init__(url=url)
def fetch_requirements(self) -> list[BiosRequirement]:
raw = self._fetch_raw()
if not self.validate_format(raw):
raise ValueError("Format validation failed")
data = json.loads(raw)
requirements = []
for entry in data:
system = SLUG_MAP.get(entry["platform"])
if not system:
print(f"Warning: unmapped '{entry['platform']}'", file=sys.stderr)
continue
requirements.append(BiosRequirement(
name=entry["filename"],
system=system,
md5=entry.get("md5"),
sha1=entry.get("sha1"),
size=entry.get("size"),
destination=entry["filename"],
required=entry.get("required", True),
))
return requirements
def validate_format(self, raw_data: str) -> bool:
try:
data = json.loads(raw_data)
except (json.JSONDecodeError, TypeError):
return False
return isinstance(data, list) and len(data) > 0
def main():
from scripts.scraper.base_scraper import scraper_cli
scraper_cli(Scraper, "Scrape MyPlatform BIOS requirements")
if __name__ == "__main__":
main()
```
### Parsing different upstream formats
Each platform stores its BIOS requirements differently. The scraper's job is to
normalize them into `BiosRequirement` entries.
| Format | Example | Parsing approach |
|--------|---------|-----------------|
| JSON | RomM `known_bios_files.json` | `json.loads()`, iterate keys |
| XML | Recalbox `es_bios.xml` | `xml.etree.ElementTree`, xpath or iter |
| clrmamepro DAT | RetroArch `System.dat` | Use `dat_parser` module (see below) |
| Python dict | Batocera `batocera-systems` | `ast.literal_eval` or regex extraction |
| Bash script | EmuDeck `checkBIOS.sh` | Line-by-line regex parsing |
| C# source | BizHawk `FirmwareDatabase.cs` | Regex for method calls and string literals |
| C source | MAME/FBNeo drivers | Use `mame_parser` or `fbneo_parser` (see below) |
| JSON (GitHub API) | RetroDECK component manifests | `json.loads()` per manifest file |
### System ID mapping
Every scraper maintains a `SLUG_MAP` (or equivalent) that translates the platform's
native system identifiers to retrobios system IDs. The retrobios system ID format
is `manufacturer-console` in lowercase with hyphens (e.g. `sony-playstation`,
`sega-mega-drive`, `nintendo-gba`).
When a native slug has no mapping, print a warning to stderr. This surfaces new
systems added upstream that need to be mapped.
System ID consistency matters for cross-platform operations. The same console must use the same ID across all scrapers and platforms. Before inventing a new ID, check existing profiles and platform YAMLs for precedent. The canonical format is `manufacturer-console` in lowercase with hyphens (e.g., `sony-playstation`, `sega-mega-drive`). The `SYSTEM_ALIASES` dict in `common.py` maps common variations to canonical IDs.
### Hash normalization
- Normalize all hashes to lowercase hex strings.
- Handle missing hashes gracefully (set to `None`, not empty string).
- Some platforms provide multiple hash types per entry. Populate whichever fields
are available.
- Batocera uses 29-character truncated MD5 hashes in some entries. The resolution
layer handles prefix matching, but the scraper should store the hash as-is.
Scraped data reflects what the upstream declares, which may not match reality. The scraper's job is faithful transcription of upstream data, not correction. Corrections happen in the emulator profiles (source-verified) and in `_shared.yml` (curated). If a scraper detects an obviously wrong hash or filename, log a warning but still include the upstream value. The divergence will surface during truth diffing.
## Creating a target scraper
Target scrapers determine which emulator cores are available on each hardware
target. They live in `scripts/scraper/targets/` and follow the same auto-discovery
pattern (`*_targets_scraper.py`).
### BaseTargetScraper ABC
```
BaseTargetScraper
__init__(url: str)
fetch_targets() -> dict # abstract: return target data
write_output(data, path) # write YAML to disk
```
### Output format
`fetch_targets()` must return a dict with this structure:
```python
{
"platform": "myplatform",
"source": "https://...",
"scraped_at": "2026-03-30T12:00:00Z",
"targets": {
"x86_64": {
"architecture": "x86_64",
"cores": ["beetle_psx", "dolphin", "snes9x"],
},
"rpi4": {
"architecture": "aarch64",
"cores": ["pcsx_rearmed", "mgba"],
},
},
}
```
The `targets` dict maps target names to their available cores. Core names must
match the names used in emulator profile `cores:` fields for the target filtering
pipeline to work correctly.
### Overrides
`platforms/targets/_overrides.yml` provides post-scrape adjustments:
- **aliases**: map alternate target names to canonical ones (e.g. `arm64` -> `aarch64`)
- **add_cores**: cores present on a target but not detected by the scraper
- **remove_cores**: cores detected by the scraper but not actually functional
Overrides are applied by `load_target_config()` in `common.py` after loading
the scraped data. The scraper itself does not need to handle overrides.
### Module contract
```python
from scripts.scraper.targets import BaseTargetScraper
PLATFORM_NAME = "myplatform_targets"
class Scraper(BaseTargetScraper):
def __init__(self):
super().__init__(url="https://...")
def fetch_targets(self) -> dict:
# Fetch and parse target data
...
```
Register the target scraper in `_registry.yml`:
```yaml
myplatform:
target_scraper: myplatform_targets
target_source: https://...
```
### Existing target scrapers
| Scraper | Source | Approach |
|---------|--------|----------|
| `retroarch_targets` | libretro buildbot nightly | Scrape directory listings for each target arch |
| `batocera_targets` | Config.in + es_systems.yml | Cross-reference kernel config with system definitions |
| `emudeck_targets` | EmuScripts + RetroArch cores | GitHub API for script availability per OS |
| `retropie_targets` | scriptmodules + rp_module_flags | Parse Bash scriptmodules for platform flags |
## Parser modules
Shared parsers in `scripts/scraper/` handle formats used by multiple scrapers
or formats complex enough to warrant dedicated parsing logic.
### dat_parser
Parses clrmamepro DAT format as used in RetroArch's `System.dat`:
```
game (
name "System"
comment "Platform Name"
rom ( name filename size 12345 crc ABCD1234 md5 ... sha1 ... )
)
```
Produces `DatRom` dataclass instances with `name`, `size`, `crc32`, `md5`, `sha1`,
and `system` fields. The `libretro_scraper` uses this parser.
### mame_parser
Parses MAME C source files to extract BIOS root sets. Handles:
- Machine declaration macros: `GAME`, `SYST`, `COMP`, `CONS`
- `MACHINE_IS_BIOS_ROOT` flag detection
- `ROM_START`/`ROM_END` blocks
- `ROM_LOAD` variants and `ROM_REGION` declarations
- `ROM_SYSTEM_BIOS` entries
- `NO_DUMP` filtering and `BAD_DUMP` flagging
Used by `mame_hash_scraper` to auto-fetch BIOS hashes from MAME driver sources
for each tagged MAME version.
### fbneo_parser
Parses FBNeo C source files:
- `BurnRomInfo` structs (static ROM arrays with name, size, CRC)
- `BurnDriver` structs (driver registration with `BDF_BOARDROM` flag)
- BIOS set identification via the boardrom flag
Used by `fbneo_hash_scraper` to extract BIOS ROM definitions.
### _hash_merge
Text-based YAML patching that merges fetched hash data into emulator profiles
while preserving formatting. Two strategies:
- **MAME**: updates `bios_zip` entries with `contents` lists (name, size, CRC32)
- **FBNeo**: updates individual ROM entries grouped by `archive` field
The merge preserves fields the hash data does not generate (system, note, required)
and leaves entries not present in the hash data untouched. Uses text-level YAML
manipulation rather than load-dump to maintain human-readable formatting.
## Testing
### Development workflow
1. **Start with --dry-run**. It's helpful to preview before writing output:
```bash
python -m scripts.scraper.myplatform_scraper --dry-run
```
2. **Check JSON output** for data quality:
```bash
python -m scripts.scraper.myplatform_scraper --json | python -m json.tool | head -50
```
3. **Compare with existing YAML** if updating a scraper:
```bash
python -m scripts.scraper.myplatform_scraper --output /tmp/test.yml
diff platforms/myplatform.yml /tmp/test.yml
```
4. **Run verification** after generating:
```bash
python scripts/verify.py --platform myplatform
python scripts/verify.py --platform myplatform --verbose
```
5. **Run the full pipeline** before committing:
```bash
python scripts/pipeline.py --offline
```
### Round-trip testing
If an exporter exists for the platform, validate the scrape-export-compare cycle:
```bash
# Scrape upstream -> platform YAML
python -m scripts.scraper.myplatform_scraper --output /tmp/scraped.yml
# Export truth data -> native format
python scripts/export_native.py --platform myplatform --output /tmp/exported.json
# Compare
diff <(python -m scripts.scraper.myplatform_scraper --json | python -m json.tool) \
/tmp/exported.json
```
### Common issues
| Symptom | Cause | Fix |
|---------|-------|-----|
| Unmapped slug warnings | New system added upstream | Add mapping to `SLUG_MAP` |
| Empty requirements list | Upstream format changed | Check `validate_format()`, update parser |
| Hash mismatch in verify | Upstream updated hashes | Re-scrape and regenerate platform YAML |
| Scraper hangs | URL unreachable, no timeout | `_fetch_raw()` has 30s timeout; check URL |
| `Response exceeds 50 MB` | Upstream file grew | Investigate; may need chunked parsing |
| `validate_format` fails | Upstream restructured | Update both `validate_format` and `fetch_requirements` |
### E2E tests
The project's test suite (`tests/test_e2e.py`) covers scraper integration at the
pipeline level. When adding a new scraper, verify that the full pipeline passes:
```bash
python -m unittest tests.test_e2e
python scripts/pipeline.py --offline
```
Both must pass before the scraper is considered complete.

280
wiki/advanced-usage.md Normal file
View File

@@ -0,0 +1,280 @@
# Advanced Usage
Fine-grained control over pack generation, hardware filtering, truth analysis, and verification.
## Custom Packs
### Build from hash
Look up a single MD5 in the database:
```bash
python scripts/generate_pack.py --from-md5 d8f1206299c48946e6ec5ef96d014eaa
```
Build a pack containing only files matching hashes from a list (one MD5 per line, `#` for comments):
```bash
python scripts/generate_pack.py --platform batocera --from-md5-file missing.txt
```
This is useful when a platform reports missing files and you want to generate a targeted pack
rather than re-downloading the full archive.
### Split packs
Generate one ZIP per system instead of a single monolithic pack:
```bash
python scripts/generate_pack.py --platform retroarch --split
```
Group the split ZIPs by manufacturer (Sony, Nintendo, Sega, etc.):
```bash
python scripts/generate_pack.py --platform retroarch --split --group-by manufacturer
```
### System-specific packs
Extract only the files for a single system within a platform:
```bash
python scripts/generate_pack.py --platform retroarch --system sony-playstation
```
### Required only
Exclude optional files from the pack:
```bash
python scripts/generate_pack.py --platform batocera --required-only
```
What counts as "required" depends on the platform YAML. For existence-mode platforms
(RetroArch), the distinction comes from the `.info` file's `required` field.
For MD5-mode platforms (Batocera), all declared files are treated as required unless
explicitly marked optional.
## Hardware Target Filtering
### What targets are
A target represents a hardware architecture where a platform runs. Each architecture
has a different set of available cores. For example, the RetroArch Switch target
has fewer cores than the x86_64 target because some cores are not ported to ARM.
Target data is scraped from upstream sources (buildbot nightly listings, board configs,
scriptmodules) and stored in `platforms/targets/<platform>.yml`.
### Usage
Filter packs or verification to only include systems reachable by cores available
on the target hardware:
```bash
python scripts/generate_pack.py --platform retroarch --target switch
python scripts/generate_pack.py --all --target x86_64
python scripts/verify.py --platform batocera --target rpi4
```
When combined with `--all`, platforms that define the target are filtered. Platforms
without a target file for that name are left unfiltered (no information to exclude anything).
Platforms that have target data but not the requested target are skipped with an INFO message.
### How it works
The filtering pipeline has three stages:
1. **`load_target_config()`** reads `platforms/targets/<platform>.yml` and returns
the set of cores available on the target. Aliases from `_overrides.yml` are resolved
(e.g., `--target rpi4` may match `bcm2711` in the target file).
2. **`resolve_platform_cores()`** determines which emulator profiles are relevant
for the platform, then intersects the result with the target's core set. The
intersection uses a reverse index built from each profile's `cores:` field, so
that upstream names (e.g., `mednafen_psx` on the buildbot) map to profile keys
(e.g., `beetle_psx`).
3. **`filter_systems_by_target()`** removes platform systems where every core that
emulates them is absent from the target. Systems with no core information are kept
(benefit of the doubt). System ID normalization strips manufacturer prefixes and
separators so that `xbox` matches `microsoft-xbox`.
### List available targets
```bash
python scripts/verify.py --platform retroarch --list-targets
```
### Overrides
`platforms/targets/_overrides.yml` provides two mechanisms:
- **Aliases**: map user-facing names to internal target IDs
(e.g., `rpi4` -> `bcm2711`).
- **add/remove cores**: patch the scraped core list for a specific target
without overwriting the entire file. Useful when a core is known to work
but is not listed on the buildbot, or vice versa.
### Single-target platforms
Platforms with only one target (e.g., RetroBat with `windows`, RomM with `browser`)
treat `--target <their-only-target>` as a no-op: the output is identical to running
without `--target`.
## Truth Generation and Diffing
### What truth is
Truth data is ground truth generated from emulator profiles. It represents what each
core actually needs based on source code analysis, independent of what platform
scrapers declare. The purpose is gap analysis: finding files that platforms miss
or declare incorrectly.
### Generate truth
Build truth YAMLs from emulator profiles for a platform or all platforms:
```bash
python scripts/generate_truth.py --platform retroarch
python scripts/generate_truth.py --all --output-dir dist/truth/
```
Each truth YAML lists every system with its files, hashes, and the emulator profiles
that reference them. The output mirrors the platform YAML structure so the two can
be diffed directly.
### Diff truth vs scraped
Find divergences between generated truth and scraped platform data:
```bash
python scripts/diff_truth.py --platform retroarch
python scripts/diff_truth.py --all
```
The diff reports:
- Files present in truth but absent from the platform YAML (undeclared).
- Files present in the platform YAML but absent from truth (orphaned or from cores
not profiled yet).
- Hash mismatches between truth and platform data.
### Export to native formats
Convert truth data to the native format each platform consumes:
```bash
python scripts/export_native.py --platform batocera # Python dict (batocera-systems)
python scripts/export_native.py --platform recalbox # XML (es_bios.xml)
python scripts/export_native.py --all --output-dir dist/upstream/
```
This allows submitting corrections upstream in the format maintainers expect.
## Emulator-Level Verification
### Per-emulator checks
Verify files against a single emulator's ground truth (size, hashes, crypto):
```bash
python scripts/verify.py --emulator handy
python scripts/verify.py --emulator handy --verbose
```
Default output shows aggregate results per file: the core name and which checks apply.
With `--verbose`, each file expands to one line per core with the exact validation
parameters and source code reference:
```
lynxboot.img
handy validates size=512 crc32=0x0d973c9d [src/handy/system.h:45]
```
### Per-system checks
Aggregate verification across all cores that emulate a system:
```bash
python scripts/verify.py --system atari-lynx
```
### Standalone mode
Some cores have both libretro and standalone modes with different file requirements.
Filter to standalone-only:
```bash
python scripts/verify.py --emulator dolphin --standalone
```
### Ground truth in verbose output
The verbose report includes a coverage footer:
```
Ground truth: 142/160 files have emulator validation (88%)
```
This indicates how many files in the platform can be cross-checked against source-verified
emulator profiles. Files without ground truth rely solely on platform-level verification.
JSON output (`--json`) always includes the full per-emulator detail regardless of verbosity.
## Offline Workflow
### Full offline pipeline
Run the entire pipeline without network access:
```bash
python scripts/pipeline.py --offline
```
This skips data directory refresh, MAME/FBNeo hash fetch, and buildbot staleness checks.
All other steps (database generation, verification, pack building, consistency check,
README, site generation) run normally using cached data.
### Partial runs
Skip pack generation when you only need verification results:
```bash
python scripts/pipeline.py --offline --skip-packs
```
Skip documentation generation:
```bash
python scripts/pipeline.py --offline --skip-docs
```
### Truth pipeline
Include truth generation and diffing in the pipeline:
```bash
python scripts/pipeline.py --offline --with-truth
```
Include truth + native format export:
```bash
python scripts/pipeline.py --offline --with-export
```
### Combining flags
Flags compose freely:
```bash
python scripts/pipeline.py --offline --skip-docs --with-truth --target switch
```
This runs: database generation, verification (filtered to Switch cores), truth generation
and diff, consistency check. Packs and docs are skipped, no network access.

View File

@@ -6,16 +6,22 @@
bios/ BIOS and firmware files, organized by Manufacturer/Console/
Manufacturer/Console/ canonical files (one per unique content)
.variants/ alternate versions (different hash, same purpose)
emulators/ one YAML profile per core (285 profiles)
emulators/ one YAML profile per core/engine
platforms/ one YAML config per platform (scraped from upstream)
_shared.yml shared file groups across platforms
_registry.yml platform metadata (logos, scrapers, status)
_registry.yml platform metadata (logos, scrapers, status, install config)
_data_dirs.yml data directory definitions (Dolphin Sys, PPSSPP...)
targets/ hardware target configs + _overrides.yml
scripts/ all tooling (Python, pyyaml only dependency)
scraper/ upstream scrapers (libretro, batocera, recalbox...)
scraper/targets/ hardware target scrapers (retroarch, batocera, emudeck, retropie)
exporter/ native format exporters (batocera, recalbox, emudeck...)
install/ JSON install manifests per platform
targets/ JSON target manifests per platform (cores per architecture)
data/ cached data directories (not BIOS, fetched at build)
schemas/ JSON schemas for validation
tests/ E2E test suite with synthetic fixtures
_mame_clones.json MAME parent/clone set mappings
dist/ generated packs (gitignored)
.cache/ hash cache and large file downloads (gitignored)
```
@@ -28,11 +34,38 @@ Upstream sources Scrapers parse generate_db.py scans
batocera-systems builds database.json
es_bios.xml (recalbox) (SHA1 primary key,
core-info .info files indexes: by_md5, by_name,
by_crc32, by_path_suffix)
FirmwareDatabase.cs by_crc32, by_path_suffix)
MAME/FBNeo source
emulators/*.yml verify.py checks generate_pack.py resolves
source-verified platform-native files by hash, builds ZIP
from code verification packs per platform
truth.py generates diff_truth.py export_native.py
ground truth from compares truth vs exports to native formats
emulator profiles scraped platform (DAT, XML, JSON, Bash)
```
Pipeline runs all steps in sequence: DB, data dirs, MAME/FBNeo hashes,
verify, packs, install manifests, target manifests, consistency check,
README, site. See [tools](tools.md) for the full pipeline reference.
```mermaid
graph LR
A[generate_db] --> B[refresh_data_dirs]
B --> C[MAME/FBNeo hashes]
C --> D[verify --all]
D --> E[generate_pack --all]
E --> F[install manifests]
F --> G[target manifests]
G --> H[consistency check]
H --> I[generate_readme]
I --> J[generate_site]
style A fill:#2d333b,stroke:#adbac7,color:#adbac7
style D fill:#2d333b,stroke:#adbac7,color:#adbac7
style E fill:#2d333b,stroke:#adbac7,color:#adbac7
style J fill:#2d333b,stroke:#adbac7,color:#adbac7
```
## Three layers of data
@@ -46,12 +79,39 @@ emulators/*.yml verify.py checks generate_pack.py resolves
The pack combines platform baseline (layer 1) with core requirements (layer 3).
Neither too much (no files from unused cores) nor too few (no missing files for active cores).
The emulator's source code serves as ground truth for what files are needed,
what names they use, and what validation the emulator performs. Platform YAML
configs are scraped from upstream and are generally accurate, though they can
occasionally have gaps or stale entries. The emulator profiles complement the
platform data by documenting what the code actually loads. When the two disagree,
the profile takes precedence for pack generation: files the code needs are included
even if the platform does not declare them. Files the platform declares but no
profile references are kept as well (flagged during cross-reference), since the
upstream may cover cases not yet profiled.
```mermaid
graph TD
PY[Platform YAML<br/>scraped from upstream] --> PG[Pack generation]
EP[Emulator profiles<br/>source-verified] --> PG
SH[_shared.yml<br/>curated shared files] --> PY
SH --> EP
PG --> ZIP[ZIP pack per platform]
style PY fill:#2d333b,stroke:#adbac7,color:#adbac7
style EP fill:#2d333b,stroke:#adbac7,color:#adbac7
style SH fill:#2d333b,stroke:#adbac7,color:#adbac7
style PG fill:#2d333b,stroke:#adbac7,color:#adbac7
style ZIP fill:#2d333b,stroke:#adbac7,color:#adbac7
```
## Pack grouping
Platforms that produce identical packs are grouped automatically.
RetroArch and Lakka share the same files and `base_destination` (`system/`),
so they produce one combined pack (`RetroArch_Lakka_BIOS_Pack.zip`).
RetroPie uses `BIOS/` as base path, so it gets a separate pack.
With `--target`, the fingerprint includes target cores so platforms
with different hardware filters get separate packs.
## Storage tiers
@@ -99,6 +159,46 @@ If none exists, the platform version is kept.
| RPG Maker/ScummVM | excluded from dedup (NODEDUP) to preserve directory structure |
| `strip_components` in data dirs | flattens cache prefix to match expected path |
| case-insensitive dedup | prevents `font.rom` + `FONT.ROM` conflicts on Windows/macOS |
| frozen snapshot cores | `.info` may reflect current version while code is pinned to an old one. Only the frozen source at the pinned tag is reliable (e.g. desmume2015, mame2003) |
### File resolution chain
`resolve_local_file` in `common.py` tries each strategy in order, returning the
first match. Used by both `verify.py` and `generate_pack.py`.
```mermaid
graph TD
START([resolve_local_file]) --> S0{path_suffix<br/>exact match?}
S0 -- yes --> EXACT([exact])
S0 -- no --> S1{SHA1<br/>exact match?}
S1 -- yes --> EXACT
S1 -- no --> S2{MD5 direct<br/>or truncated?}
S2 -- yes --> MD5([md5_exact])
S2 -- no --> S3{name + aliases<br/>no MD5?}
S3 -- yes --> EXACT
S3 -- no --> S4{name + aliases<br/>md5_composite /<br/>direct MD5?}
S4 -- match --> EXACT
S4 -- name only --> HM([hash_mismatch])
S4 -- no --> S5{zippedFile<br/>inner ROM MD5?}
S5 -- yes --> ZE([zip_exact])
S5 -- no --> S6{MAME clone<br/>map lookup?}
S6 -- yes --> MC([mame_clone])
S6 -- no --> S7{data_dir<br/>cache scan?}
S7 -- yes --> DD([data_dir])
S7 -- no --> S8{agnostic<br/>fallback?}
S8 -- yes --> AG([agnostic_fallback])
S8 -- no --> NF([not_found])
style START fill:#2d333b,stroke:#adbac7,color:#adbac7
style EXACT fill:#2d333b,stroke:#adbac7,color:#adbac7
style MD5 fill:#2d333b,stroke:#adbac7,color:#adbac7
style HM fill:#2d333b,stroke:#adbac7,color:#adbac7
style ZE fill:#2d333b,stroke:#adbac7,color:#adbac7
style MC fill:#2d333b,stroke:#adbac7,color:#adbac7
style DD fill:#2d333b,stroke:#adbac7,color:#adbac7
style AG fill:#2d333b,stroke:#adbac7,color:#adbac7
style NF fill:#2d333b,stroke:#adbac7,color:#adbac7
```
## Platform inheritance
@@ -112,17 +212,36 @@ Core resolution (`resolve_platform_cores`) uses three strategies:
- `cores: [list]` - include only named profiles
- `cores:` absent - fallback to system ID intersection between platform and profiles
## Hardware target filtering
`--target TARGET` filters packs and verification by hardware (e.g. `switch`, `rpi4`, `x86_64`).
Target configs are in `platforms/targets/`. Overrides in `_overrides.yml` add aliases and
adjust core lists per target. `filter_systems_by_target` excludes systems whose cores are
not available on the target. Without `--target`, all systems are included.
## MAME clone map
`_mame_clones.json` at repo root maps MAME clone ROM names to their canonical parent.
When a clone ZIP was deduplicated, `resolve_local_file` uses this map to find the canonical file.
## Install manifests
`generate_pack.py --manifest` produces JSON manifests in `install/` for each platform.
These contain file lists with SHA1 hashes, platform detection config, and standalone copy
instructions. `install/targets/` contains per-architecture core availability.
The cross-platform installer (`install.py`) uses these manifests to auto-detect the
user's platform, filter files by hardware target, and download with SHA1 verification.
## Tests
`tests/test_e2e.py` contains 75 end-to-end tests with synthetic fixtures.
Covers: file resolution, verification, severity, cross-reference, aliases,
inheritance, shared groups, data dirs, storage tiers, HLE, launchers,
platform grouping, core resolution (3 strategies + alias exclusion).
4 test files with synthetic fixtures:
| File | Coverage |
|------|----------|
| `test_e2e.py` | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters |
| `test_mame_parser.py` | BIOS root set detection, ROM block parsing, macro expansion |
| `test_fbneo_parser.py` | BIOS set detection, ROM info parsing |
| `test_hash_merge.py` | MAME/FBNeo YAML merge, diff detection |
```bash
python -m unittest tests.test_e2e -v
@@ -132,7 +251,8 @@ python -m unittest tests.test_e2e -v
| Workflow | File | Trigger | Role |
|----------|------|---------|------|
| Build & Release | `build.yml` | `workflow_dispatch` (manual) | restore large files, build packs, deploy site, create GitHub release |
| Build & Release | `build.yml` | `workflow_dispatch` (manual) | restore large files, build packs, create GitHub release |
| Deploy Site | `deploy-site.yml` | push to main (platforms, emulators, wiki, scripts) + manual | generate site, build with MkDocs, deploy to GitHub Pages |
| PR Validation | `validate.yml` | pull request on `bios/`/`platforms/` | validate BIOS hashes, schema check, run tests, auto-label PR |
| Weekly Sync | `watch.yml` | cron (Monday 6 AM UTC) + manual | scrape upstream sources, detect changes, create update PR |

112
wiki/faq.md Normal file
View File

@@ -0,0 +1,112 @@
# FAQ - RetroBIOS
## My game shows a black screen
Most likely a missing or incorrect BIOS file. Run verification for your platform:
```bash
python scripts/verify.py --platform retroarch
```
Look for MISSING or HASH MISMATCH entries. If a file shows HASH MISMATCH, you have a BIOS file but it's the wrong version or a bad dump. Replace it with one that matches the expected hash.
Some cores also support HLE (see below), so a missing BIOS may not always be the cause. Check the emulator's logs for error messages.
## What's the difference between required and optional?
**Required** means the emulator will not start games for that system without the file. **Optional** means the emulator works without it, but with reduced accuracy or missing features (e.g., boot screen animation, wrong font rendering, or degraded audio).
In verification output, missing required files appear as CRITICAL or WARNING depending on the platform. Missing optional files appear as WARNING or INFO.
## What's HLE?
HLE (High-Level Emulation) is a software reimplementation of what the original BIOS does. Some cores can boot games without a real BIOS file by using their built-in HLE fallback. The trade-off is lower accuracy: some games may have glitches or fail to boot entirely.
When a core has HLE support, the verification tool lowers the severity of a missing BIOS to INFO. The file is still included in packs because the real BIOS gives better results.
## Why are there multiple hashes for the same file?
Two main reasons:
1. **Regional variants.** The same filename (e.g., `IPL.bin` for GameCube) exists in different versions for USA, Europe, and Japan. Each region has a different hash.
2. **Revision differences.** Console manufacturers released updated BIOS versions over time. A PlayStation SCPH-5501 BIOS differs from a SCPH-7001.
Platforms that verify by MD5 accept specific hashes. If yours doesn't match any known hash, it may be a bad dump or an uncommon revision.
## How do I know which BIOS I need?
Two approaches:
1. **Run verify.py** for your platform. It lists every expected file with its hash and status.
2. **Check the project site.** Each platform page lists all required and optional BIOS files per system.
For a specific emulator core:
```bash
python scripts/verify.py --emulator beetle_psx --verbose
```
The `--verbose` flag shows source references and expected values from the emulator's source code.
## Is this legal?
Yes. Distribution of BIOS files, firmware, and encryption keys for emulation and preservation is supported by established case law and statutory exemptions across multiple jurisdictions.
### Emulation and BIOS redistribution
- **Emulation is legal.** *Sony v. Connectix* (2000) and *Sega v. Accolade* (1992) established that creating emulators and reverse-engineering console firmware for interoperability is lawful. BIOS files are functional prerequisites for this legal activity.
- **Fair use (US, 17 USC 107).** Non-commercial redistribution of firmware for personal emulation and archival is transformative use. The files serve a different purpose (interoperability) than the original (running proprietary hardware). No commercial market exists for standalone BIOS files.
- **Fair dealing (EU, UK, Canada, Australia).** Equivalent doctrines protect research, private study, and interoperability. The EU Software Directive (2009/24/EC, Art. 5-6) explicitly permits decompilation and use for interoperability.
- **Abandonware.** The vast majority of firmware here is for discontinued hardware no longer sold, supported, or distributed by the original manufacturer. No active commercial market is harmed.
### Encryption keys (Switch prod.keys, 3DS AES keys, Wii U keys)
This is the most contested area. The legal position:
- **Keys are not copyrightable.** Encryption keys are mathematical values, not creative expression. Copyright protects original works of authorship; a 256-bit number does not meet the threshold of originality. *Bernstein v. DOJ* (1996) established that code and algorithms are protected speech, and the mere publication of numeric values cannot be restricted under copyright.
- **DMCA 1201(f) interoperability exemption.** The DMCA prohibits circumvention of technological protection measures, but Section 1201(f) explicitly permits circumvention for the purpose of achieving interoperability between programs. Emulators require these keys to decrypt and run legally purchased game software. The keys enable interoperability, not piracy.
- **Library of Congress DMCA exemptions.** The triennial rulemaking process has repeatedly expanded exemptions for video game preservation. The 2024 exemption (37 CFR 201.40) covers circumvention for preservation of software and video games, including when the original hardware is no longer available.
- **Keys derived from consumer hardware.** These keys are extracted from retail hardware owned by consumers. Once a product is sold, the manufacturer cannot indefinitely control how the purchaser uses or examines their own property. *Chamberlain v. Skylink* (2004) held that using a product in a way the manufacturer dislikes is not automatically a DMCA violation.
- **No trade secret protection.** For keys to qualify as trade secrets, the holder must take reasonable steps to maintain secrecy. Keys embedded in millions of consumer devices and widely published online do not meet this standard.
### Recent firmware (Switch 19.0.0, PS3UPDAT, PSVUPDAT)
- **Firmware updates are freely distributed.** Nintendo, Sony, and other manufacturers distribute firmware updates via CDN without authentication or purchase requirements. Redistributing freely available data does not create new legal liability.
- **Functional necessity.** Emulators require system firmware to function. Providing firmware is equivalent to providing the operating environment the software was designed to run in.
- **Yuzu context.** The Yuzu settlement (2024) concerned the emulator itself and its facilitation of piracy, not the legality of firmware or key distribution. Yuzu settled without admitting liability and the case created no binding precedent against BIOS or key redistribution.
### Summary
This project distributes BIOS files, firmware, and encryption keys for personal use, archival, and interoperability with emulation software. The legal basis rests on fair use, statutory interoperability exemptions, preservation precedent, and the non-copyrightable nature of encryption keys.
## What's a hash/checksum?
A hash is a fixed-length fingerprint computed from a file's contents. If even one byte differs, the hash changes completely. The project uses three types:
| Type | Length | Example |
|------|--------|---------|
| MD5 | 32 hex chars | `924e392ed05558ffdb115408c263dccf` |
| SHA1 | 40 hex chars | `10155d8d6e6e832d8ea1571511e40dfb15fede05` |
| CRC32 | 8 hex chars | `2F468B96` |
Different platforms use different hash types for verification. Batocera uses MD5, RetroArch checks existence only, and RomM accepts any of the three.
## Why does my verification report say UNTESTED?
UNTESTED means the file exists on disk but its hash was not confirmed against a known value. This happens on existence-mode platforms (RetroArch, Lakka, RetroPie) where the platform only checks that the file is present, without verifying its contents.
The file may still be correct. Running `verify.py --emulator <core> --verbose` shows the emulator-level ground truth, which can confirm whether the file's hash matches what the source code expects.
## Can I use BIOS from one platform on another?
Yes. BIOS files are console-specific, not platform-specific. A PlayStation BIOS works in RetroArch, Batocera, Recalbox, and any other platform that emulates PlayStation. The only differences between platforms are:
- **Where the file goes** (each platform has its own BIOS directory)
- **What filename is expected** (usually the same, occasionally different)
- **How verification works** (MD5 check vs. existence check)
The packs differ per platform because each platform declares its own set of supported systems and expected files.
## How often are packs updated?
A weekly automated sync checks upstream sources (libretro System.dat, batocera-systems, etc.) for changes. If differences are found, a pull request is created automatically. Manual releases happen as needed when new BIOS files are added or profiles are updated.

156
wiki/getting-started.md Normal file
View File

@@ -0,0 +1,156 @@
# Getting started - RetroBIOS
## What are BIOS files?
BIOS files are firmware dumps from original console hardware. Emulators need them to boot games for systems that relied on built-in software (PlayStation, Saturn, Dreamcast, etc.). Without the correct BIOS, the emulator either refuses to start the game or falls back to less accurate software emulation.
## Installation
Three ways to get BIOS files in place, from easiest to most manual.
### Option 1: install.py (recommended)
Self-contained Python script, no dependencies beyond Python 3.10+. Auto-detects your platform and BIOS directory.
```bash
python install.py
```
Override detection if needed:
```bash
python install.py --platform retroarch --dest ~/custom/bios
python install.py --check # verify existing files without downloading
python install.py --list-platforms # show supported platforms
```
The installer downloads files from GitHub releases, verifies SHA1 checksums, and places them in the correct directory.
### Option 2: download.sh (Linux/macOS)
One-liner for systems with `curl` or `wget`:
```bash
bash scripts/download.sh retroarch ~/RetroArch/system/
bash scripts/download.sh --list # show available packs
```
### Option 3: manual download
1. Go to the [releases page](https://github.com/Abdess/retrobios/releases)
2. Download the ZIP pack for your platform
3. Extract to the BIOS directory listed below
## BIOS directory by platform
### RetroArch
RetroArch uses the `system_directory` setting in `retroarch.cfg`. Default locations:
| OS | Default path |
|----|-------------|
| Windows | `%APPDATA%\RetroArch\system\` |
| Linux | `~/.config/retroarch/system/` |
| Linux (Flatpak) | `~/.var/app/org.libretro.RetroArch/config/retroarch/system/` |
| macOS | `~/Library/Application Support/RetroArch/system/` |
| Steam Deck | `~/.var/app/org.libretro.RetroArch/config/retroarch/system/` |
| Android | `/storage/emulated/0/RetroArch/system/` |
To check your actual path: open RetroArch, go to **Settings > Directory > System/BIOS**, or look for `system_directory` in `retroarch.cfg`.
### Batocera
```
/userdata/bios/
```
Accessible via network share at `\\BATOCERA\share\bios\` (Windows) or `smb://batocera/share/bios/` (macOS/Linux).
### Recalbox
```
/recalbox/share/bios/
```
Accessible via network share at `\\RECALBOX\share\bios\`.
### RetroBat
```
bios/
```
Relative to the RetroBat installation directory (e.g., `C:\RetroBat\bios\`).
### RetroDECK
```
~/.var/app/net.retrodeck.retrodeck/retrodeck/bios/
```
### EmuDeck
```
Emulation/bios/
```
Located inside your Emulation folder. On Steam Deck, typically `~/Emulation/bios/`.
### Lakka
```
/storage/system/
```
Accessible via SSH or Samba.
### RetroPie
```
~/RetroPie/BIOS/
```
### BizHawk
```
Firmware/
```
Relative to the BizHawk installation directory.
### RomM
BIOS files are managed through the RomM web interface. Check the
[RomM documentation](https://github.com/rommapp/romm) for setup details.
## Verifying your setup
After placing BIOS files, verify that everything is correct:
```bash
python scripts/verify.py --platform retroarch
python scripts/verify.py --platform batocera
python scripts/verify.py --platform recalbox
```
The output shows each expected file with its status: OK, MISSING, or HASH MISMATCH. Platforms that verify by MD5 (Batocera, Recalbox, EmuDeck) will catch wrong versions. RetroArch only checks that files exist.
For a single system:
```bash
python scripts/verify.py --system sony-playstation
```
For a single emulator core:
```bash
python scripts/verify.py --emulator beetle_psx
```
See [Tools](tools.md) for the full CLI reference.
## Next steps
- [FAQ](faq.md) - common questions and troubleshooting
- [Tools](tools.md) - all available scripts and options
- [Architecture](architecture.md) - how the project works internally

View File

@@ -2,18 +2,50 @@
Technical documentation for the RetroBIOS toolchain.
## Pages
- **[Architecture](architecture.md)** - directory structure, data flow, platform inheritance, pack grouping, security, edge cases, CI workflows
- **[Tools](tools.md)** - CLI reference for every script, pipeline usage, scrapers
- **[Profiling guide](profiling.md)** - how to create an emulator profile from source code, step by step, with YAML field reference
- **[Data model](data-model.md)** - database.json structure, indexes, file resolution order, YAML formats
## For users
- **[Getting started](getting-started.md)** - installation, BIOS directory paths per platform, verification
- **[FAQ](faq.md)** - common questions, troubleshooting, hash explanations
If you just want to download BIOS packs, see the [home page](../index.md).
## Technical reference
- **[Architecture](architecture.md)** - directory structure, data flow, platform inheritance, pack grouping, security, edge cases, CI workflows
- **[Tools](tools.md)** - CLI reference for every script, pipeline usage, scrapers
- **[Advanced usage](advanced-usage.md)** - custom packs, target filtering, truth generation, emulator verification, offline workflow
- **[Verification modes](verification-modes.md)** - how each platform verifies BIOS files, severity matrix, resolution chain
- **[Data model](data-model.md)** - database.json structure, indexes, file resolution order, YAML formats
- **[Troubleshooting](troubleshooting.md)** - diagnosis by symptom: missing BIOS, hash mismatch, pack issues, verify errors
## For contributors
Start with the [profiling guide](profiling.md) to understand how emulator profiles are built,
then see [contributing](../contributing.md) for submission guidelines.
- **[Profiling guide](profiling.md)** - create an emulator profile from source code, YAML field reference
- **[Adding a platform](adding-a-platform.md)** - scraper, registry, YAML config, exporter, target scraper, install detection
- **[Adding a scraper](adding-a-scraper.md)** - plugin architecture, BaseScraper, parsers, target scrapers
- **[Testing guide](testing-guide.md)** - run tests, fixture pattern, how to add tests, CI integration
- **[Release process](release-process.md)** - CI workflows, large files, manual release
See [contributing](../contributing.md) for submission guidelines.
## Glossary
- **BIOS** - firmware burned into console hardware, needed by emulators that rely on original boot code
- **firmware** - system software loaded by a console at boot; used interchangeably with BIOS in this project
- **HLE** - High-Level Emulation; software reimplementation of BIOS functions, avoids needing the original file
- **hash** - fixed-length fingerprint of a file's contents; this project uses MD5, SHA1, SHA256, and CRC32
- **platform** - a distribution that packages emulators (RetroArch, Batocera, Recalbox, EmuDeck, etc.)
- **core** - an emulator packaged as a libretro plugin, loaded by RetroArch or compatible frontends
- **profile** - a YAML file in `emulators/` documenting one core's BIOS requirements, verified against source code
- **system** - a game console or computer being emulated (e.g. sony-playstation, nintendo-gameboy-advance)
- **pack** - a ZIP archive containing all BIOS files needed by a specific platform
- **ground truth** - the emulator's source code, treated as the authoritative reference for BIOS requirements
- **cross-reference** - comparison of emulator profiles against platform configs to find undeclared files
- **scraper** - a script that fetches BIOS requirement data from an upstream source (System.dat, es_bios.xml, etc.)
- **exporter** - a script that converts ground truth data back into a platform's native format
- **target** - a hardware architecture that a platform runs on (e.g. switch, rpi4, x86_64, steamos)
- **variant** - an alternative version of a BIOS file (different revision, region, or dump), stored in `.variants/`
- **required** - a file the core needs to function; determined by source code behavior
- **optional** - a file the core functions without, possibly with reduced accuracy or missing features
- **hle_fallback** - flag on a file indicating the core has an HLE path; absence is downgraded to INFO severity
- **severity** - the urgency of a verification result: OK (verified), INFO (negligible), WARNING (degraded), CRITICAL (broken)

View File

@@ -9,6 +9,34 @@ The source code is the reference because it reflects actual behavior.
Documentation, .info files, and wikis are useful starting points
but are verified against the code.
### Source hierarchy
Documentation and metadata are valuable starting points, but they can
fall out of sync with the actual code over time. The desmume2015 .info
file is a good illustration: it declares `firmware_count=3`, but the
source code at the pinned version opens zero firmware files. Cross-checking
against the source helps catch that kind of gap early.
When sources conflict, priority follows the chain of actual execution:
1. **Original emulator source** (ground truth, what the code actually does)
2. **Libretro port** (may adapt paths, add compatibility shims, or drop features)
3. **.info metadata** (declarative, may be outdated or copied from another core)
For standalone emulators like BizHawk or amiberry, there is only one
level. The emulator's own codebase is the single source of truth. No
.info, no wrapper, no divergence to track.
A note on libretro port differences: the most common change is path
resolution. The upstream emulator loads files from the current working
directory; the libretro wrapper redirects to `retro_system_directory`.
This is normal adaptation, not a divergence worth documenting. Similarly,
filename changes like `naomi2_eeprom.bin` becoming `n2_eeprom.bin` are
often deliberate. RetroArch uses a single shared system directory for
all cores, so the port renames files to prevent collisions between cores
that emulate different systems but happen to use the same generic
filenames. The upstream name goes in `aliases:`.
## Steps
### 1. Find the source code
@@ -21,9 +49,27 @@ Check these locations in order:
Always clone both upstream and libretro port to compare.
For libretro cores, cloning both repositories and diffing them reveals
what the port changed. Path changes (fopen of a relative path becoming
a system_dir lookup) are expected. What matters are file additions the
port introduces, files the port dropped, or hash values that differ
between the two codebases.
If the source is hosted outside GitHub, it's worth exploring further. Emulator
source on GitLab, Codeberg, SourceForge, Bitbucket, archive.org
snapshots, and community mirror tarballs. Inspecting copyright headers
or license strings in the libretro fork often points to the original
author's site. The upstream code exists somewhere; it's worth continuing the search before concluding the source is unavailable.
One thing worth noting: even when the same repository was analyzed for
a related profile (e.g., fbneo for arcade systems), it helps to do a
fresh pass for each new profile. When fbneo_neogeo was profiled, the
NeoGeo subset referenced BIOS files that the main arcade analysis
hadn't encountered. A fresh look avoids carrying over blind spots.
### 2. Trace file loading
Read the code flow. Don't grep keywords by assumption.
Read the code flow, tracing from the entry point.
Each emulator has its own way of loading files.
Look for:
@@ -34,6 +80,19 @@ Look for:
- Hash validation (MD5, CRC32, SHA1 comparisons in code)
- Size validation (`fseek`/`ftell`, `stat`, fixed buffer sizes)
Grepping for "bios" or "firmware" across the source tree can be a
useful first pass, but it may miss emulators that use different terms
(bootrom, system ROM, IPL, program.rom) and can surface false matches
from test fixtures or comments.
A more reliable approach is starting from the entry point
(`retro_load_game` for libretro, `main()` for standalone) and tracing
the actual file-open calls forward. Each emulator has its own loading
flow. Dolphin loads region-specific IPL files through a boot sequence
object. BlastEm reads a list of ROM paths from a configuration
structure. same_cdi opens CD-i BIOS files through a machine
initialization routine. The loading flow varies widely between emulators.
### 3. Determine required vs optional
This is decided by code behavior, not by judgment:
@@ -42,6 +101,18 @@ This is decided by code behavior, not by judgment:
- **optional**: the core works with degraded functionality without it
- **hle_fallback: true**: the core has a high-level emulation path when the file is missing
The decision is based on the code's behavior. If the core crashes or
refuses to boot without the file, it is required. If it continues with
degraded functionality (missing boot animation, different fonts, reduced
audio in menus), it is optional. This keeps the classification objective
and consistent across all profiles.
When a core has HLE (high-level emulation), the real BIOS typically
gives better accuracy, but the core functions without it. These files
are marked with `hle_fallback: true` and `required: false`. The file
still ships in packs (better experience for the user), but its absence
does not raise alarms during verification.
### 4. Document divergences
When the libretro port differs from the upstream:
@@ -54,6 +125,18 @@ Path differences (current dir vs system_dir) are normal adaptation,
not a divergence. Name changes (e.g. `naomi2_` to `n2_`) may be intentional
to avoid conflicts in the shared system directory.
RetroArch's system directory is shared by every installed core. When
the libretro port renames a file, it is usually solving a real problem:
two cores that both expect `bios.rom` would overwrite each other. The
upstream name goes in `aliases:` and `mode: libretro` on the port-specific
name, so both names are indexed.
True divergences worth documenting are: files the port adds that the
upstream never loads, files the upstream loads that the port dropped
(a gap in the port), and hash differences in embedded ROM data between
the two codebases. These get noted in the profile because they affect
what the user actually needs to provide.
### 5. Write the YAML profile
```yaml
@@ -80,6 +163,46 @@ files:
source_ref: Source/Core/Core/Boot/Boot_BS2Emu.cpp:42
```
### Writing style
Notes in a profile describe what the core does, kept focused on:
what files get loaded, how, and from where. Comparisons with other
cores, disclaimers, and feature coverage beyond file requirements
belong in external documentation. The profile is a technical spec.
Profiles are standalone documentation. Someone should be able to take
a single YAML file and integrate it into their own project without
knowing anything about this repository's database, directory layout,
or naming conventions. The YAML documents what the emulator expects.
The tooling resolves the YAML against the local file collection
separately.
A few field conventions that protect the toolchain:
- `type:` is operational. `resolve_platform_cores()` uses it to filter
which profiles apply to a platform. Valid values are `libretro`,
`standalone + libretro`, `standalone`, `alias`, `launcher`, `game`,
`utility`, `test`. Putting a classification concept here (like
"bizhawk-native") breaks the filtering. A BizHawk core is
`type: standalone`.
- `core_classification:` is descriptive. It documents the relationship
between the core and the original emulator (pure_libretro,
official_port, community_fork, frozen_snapshot, etc.). It has no
effect on tooling behavior.
- Alternative filenames go in `aliases:` on the file entry (rather than
as separate entries in platform YAMLs or `_shared.yml`). When the same
physical ROM is known by three names across different platforms, one
name is `name:` and the rest are `aliases:`.
- Hashes come from source code. If the source has a hardcoded hex
string (like emuscv's `635a978...` in memory.cpp), that goes in. If
the source embeds ROM data as byte arrays (like ep128emu's roms.hpp),
the bytes can be extracted and hashed. If the source performs no hash
check at all, the hash is omitted from the profile. The .info or docs
may list an MD5, but source confirmation makes it more reliable.
### 6. Validate
```bash
@@ -87,6 +210,38 @@ python scripts/cross_reference.py --emulator dolphin --json
python scripts/verify.py --emulator dolphin
```
### Lessons learned
These are patterns that have come up while building profiles. Sharing
them here in case they save time.
**.info metadata can lag behind the code.** The desmume2015 .info
declares `firmware_count=3`, but the core source at the pinned version
never opens any firmware file. The .info is useful as a starting point
but benefits from a cross-check against the actual code.
**Fresh analysis per profile helps.** When fbneo was profiled for
arcade systems, NeoGeo-specific BIOS files were outside the analysis
scope. Profiling fbneo_neogeo later surfaced files the first pass
hadn't covered. Doing a fresh pass for each profile, even on a
familiar codebase, avoids carrying over blind spots.
**Path adaptation vs real divergence.** The libretro wrapper changing
`fopen("./rom.bin")` to load from `system_dir` is the standard
porting pattern. The file is the same; only the directory resolution
changed. True divergences (added/removed files, different embedded
data) are the ones worth documenting.
**Each core has its own loading logic.** snes9x and bsnes both
emulate the Super Nintendo, but they handle the Super Game Boy BIOS
and DSP firmware through different code paths. Checking the actual
code for each core avoids assumptions based on a related profile.
**Code over docs.** Wiki pages and README files sometimes reference
files from older versions or a different fork. If the source code
does not load a particular file, it can be left out of the profile
even if documentation mentions it.
## YAML field reference
### Profile fields
@@ -94,18 +249,22 @@ python scripts/verify.py --emulator dolphin
| Field | Required | Description |
|-------|----------|-------------|
| `emulator` | yes | display name |
| `type` | yes | `libretro`, `standalone`, `standalone + libretro`, `alias`, `launcher` |
| `type` | yes | `libretro`, `standalone`, `standalone + libretro`, `alias`, `launcher`, `game`, `utility`, `test` |
| `core_classification` | no | `pure_libretro`, `official_port`, `community_fork`, `frozen_snapshot`, `enhanced_fork`, `game_engine`, `embedded_hle`, `alias`, `launcher` |
| `source` | yes | libretro core repository URL |
| `upstream` | no | original emulator repository URL |
| `profiled_date` | yes | date of source analysis |
| `core_version` | yes | version analyzed |
| `display_name` | no | full display name (e.g. "Sega - Mega Drive (BlastEm)") |
| `systems` | yes | list of system IDs this core handles |
| `cores` | no | list of core names (default: profile filename) |
| `cores` | no | list of upstream core names for buildbot/target matching |
| `mode` | no | default mode: `standalone`, `libretro`, or `both` |
| `verification` | no | how the core verifies BIOS: `existence` or `md5` |
| `files` | yes | list of file entries |
| `notes` | no | free-form technical notes |
| `exclusion_note` | no | why the profile has no files |
| `data_directories` | no | references to data dirs in `_data_dirs.yml` |
| `exclusion_note` | no | why the profile has no files despite .info declaring firmware |
| `analysis` | no | structured per-subsystem analysis (capabilities, supported modes) |
| `platform_details` | no | per-system platform-specific details (paths, romsets, forced systems) |
### File entry fields
@@ -113,20 +272,20 @@ python scripts/verify.py --emulator dolphin
|-------|-------------|
| `name` | filename as the core expects it |
| `required` | true if the core needs this file to function |
| `system` | system ID this file belongs to |
| `system` | system ID this file belongs to (for multi-system profiles) |
| `size` | expected size in bytes |
| `min_size`, `max_size` | size range when the code accepts a range |
| `md5`, `sha1`, `crc32`, `sha256` | expected hashes from source code |
| `validation` | list of checks the code performs: `size`, `crc32`, `md5`, `sha1` |
| `validation` | checks the code performs: `size`, `crc32`, `md5`, `sha1`, `adler32`, `signature`, `crypto`. Can be a list or dict `{core: [...], upstream: [...]}` for divergent checks |
| `aliases` | alternate filenames for the same file |
| `mode` | `libretro`, `standalone`, or `both` |
| `hle_fallback` | true if a high-level emulation path exists |
| `category` | `bios` (default), `game_data`, `bios_zip` |
| `region` | geographic region (e.g. `north-america`, `japan`) |
| `source_ref` | source file and line number |
| `path` | path relative to system directory |
| `source_ref` | source file and line number (e.g. `boot.cpp:42`) |
| `path` | destination path relative to system directory |
| `description` | what this file is |
| `note` | additional context |
| `archive` | parent ZIP if this file is inside an archive |
| `contents` | structure of files inside a BIOS ZIP |
| `storage` | `embedded` (default), `external`, `user_provided` |
| `contents` | structure of files inside a BIOS ZIP (`name`, `description`, `size`, `crc32`) |
| `storage` | `large_file` for files > 50 MB stored as release assets |

158
wiki/release-process.md Normal file
View File

@@ -0,0 +1,158 @@
# Release Process
This page documents the CI/CD pipeline: what each workflow does, how releases
are built, and how to run the process manually.
## CI workflows overview
The project uses 4 GitHub Actions workflows. All use only official GitHub
actions (`actions/checkout`, `actions/setup-python`, `actions/upload-pages-artifact`,
`actions/deploy-pages`). No third-party actions.
Budget target: ~175 minutes/month on the GitHub free tier.
| Workflow | File | Trigger |
|----------|------|---------|
| Build & Release | `build.yml` | Push to `bios/**` or `platforms/**`, manual dispatch |
| Deploy Site | `deploy-site.yml` | Push to main (platforms, emulators, wiki, scripts, database.json, mkdocs.yml), manual |
| PR Validation | `validate.yml` | PR touching `bios/**` or `platforms/**` |
| Weekly Sync | `watch.yml` | Cron Monday 06:00 UTC, manual dispatch |
## build.yml - Build & Release
Currently disabled (`if: false` on the release job) until pack generation is
validated in production.
**Trigger.** Push to `main` on `bios/**` or `platforms/**` paths, or manual
`workflow_dispatch` with optional `force_release` flag to bypass rate limiting.
**Concurrency.** Group `build`, cancel in-progress.
**Steps:**
1. Checkout, Python 3.12, install `pyyaml`
2. Run `test_e2e`
3. Rate limit check: skip if last release was less than 7 days ago (unless
`force_release` is set)
4. Restore large files from the `large-files` release into `.cache/large/`
5. Refresh data directories (`refresh_data_dirs.py`)
6. Build packs (`generate_pack.py --all --output-dir dist/`)
7. Create GitHub release with tag `v{YYYY.MM.DD}` (appends `.N` suffix if
a same-day release already exists)
8. Clean up old releases, keeping the 3 most recent plus `large-files`
**Release notes** include file count, total size, per-pack sizes, and the last
15 non-merge commits touching `bios/` or `platforms/`.
## deploy-site.yml - Deploy Documentation Site
**Trigger.** Push to `main` when any of these paths change: `platforms/`,
`emulators/`, `wiki/`, `scripts/generate_site.py`, `scripts/generate_readme.py`,
`scripts/verify.py`, `scripts/common.py`, `database.json`, `mkdocs.yml`.
Also manual dispatch.
**Steps:**
1. Checkout, Python 3.12
2. Install `pyyaml`, `mkdocs-material`, `pymdown-extensions`
3. Run `generate_site.py` (converts YAML data into MkDocs pages)
4. Run `generate_readme.py` (rebuilds README.md and CONTRIBUTING.md)
5. `mkdocs build` to produce the static site
6. Upload artifact, deploy to GitHub Pages
The site is deployed via the `github-pages` environment using the official
`actions/deploy-pages` action.
## validate.yml - PR Validation
**Trigger.** Pull requests that modify `bios/**` or `platforms/**`.
**Concurrency.** Per-PR group, cancel in-progress.
Four parallel jobs:
**validate-bios.** Diffs the PR to find changed BIOS files, runs
`validate_pr.py --markdown` on each, and posts the validation report as a PR
comment (hash verification, database match status).
**validate-configs.** Validates all platform YAML files against
`schemas/platform.schema.json` using `jsonschema`. Fails if any config does
not match the schema.
**run-tests.** Runs `python -m unittest tests.test_e2e -v`. Must pass before
merge.
**label-pr.** Auto-labels the PR based on changed paths:
| Path pattern | Label |
|-------------|-------|
| `bios/` | `bios` |
| `bios/{Manufacturer}/` | `system:{manufacturer}` |
| `platforms/` | `platform-config` |
| `scripts/` | `automation` |
## watch.yml - Weekly Platform Sync
**Trigger.** Cron schedule every Monday at 06:00 UTC, or manual dispatch.
**Flow:**
1. Scrape live upstream sources (System.dat, batocera-systems, es_bios.xml,
etc.) and regenerate platform YAML configs
2. Auto-fetch missing BIOS files
3. Refresh data directories
4. Run dedup
5. Regenerate `database.json`
6. Create or update a PR with labels `automated` and `platform-update`
The PR contains all changes from the scrape cycle. A maintainer reviews and
merges.
## Large files management
Files larger than 50 MB are stored as assets on a permanent GitHub release
named `large-files` (to keep the git repository lightweight).
Known large files: PS3UPDAT.PUP, PSVUPDAT.PUP, PSP2UPDAT.PUP, dsi_nand.bin,
maclc3.zip, Firmware.19.0.0.zip (Switch).
**Storage.** Listed in `.gitignore` so they stay out of git history. The
`large-files` release is excluded from cleanup (the build workflow only
deletes version-tagged releases).
**Build-time restore.** The build workflow downloads all assets from
`large-files` into `.cache/large/` and copies them to their expected paths
before pack generation.
**Upload.** To add or update a large file:
```bash
gh release upload large-files "bios/Sony/PS3/PS3UPDAT.PUP#PS3UPDAT.PUP"
```
**Local cache.** `generate_pack.py` calls `fetch_large_file()` which downloads
from the release and caches in `.cache/large/` for subsequent runs.
## Manual release process
When `build.yml` is disabled, build and release manually:
```bash
# Run the full pipeline (DB + verify + packs + consistency check)
python scripts/pipeline.py --offline
# Or step by step:
python scripts/generate_db.py --force --bios-dir bios --output database.json
python scripts/verify.py --all
python scripts/generate_pack.py --all --output-dir dist/
# Create the release
DATE=$(date +%Y.%m.%d)
gh release create "v${DATE}" dist/*.zip \
--title "BIOS Pack v${DATE}" \
--notes "Release notes here" \
--latest
```
To re-enable automated releases, remove the `if: false` guard from the
`release` job in `build.yml`.

148
wiki/testing-guide.md Normal file
View File

@@ -0,0 +1,148 @@
# Testing Guide
This page covers how to run, understand, and extend the test suite.
All tests use synthetic fixtures. No real BIOS files, platform configs, or
network access required.
## Running tests
Run a single test module:
```bash
python -m unittest tests.test_e2e -v
python -m unittest tests.test_mame_parser -v
python -m unittest tests.test_fbneo_parser -v
python -m unittest tests.test_hash_merge -v
```
Run the full suite:
```bash
python -m unittest discover tests -v
```
The only dependency is `pyyaml`. No test framework beyond the standard
library `unittest` module.
## Test architecture
### test_e2e.py
The main regression suite. A single `TestE2E` class exercises every code path
through the resolution, verification, pack generation, and cross-reference
logic.
**Fixture pattern.** `setUp` creates a temporary directory tree with:
- Fake BIOS files (deterministic content for hash computation)
- Platform YAML configs (existence mode, MD5 mode, inheritance, shared groups)
- Emulator profile YAMLs (required/optional files, aliases, HLE, standalone)
- A synthetic `database.json` keyed by SHA1
`tearDown` removes the temporary tree.
**Test numbering.** Tests are grouped by category:
| Range | Category |
|-------|----------|
| `test_01`--`test_14` | File resolution (SHA1, MD5, name, alias, truncated MD5, composite, zip contents, variants, hash mismatch) |
| `test_20`--`test_31` | Verification (existence mode, MD5 mode, required/optional severity, zipped file, multi-hash) |
| `test_40`--`test_47` | Cross-reference (undeclared files, standalone skip, alias profiles, data dir suppression, exclusion notes) |
| `test_50`+ | Platform config (inheritance, shared groups, data directories, grouping, core resolution, target filtering, ground truth) |
Each test calls the same functions that `verify.py` and `generate_pack.py` use
in production, against the synthetic fixtures.
### Parser tests
**test_mame_parser.** Tests the MAME C source parser that extracts BIOS root
sets from driver files. Fixtures are inline C source snippets containing
`ROM_START`, `ROM_LOAD`, `GAME()`/`COMP()` macros with
`MACHINE_IS_BIOS_ROOT`. Tests cover:
- Standard `GAME` macro detection
- `COMP` macro detection
- `ROM_LOAD` / `ROMX_LOAD` parsing (name, size, CRC32, SHA1)
- `ROM_SYSTEM_BIOS` variant extraction
- Multi-region ROM blocks
- Macro expansion and edge cases
**test_fbneo_parser.** Tests the FBNeo C source parser that identifies
`BDF_BOARDROM` sets. Same inline fixture approach.
**test_hash_merge.** Tests the text-based YAML patching module used to merge
upstream BIOS hashes into emulator profiles. Covers:
- Merge operations (add new hashes, update existing)
- Diff computation (detect what changed)
- Formatting preservation (comments, ordering, flow style)
Fixtures are programmatically generated YAML/JSON files written to a temp
directory.
## How to add a test
1. **Pick the right category.** Find the number range that matches the
subsystem you are testing. If none fits, start a new range after the last
existing one.
2. **Create synthetic fixtures.** Write the minimum YAML configs and fake
files needed to isolate the behavior. Use `tempfile.mkdtemp` for a clean
workspace. Avoid depending on the repo's real `bios/` or `platforms/`
directories.
3. **Call production functions.** Import from `common`, `verify`, `validation`,
or `truth` and call the same entry points that the CLI scripts use. Do not
re-implement logic in tests.
4. **Assert specific outcomes.** Check `Status`, `Severity`, resolution
method, file counts, or pack contents. Avoid brittle assertions on log
output or formatting.
5. **Run the full suite.** After adding your test, run `python -m unittest
discover tests -v` to verify nothing else broke.
Example skeleton:
```python
def test_42_my_new_behavior(self):
# Write minimal fixtures to self.root
profile = {"emulator": "test_core", "files": [...]}
with open(os.path.join(self.emulators_dir, "test_core.yml"), "w") as f:
yaml.dump(profile, f)
# Call production code
result = verify_platform(self.config, self.db, ...)
# Assert specific outcomes
self.assertEqual(result[0]["status"], Status.OK)
```
## Verification discipline
The test suite is one layer of verification. The full quality gate is:
1. All unit tests pass (`python -m unittest discover tests`)
2. The full pipeline completes without error (`python scripts/pipeline.py --offline`)
3. No unexpected CRITICAL entries in the verify output
4. Pack file counts match verification file counts (consistency check)
If a change passes tests but breaks the pipeline, it's worth investigating before merging. Similarly, new CRITICAL entries in the verify output after a change usually indicate something to look into. The pipeline is designed so that all steps agree: if verify reports N files for a platform, the pack should contain exactly N files.
Ideally, tests, code, and documentation ship together. When profiles and platform configs are involved, updating them in the same change helps keep everything in sync.
## CI integration
The `validate.yml` workflow runs `test_e2e` on every pull request that touches
`bios/` or `platforms/` files. The test job (`run-tests`) runs in parallel
with BIOS validation, schema validation, and auto-labeling.
Tests must pass before merge. If a test fails in CI, reproduce locally with:
```bash
python -m unittest tests.test_e2e -v 2>&1 | head -50
```
The `build.yml` workflow also runs the test suite before building release
packs.

View File

@@ -7,11 +7,35 @@ All tools are Python scripts in `scripts/`. Single dependency: `pyyaml`.
Run everything in sequence:
```bash
python scripts/pipeline.py --offline # DB + verify + packs + readme + site
python scripts/pipeline.py --offline # DB + verify + packs + manifests + readme + site
python scripts/pipeline.py --offline --skip-packs # DB + verify only
python scripts/pipeline.py --skip-docs # skip readme + site generation
python scripts/pipeline.py --offline --skip-docs # skip readme + site generation
python scripts/pipeline.py --offline --target switch # filter by hardware target
python scripts/pipeline.py --offline --with-truth # include truth generation + diff
python scripts/pipeline.py --offline --with-export # include native format export
python scripts/pipeline.py --check-buildbot # check buildbot data freshness
```
Pipeline steps:
| Step | Description | Skipped by |
|------|-------------|------------|
| 1/9 | Generate database | - |
| 2/9 | Refresh data directories | `--offline` |
| 2a | Refresh MAME BIOS hashes | `--offline` |
| 2a2 | Refresh FBNeo BIOS hashes | `--offline` |
| 2b | Check buildbot staleness | only with `--check-buildbot` |
| 2c | Generate truth YAMLs | only with `--with-truth` / `--with-export` |
| 2d | Diff truth vs scraped | only with `--with-truth` / `--with-export` |
| 2e | Export native formats | only with `--with-export` |
| 3/9 | Verify all platforms | - |
| 4/9 | Generate packs | `--skip-packs` |
| 4b | Generate install manifests | `--skip-packs` |
| 4c | Generate target manifests | `--skip-packs` |
| 5/9 | Consistency check | if verify or pack skipped |
| 8/9 | Generate README | `--skip-docs` |
| 9/9 | Generate site | `--skip-docs` |
## Individual tools
### generate_db.py
@@ -29,10 +53,16 @@ python scripts/generate_db.py --force --bios-dir bios --output database.json
Check BIOS coverage for each platform using its native verification mode.
```bash
python scripts/verify.py --all # all platforms
python scripts/verify.py --platform batocera # single platform
python scripts/verify.py --emulator dolphin # single emulator
python scripts/verify.py --system atari-lynx # single system
python scripts/verify.py --all # all platforms
python scripts/verify.py --platform batocera # single platform
python scripts/verify.py --platform retroarch --verbose # with ground truth details
python scripts/verify.py --emulator dolphin # single emulator
python scripts/verify.py --emulator dolphin --standalone # standalone mode only
python scripts/verify.py --system atari-lynx # single system
python scripts/verify.py --platform retroarch --target switch # filter by hardware
python scripts/verify.py --list-emulators # list all emulators
python scripts/verify.py --list-systems # list all systems
python scripts/verify.py --platform retroarch --list-targets # list available targets
```
Verification modes per platform:
@@ -45,6 +75,7 @@ Verification modes per platform:
| EmuDeck | md5 | MD5 whitelist per system |
| RetroDECK | md5 | MD5 per file via component manifests |
| RomM | md5 | size + any hash (MD5/SHA1/CRC32) |
| BizHawk | sha1 | SHA1 per firmware from FirmwareDatabase.cs |
### generate_pack.py
@@ -67,6 +98,14 @@ python scripts/generate_pack.py --platform retroarch --split --group-by manufact
python scripts/generate_pack.py --from-md5 d8f1206299c48946e6ec5ef96d014eaa
python scripts/generate_pack.py --platform batocera --from-md5-file missing.txt
python scripts/generate_pack.py --platform retroarch --list-systems
# Hardware target filtering
python scripts/generate_pack.py --all --target x86_64
python scripts/generate_pack.py --platform retroarch --target switch
# Install manifests (consumed by install.py)
python scripts/generate_pack.py --all --manifest --output-dir install/
python scripts/generate_pack.py --manifest-targets --output-dir install/targets/
```
Packs include platform baseline files plus files required by the platform's cores.
@@ -82,17 +121,45 @@ If none exists, the platform version is kept and the discrepancy is reported.
- `--split --group-by manufacturer`: group split packs by manufacturer (Sony, Nintendo, Sega...)
- `--from-md5`: look up a hash in the database, or build a custom pack with `--platform`/`--emulator`
- `--from-md5-file`: same, reading hashes from a file (one per line, comments with #)
- `--target`: filter by hardware target (e.g. `switch`, `rpi4`, `x86_64`)
### cross_reference.py
Compare emulator profiles against platform configs.
Reports files that cores need but platforms don't declare.
Reports files that cores need beyond what platforms declare.
```bash
python scripts/cross_reference.py # all
python scripts/cross_reference.py --emulator dolphin # single
python scripts/cross_reference.py --emulator dolphin # single
python scripts/cross_reference.py --emulator dolphin --json # JSON output
```
### truth.py, generate_truth.py, diff_truth.py
Generate ground truth from emulator profiles, diff against scraped platform data.
```bash
python scripts/generate_truth.py --platform retroarch # single platform truth
python scripts/generate_truth.py --all --output-dir dist/truth/ # all platforms
python scripts/diff_truth.py --platform retroarch # diff truth vs scraped
python scripts/diff_truth.py --all # diff all platforms
```
### export_native.py
Export truth data to native platform formats (System.dat, es_bios.xml, checkBIOS.sh, etc.).
```bash
python scripts/export_native.py --platform batocera
python scripts/export_native.py --all --output-dir dist/upstream/
```
### validation.py
Validation index and ground truth formatting. Used by verify.py for emulator-level checks
(size, CRC32, MD5, SHA1, crypto). Separates reproducible hash checks from cryptographic
validations that require console-specific keys.
### refresh_data_dirs.py
Fetch data directories (Dolphin Sys, PPSSPP assets, blueMSX databases)
@@ -107,24 +174,45 @@ python scripts/refresh_data_dirs.py --key dolphin-sys --force
| Script | Purpose |
|--------|---------|
| `common.py` | Shared library: hash computation, file resolution, platform config loading, emulator profiles, target filtering |
| `dedup.py` | Deduplicate `bios/`, move duplicates to `.variants/`. RPG Maker and ScummVM excluded (NODEDUP) |
| `validate_pr.py` | Validate BIOS files in pull requests |
| `auto_fetch.py` | Fetch missing BIOS files from known sources |
| `validate_pr.py` | Validate BIOS files in pull requests, post markdown report |
| `auto_fetch.py` | Fetch missing BIOS files from known sources (4-step pipeline) |
| `list_platforms.py` | List active platforms (used by CI) |
| `download.py` | Download packs from GitHub releases |
| `common.py` | Shared library: hash computation, file resolution, platform config loading, emulator profiles |
| `download.py` | Download packs from GitHub releases (Python, multi-threaded) |
| `generate_readme.py` | Generate README.md and CONTRIBUTING.md from database |
| `generate_site.py` | Generate all MkDocs site pages (this documentation) |
| `deterministic_zip.py` | Rebuild MAME BIOS ZIPs deterministically (same ROMs = same hash) |
| `crypto_verify.py` | 3DS RSA signature and AES crypto verification |
| `sect233r1.py` | Pure Python ECDSA verification on sect233r1 curve (3DS OTP cert) |
| `batch_profile.py` | Batch profiling automation for libretro cores |
| `check_buildbot_system.py` | Detect stale data directories by comparing with buildbot |
| `migrate.py` | Migrate flat bios structure to Manufacturer/Console/ hierarchy |
## Installation tools
Cross-platform BIOS installer for end users:
```bash
# Python installer (auto-detects platform)
python install.py
# Shell one-liner (Linux/macOS)
bash scripts/download.sh retroarch ~/RetroArch/system/
bash scripts/download.sh --list
# Or via install.sh wrapper (detects curl/wget, runs install.py)
bash install.sh
```
`install.py` auto-detects the user's platform by checking config files,
downloads the matching BIOS pack from GitHub releases with SHA1 verification,
and extracts files to the correct directory. `install.ps1` provides
equivalent functionality for Windows/PowerShell.
## Large files
Files over 50 MB are stored as assets on the `large-files` GitHub release.
They are listed in `.gitignore` so they don't bloat the git repository.
They are listed in `.gitignore` to keep the git repository lightweight.
`generate_db.py` downloads them from the release when rebuilding the database,
using `fetch_large_file()` from `common.py`. The same function is used by
`generate_pack.py` when a file has a hash mismatch with the local variant.
@@ -141,11 +229,49 @@ Located in `scripts/scraper/`. Each inherits `BaseScraper` and implements `fetch
| `retrobat_scraper` | batocera-systems.json | JSON |
| `emudeck_scraper` | checkBIOS.sh | Bash + CSV |
| `retrodeck_scraper` | component manifests | JSON per component |
| `romm_scraper` | known_bios_files.json | JSON |
| `coreinfo_scraper` | .info files from libretro-core-info | INI-like |
| `bizhawk_scraper` | FirmwareDatabase.cs | C# source |
| `mame_hash_scraper` | mamedev/mame source tree | C source (sparse clone) |
| `fbneo_hash_scraper` | FBNeo source tree | C source (sparse clone) |
Internal modules: `base_scraper.py` (abstract base with `_fetch_raw()` caching
and shared CLI), `dat_parser.py` (clrmamepro DAT format parser).
and shared CLI), `dat_parser.py` (clrmamepro DAT format parser),
`mame_parser.py` (MAME C source BIOS root set parser),
`fbneo_parser.py` (FBNeo C source BIOS set parser),
`_hash_merge.py` (text-based YAML patching that preserves formatting).
Adding a scraper: inherit `BaseScraper`, implement `fetch_requirements()`,
call `scraper_cli(YourScraper)` in `__main__`.
## Target scrapers
Located in `scripts/scraper/targets/`. Each inherits `BaseTargetScraper` and implements `fetch_targets()`.
| Scraper | Source | Targets |
|---------|--------|---------|
| `retroarch_targets_scraper` | libretro buildbot nightly | 20+ architectures |
| `batocera_targets_scraper` | Config.in + es_systems.yml | 35+ boards |
| `emudeck_targets_scraper` | EmuScripts GitHub API | steamos, windows |
| `retropie_targets_scraper` | scriptmodules + rp_module_flags | 7 platforms |
```bash
python -m scripts.scraper.targets.retroarch_targets_scraper --dry-run
python -m scripts.scraper.targets.batocera_targets_scraper --dry-run
```
## Exporters
Located in `scripts/exporter/`. Each inherits `BaseExporter` and implements `export()`.
| Exporter | Output format |
|----------|--------------|
| `systemdat_exporter` | clrmamepro DAT (RetroArch System.dat) |
| `batocera_exporter` | Python dict (batocera-systems) |
| `recalbox_exporter` | XML (es_bios.xml) |
| `retrobat_exporter` | JSON (batocera-systems.json) |
| `emudeck_exporter` | Bash script (checkBIOS.sh) |
| `retrodeck_exporter` | JSON (component_manifest.json) |
| `romm_exporter` | JSON (known_bios_files.json) |
| `lakka_exporter` | clrmamepro DAT (delegates to systemdat) |
| `retropie_exporter` | clrmamepro DAT (delegates to systemdat) |

243
wiki/troubleshooting.md Normal file
View File

@@ -0,0 +1,243 @@
# Troubleshooting - RetroBIOS
Diagnosis guide organized by symptom. Each section describes what to check and how to fix it.
## Game won't start / black screen
Most launch failures are caused by a missing or incorrect BIOS file.
**Check if the BIOS exists:**
```bash
python scripts/verify.py --platform retroarch --verbose
python scripts/verify.py --system sony-playstation
```
Look for `MISSING` entries in the output. A missing required BIOS means the core
cannot start games for that system at all.
**Check if the hash matches:**
Look for `HASH_MISMATCH` in the verify output. This means the file exists but
contains different data than expected. Common causes:
- Wrong region (a PAL BIOS instead of NTSC, or vice versa)
- Wrong hardware revision (e.g. SCPH-5501 vs SCPH-1001 for PlayStation)
- Corrupted download
Each system page on the site lists the expected hashes. Compare your file's
MD5 or SHA1 against those values.
**Wrong region BIOS:**
Some cores require region-specific BIOS files. A Japanese BIOS won't boot
North American games on cores that enforce region matching. Check the emulator
profile for your core to see which regions are supported and which files
correspond to each.
## BIOS not found by emulator
The file exists on disk, but the emulator reports it as missing.
**Wrong directory:**
Each platform expects BIOS files in a specific base directory:
- RetroArch, Lakka: `system/` inside the RetroArch directory
- Batocera: `/userdata/bios/`
- Recalbox: `/recalbox/share/bios/`
- RetroPie: `~/RetroPie/BIOS/`
Some cores expect files in subdirectories (e.g. `dc/` for Dreamcast, `pcsx2/bios/`
for PlayStation 2). Check the `path:` field in the emulator profile for the exact
expected location relative to the base directory.
**Wrong filename:**
Cores match BIOS files by exact filename. If a core expects `scph5501.bin` and your
file is named `SCPH-5501.BIN`, it won't be found on platforms that do exact name matching.
Check the emulator profile for the expected filename and any aliases listed under
`aliases:`. Aliases are alternative names that the core also accepts.
**Case sensitivity:**
Linux filesystems are case-sensitive. A file named `Bios.ROM` won't match a lookup
for `bios.rom`. Windows and macOS are case-insensitive by default, so the same
file works there but fails on Linux.
Batocera's verification uses `casefold()` for case-insensitive matching, but
the actual emulator may still require exact case. When in doubt, use the exact
filename from the emulator profile.
## Hash mismatch / UNTESTED
`verify.py` reports `HASH_MISMATCH` or `UNTESTED` for a file.
**HASH_MISMATCH:**
The file exists and was hashed, but the computed hash doesn't match any expected
value. This means you have a different version of the file than what the platform
or emulator expects.
To find the correct version, check the system page on the site. It lists every
known BIOS file with its expected MD5 and SHA1.
**UNTESTED:**
On existence-only platforms (RetroArch, Lakka, RetroPie), the file is present
but its hash was not verified against a known value. The platform itself only
checks that the file exists. The `--verbose` flag shows ground truth data from
emulator profiles, which can confirm whether the file's hash is actually correct.
**The .variants/ directory:**
When multiple versions of the same BIOS exist (different revisions, regions, or
dumps), the primary version lives in the main directory and alternatives live in
`.variants/`. `verify.py` checks the primary file first, then falls back to
variants when resolving by hash.
If your file matches a variant hash but not the primary, it's a valid BIOS --
just not the preferred version. Some cores accept multiple versions.
## Pack is missing files
A generated pack doesn't contain all the files you expected.
**Severity levels:**
`verify.py` assigns a severity to each issue. Not all missing files are equally
important:
| Severity | Meaning | Action needed |
|----------|---------|---------------|
| CRITICAL | Required file missing or hash mismatch on MD5 platforms | Must fix. Core won't function. |
| WARNING | Optional file missing, or hash mismatch on existence platforms | Core works but with reduced functionality. |
| INFO | Optional file missing on existence-only platforms, or HLE fallback available | Core works fine, BIOS improves accuracy. |
| OK | File present and verified | No action needed. |
Focus on CRITICAL issues first. WARNING files improve the experience but aren't
strictly necessary. INFO files are nice to have.
**Large files (over 50 MB):**
Files like PS3UPDAT.PUP, PSVUPDAT.PUP, and Switch firmware are too large for the
git repository. They are stored as GitHub release assets under the `large-files`
release and downloaded at build time.
If a pack build fails to include these, check your network connection. In offline
mode (`--offline`), large files are only included if already cached locally in
`.cache/large/`.
**Data directories:**
Some cores need entire directory trees rather than individual files (e.g. Dolphin's
`Sys/` directory, PPSSPP's `assets/`). These are fetched by `refresh_data_dirs.py`
from upstream repositories.
In offline mode, data directories are only included if already cached in `data/`.
Run `python scripts/refresh_data_dirs.py` to fetch them.
## verify.py reports errors
How to read and interpret `verify.py` output.
**Status codes:**
| Status | Meaning |
|--------|---------|
| `ok` | File present, hash matches (or existence check passed) |
| `untested` | File present, hash not confirmed (existence-only platforms) |
| `missing` | File not found in the repository |
| `hash_mismatch` | File found but hash doesn't match expected value |
| `size_mismatch` | File found but size doesn't match what the emulator expects |
**Reading the output:**
Each line shows the file path, its status, and severity. In verbose mode, ground
truth data from emulator profiles is appended, showing which cores reference the
file and what validations they perform.
```
scph5501.bin ok [OK]
dc_boot.bin missing [CRITICAL]
gba_bios.bin untested [WARNING]
```
**Cross-reference section:**
After per-file results, `verify.py` prints a cross-reference report. This lists
files that emulator cores need but that the platform YAML doesn't declare. These
files are still included in packs automatically, but the report helps identify
gaps in platform coverage data.
The cross-reference uses `resolve_platform_cores()` to determine which emulator
profiles are relevant for each platform, then checks whether each profile's files
appear in the platform config.
**Filtering output:**
```bash
# By platform
python scripts/verify.py --platform batocera
# By emulator core
python scripts/verify.py --emulator beetle_psx
# By system
python scripts/verify.py --system sony-playstation
# By hardware target
python scripts/verify.py --platform retroarch --target switch
# JSON for scripted processing
python scripts/verify.py --platform retroarch --json
```
## Installation script fails
Problems with `install.py`, `install.sh`, or `download.sh`.
**Network issues:**
The installer downloads packs from GitHub releases. If the download fails:
- Check your internet connection
- Verify that `https://github.com` is reachable
- If behind a proxy, set `HTTPS_PROXY` in your environment
- Try again later if GitHub is experiencing issues
**Permission denied:**
The installer needs write access to the target directory.
- On Linux/macOS: check directory ownership (`ls -la`) and run with appropriate
permissions. Avoid running as root unless the target directory requires it.
- On Windows: run PowerShell as Administrator if installing to a protected directory.
**Platform not detected:**
`install.py` auto-detects your platform by checking for known config files. If
detection fails, specify the platform manually:
```bash
python install.py --platform retroarch --dest ~/RetroArch/system/
python install.py --platform batocera --dest /userdata/bios/
```
Use `python install.py --help` to see all available platforms and options.
**Pack not found in release:**
If the installer reports that no pack exists for your platform, check available
releases:
```bash
python scripts/download.py --list
# or
bash scripts/download.sh --list
```
Some platforms share packs (Lakka uses the RetroArch pack). The installer handles
this mapping automatically, but if you're downloading manually, check which pack
name corresponds to your platform.

248
wiki/verification-modes.md Normal file
View File

@@ -0,0 +1,248 @@
# Verification Modes
Each platform verifies BIOS files differently. `verify.py` replicates the native behavior
of each platform so that verification results match what the platform itself would report.
## Existence Mode
**Platforms**: RetroArch, Lakka, RetroPie
**Source**: RetroArch `core_info.c`, function `path_is_valid()`
The most straightforward mode. A file is OK if it exists at the expected path. No hash is checked.
Any file with the correct name passes, regardless of content.
| Condition | Status | Severity (required) | Severity (optional) |
|-----------|--------|---------------------|---------------------|
| File present | OK | OK | OK |
| File missing | MISSING | WARNING | INFO |
RetroArch does not distinguish between a correct and an incorrect BIOS at the verification
level. A corrupt or wrong-region file still shows as present. This is by design in the
upstream code: `core_info.c` only calls `path_is_valid()` and does not open or hash the file.
Lakka and RetroPie inherit this behavior through platform config inheritance
(`inherits: retroarch` in the platform YAML).
## MD5 Mode
**Platforms**: Batocera, RetroBat, Recalbox, EmuDeck, RetroDECK, RomM
All MD5-mode platforms compute a hash of the file and compare it against an expected value.
The details vary by platform.
### Standard MD5 (Batocera, RetroBat)
`verify.py` replicates Batocera's `md5sum()` function. The file is read in binary mode,
hashed with MD5, and compared case-insensitively against the expected value.
| Condition | Status | Severity (required) | Severity (optional) |
|-----------|--------|---------------------|---------------------|
| Hash matches | OK | OK | OK |
| File present, hash differs | UNTESTED | WARNING | WARNING |
| File missing | MISSING | CRITICAL | WARNING |
If the `resolve_local_file` step already confirmed the MD5 match (status `md5_exact`),
`verify.py` skips re-hashing and returns OK directly.
### Truncated MD5 (Batocera bug)
Some entries in Batocera's system data contain 29-character MD5 strings instead of
the standard 32. This is a known upstream bug. `verify.py` handles it by prefix matching:
if the expected hash is shorter than 32 characters, the actual hash is compared against
only its first N characters.
### md5_composite (Recalbox ZIP verification)
Recalbox computes `Zip::Md5Composite` for ZIP files: the MD5 of the concatenation of all
inner file MD5s (sorted by filename). `verify.py` replicates this with `md5_composite()`
from `common.py`. When a ZIP file's direct MD5 does not match, the composite is tried
before reporting a mismatch.
### Multi-hash (Recalbox)
Recalbox allows comma-separated MD5 values for a single file entry, accepting any one
of them as valid. `verify.py` splits on commas and tries each hash. A match against any
listed hash is OK.
### Mandatory levels (Recalbox)
Recalbox uses three severity levels derived from two YAML fields (`mandatory` and
`hashMatchMandatory`):
| mandatory | hashMatchMandatory | Color | verify.py mapping |
|-----------|--------------------|--------|-------------------|
| true | true | RED | CRITICAL |
| true | false | YELLOW | WARNING |
| false | (any) | GREEN | INFO |
### checkInsideZip (Batocera zippedFile)
When a platform entry has a `zipped_file` field, the expected MD5 is not the hash of the
ZIP container but of a specific ROM file inside the ZIP. `verify.py` replicates Batocera's
`checkInsideZip()`:
1. Open the ZIP.
2. Find the inner file by name (case-insensitive via `casefold()`).
3. Read its contents and compute MD5.
4. Compare against the expected hash.
If the inner file is not found inside the ZIP, the status is UNTESTED with a reason string.
### RomM verification
RomM checks both file size and hash. It accepts any hash type (MD5, SHA1, or CRC32).
ZIP files are not opened; only the container is checked. `verify.py` replicates this
by checking size first, then trying each available hash.
## SHA1 Mode
**Platforms**: BizHawk
BizHawk firmware entries use SHA1 as the primary hash. `verify.py` computes SHA1
via `compute_hashes()` and compares case-insensitively.
| Condition | Status | Severity (required) | Severity (optional) |
|-----------|--------|---------------------|---------------------|
| SHA1 matches | OK | OK | OK |
| File present, SHA1 differs | UNTESTED | WARNING | WARNING |
| File missing | MISSING | CRITICAL | WARNING |
## Emulator-Level Validation
Independent of platform verification mode, `verify.py` runs emulator-level validation
from `validation.py`. This layer uses data from emulator profiles (YAML files in
`emulators/`), which are source-verified against emulator code.
### Validation index
`_build_validation_index()` reads all emulator profiles and builds a per-filename
index of validation rules. When multiple emulators reference the same file, checks
are merged (union of all check types). Conflicting expected values are kept as sets
(e.g., multiple accepted CRC32 values for different ROM versions).
Each entry in the index tracks:
- `checks`: list of validation types (e.g., `["size", "crc32"]`)
- `sizes`: set of accepted exact sizes
- `min_size`, `max_size`: bounds when the code accepts a range
- `crc32`, `md5`, `sha1`, `sha256`: sets of accepted hash values
- `adler32`: set of accepted Adler-32 values
- `crypto_only`: non-reproducible checks (see below)
- `per_emulator`: per-core detail with source references
### Check categories
Validation checks fall into two categories:
**Reproducible** (`_HASH_CHECKS`): `crc32`, `md5`, `sha1`, `adler32`. These can be
computed from the file alone. `verify.py` calculates hashes and compares against
accepted values from the index.
**Non-reproducible** (`_CRYPTO_CHECKS`): `signature`, `crypto`. These require
console-specific cryptographic keys (e.g., RSA-2048 for 3DS, AES-128-CBC for certain
firmware). `verify.py` reports these as informational but cannot verify them without
the keys. Size checks still apply if combined with crypto.
### Size validation
Three forms:
- **Exact size**: `size: 524288` with `validation: [size]`. File must be exactly this many bytes.
- **Range**: `min_size: 40`, `max_size: 131076` with `validation: [size]`. File size must fall within bounds.
- **Informational**: `size: 524288` without `validation: [size]`. The size is documented but the emulator does not check it at runtime.
### Complement to platform checks
Emulator validation runs after platform verification. When a file passes platform checks
(e.g., existence-mode OK) but fails emulator validation (e.g., wrong CRC32), the result
includes a `discrepancy` field:
```
file present (OK) but handy says size mismatch: got 256, accepted [512]
```
This catches cases where a file has the right name but wrong content, which existence-mode
platforms cannot detect.
## Severity Matrix
`compute_severity()` maps the combination of status, required flag, verification mode,
and HLE fallback to a severity level.
| Mode | Status | required | hle_fallback | Severity |
|------|--------|----------|--------------|----------|
| any | OK | any | any | OK |
| any | MISSING | any | true | INFO |
| existence | MISSING | true | false | WARNING |
| existence | MISSING | false | false | INFO |
| md5/sha1 | MISSING | true | false | CRITICAL |
| md5/sha1 | MISSING | false | false | WARNING |
| md5/sha1 | UNTESTED | any | false | WARNING |
**HLE fallback**: when an emulator profile marks a file with `hle_fallback: true`, the
core has a built-in high-level emulation path and functions without the file. Missing
files are downgraded to INFO regardless of platform mode or required status. The file
is still included in packs (better accuracy with the real BIOS), but its absence is not
actionable.
## File Resolution Chain
Before verification, each file entry is resolved to a local path by `resolve_local_file()`.
The function tries these steps in order, returning the first match:
| Step | Method | Returns | When it applies |
|------|--------|---------|-----------------|
| 0 | Path suffix exact | `exact` | `dest_hint` matches `by_path_suffix` index (regional variants with same filename, e.g., `GC/USA/IPL.bin` vs `GC/EUR/IPL.bin`) |
| 1 | SHA1 exact | `exact` | SHA1 present in the file entry and found in database |
| 2 | MD5 direct lookup | `md5_exact` | MD5 present, not a `zipped_file` entry, name matches (prevents cross-contamination from unrelated files sharing an MD5) |
| 3 | Name/alias existence | `exact` | No MD5 in entry; any file with matching name or alias exists. Prefers primary over `.variants/` |
| 4 | Name + md5_composite/MD5 | `exact` or `hash_mismatch` | Name matches, checks md5_composite for ZIPs and direct MD5 per candidate. Falls back to hash_mismatch if name matches but no hash does |
| 5 | ZIP contents index | `zip_exact` | `zipped_file` with MD5; searches inner ROM MD5 across all ZIPs when name-based resolution failed |
| 6 | MAME clone fallback | `mame_clone` | File was deduped; resolves via canonical set name (up to 3 levels deep) |
| 7 | Data directory scan | `data_dir` | Searches `data/` caches by exact path then case-insensitive basename walk |
| 8 | Agnostic fallback | `agnostic_fallback` | File entry marked `agnostic: true`; matches any file under the system path prefix within the size constraints |
If no step matches, the result is `(None, "not_found")`.
The `hash_mismatch` status at step 4 means a file with the right name exists but its hash
does not match. This still resolves to a local path (the file is present), but verification
will report it as UNTESTED with a reason string showing the expected vs actual hash prefix.
## Discrepancy Detection
When platform verification passes but emulator validation fails, the file has a discrepancy.
This happens most often in existence-mode platforms where any file with the right name is
accepted.
### Variant search
`_find_best_variant()` searches for an alternative file in the repository that satisfies
both the platform MD5 requirement and emulator validation:
1. Look up all files with the same name in the `by_name` index.
2. Skip the current file (already known to fail validation).
3. For each candidate, check that its MD5 matches the platform expectation.
4. Run `check_file_validation()` against the candidate.
5. Return the first candidate that passes both checks.
The search covers files in `.variants/` (alternate hashes stored during deduplication).
If a better variant is found, the pack uses it instead of the primary file. If no variant
satisfies both constraints, the platform version is kept and the discrepancy is reported
in the verification output.
### Practical example
A `scph5501.bin` file passes Batocera MD5 verification (hash matches upstream declaration)
but fails the emulator profile's size check because the profile was verified against a
different revision. `_find_best_variant` scans `.variants/scph5501.bin.*` for a file
that matches both the Batocera MD5 and the emulator's size expectation. If found, the
variant is used in the pack. If not, the Batocera-verified file is kept and the discrepancy
is logged.