mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-10-10 13:33:24 -05:00
docs: correct and complete the documentation set
This commit is contained in:
Abdessamad Derraz committed
1 parent
45dbc30301
commit
5f60138dce
21 files changed
+986
-216
No files matched your search
@@ -141,15 +141,20 @@ jobs:
|
||||
- Linux/macOS: \`cat PackName.zip.0* > PackName.zip\`
|
||||
- Windows (cmd): \`copy /b PackName.zip.001+PackName.zip.002 PackName.zip\`
|
||||
|
||||
| Platform | Pack | Path |
|
||||
|----------|------|------|
|
||||
| RetroArch / Lakka | RetroArch_Lakka_BIOS_Pack.zip | system/ |
|
||||
| Batocera | Batocera_BIOS_Pack.zip | /userdata/bios/ |
|
||||
| Recalbox | Recalbox_BIOS_Pack.zip | /recalbox/share/bios/ |
|
||||
| RetroBat | RetroBat_BIOS_Pack.zip | bios/ |
|
||||
| RetroDECK | RetroDECK_BIOS_Pack.zip | ~/retrodeck/bios/ |
|
||||
| EmuDeck | EmuDeck_BIOS_Pack.zip | Emulation/bios/ |
|
||||
| RomM | RomM_BIOS_Pack.zip | bios/{platform_slug}/ |
|
||||
| Platform | Extract to |
|
||||
|----------|------------|
|
||||
| RetroArch | system/ |
|
||||
| Lakka | /storage/system/ |
|
||||
| RetroPie | ~/RetroPie/BIOS/ |
|
||||
| Batocera | /userdata/bios/ |
|
||||
| Recalbox | /recalbox/share/bios/ |
|
||||
| RetroBat | bios/ |
|
||||
| RetroDECK | ~/retrodeck/ (the pack carries its own bios/) |
|
||||
| EmuDeck | ~/Emulation/bios/ |
|
||||
| ROCKNIX | /storage/roms/bios/ |
|
||||
| MiSTer FPGA | /media/fat/games/ |
|
||||
| BizHawk | Firmware/ |
|
||||
| RomM | bios/{platform_slug}/ |
|
||||
|
||||
### Changes
|
||||
${CHANGES}
|
||||
|
||||
+42
-10
@@ -4,21 +4,53 @@
|
||||
|
||||
1. Fork this repository
|
||||
2. Place the file in `bios/Manufacturer/Console/filename`
|
||||
3. Variants (alternate hashes): `bios/Manufacturer/Console/.variants/`
|
||||
4. Create a Pull Request - checksums are verified automatically
|
||||
3. Variants (alternate hashes for the same file): `bios/Manufacturer/Console/.variants/`
|
||||
4. Open a Pull Request - hashes are verified automatically and reported as a comment
|
||||
|
||||
## Add a new platform
|
||||
The [dump provenance](https://abdess.github.io/retrobios/provenance/) page lists catalogued dumps still
|
||||
missing from the collection, with their hashes. A file matching one of those is
|
||||
the most useful contribution.
|
||||
|
||||
1. Write a scraper in `scripts/scraper/`
|
||||
2. Create the platform YAML in `platforms/`
|
||||
3. Register in `platforms/_registry.yml`
|
||||
4. Submit a Pull Request
|
||||
## Add a platform
|
||||
|
||||
Contributors who add platform support are credited in the README,
|
||||
on the documentation site, and in the BIOS packs.
|
||||
1. Write a scraper in `scripts/scraper/` (inherit `BaseScraper`)
|
||||
2. Read the platform's upstream source to determine how it checks BIOS files
|
||||
3. Register it in `platforms/_registry.yml`
|
||||
4. Generate the platform YAML and test: `python scripts/verify.py --platform <name>`
|
||||
|
||||
Full walkthrough: [adding a platform](https://abdess.github.io/retrobios/wiki/adding-a-platform/).
|
||||
|
||||
## Add an emulator profile
|
||||
|
||||
1. Clone the emulator's source code, upstream and libretro port
|
||||
2. Trace the file loading from the entry point, not from a keyword grep
|
||||
3. Document every file the code loads, with a `source_ref` line reference
|
||||
4. Write the YAML to `emulators/<name>.yml`
|
||||
5. Test: `python scripts/cross_reference.py --emulator <name>`
|
||||
|
||||
Full walkthrough: [profiling guide](https://abdess.github.io/retrobios/wiki/profiling/).
|
||||
|
||||
## File conventions
|
||||
|
||||
- `bios/Manufacturer/Console/filename` for canonical files
|
||||
- `bios/Manufacturer/Console/.variants/filename.sha1prefix` for alternate versions
|
||||
- Files >50 MB go in GitHub release assets (`large-files` release)
|
||||
- RPG Maker and ScummVM directories are excluded from deduplication
|
||||
- See the [documentation site](https://abdess.github.io/retrobios/) for full details
|
||||
- Two paths differing only by case break clones on Windows and macOS;
|
||||
`tests/test_no_case_collisions.py` enforces this
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
```bash
|
||||
python -m unittest discover tests
|
||||
python scripts/pipeline.py --offline
|
||||
```
|
||||
|
||||
## PR validation
|
||||
|
||||
CI computes SHA1/MD5/CRC32 for every new file, checks them against the platform
|
||||
configs, validates the YAML against the schemas, runs the test suite, and posts
|
||||
a report on the PR.
|
||||
|
||||
Contributors who add platform support are credited in the README,
|
||||
on the documentation site, and in the BIOS packs.
|
||||
@@ -36,26 +36,30 @@ Pick your platform, download the ZIP, extract to the BIOS path.
|
||||
|----------|-----------|-----------|----------|
|
||||
| Batocera | 353 | `/userdata/bios/` | [Download](../../releases/latest) |
|
||||
| BizHawk | 118 | `Firmware/` | [Download](../../releases/latest) |
|
||||
| EmuDeck | 161 | `Emulation/bios/` | [Download](../../releases/latest) |
|
||||
| Lakka | 530 | `system/` | [Download](../../releases/latest) |
|
||||
| EmuDeck | 161 | `~/Emulation/bios/` | [Download](../../releases/latest) |
|
||||
| Lakka | 530 | `/storage/system/` | [Download](../../releases/latest) |
|
||||
| MiSTer FPGA | 65 | `/media/fat/games/` | [Download](../../releases/latest) |
|
||||
| ROCKNIX | 38 | `/storage/roms/bios/` | [Download](../../releases/latest) |
|
||||
| Recalbox | 346 | `/recalbox/share/bios/` | [Download](../../releases/latest) |
|
||||
| RetroArch | 530 | `system/` | [Download](../../releases/latest) |
|
||||
| RetroBat | 341 | `bios/` | [Download](../../releases/latest) |
|
||||
| RetroDECK | 2008 | `~/retrodeck/bios/` | [Download](../../releases/latest) |
|
||||
| RetroPie | 530 | `BIOS/` | [Download](../../releases/latest) |
|
||||
| RetroDECK | 2008 | `~/retrodeck/` | [Download](../../releases/latest) |
|
||||
| RetroPie * | 530 | `~/RetroPie/BIOS/` | [Download](../../releases/latest) |
|
||||
| RomM | 374 | `bios/{platform_slug}/` | [Download](../../releases/latest) |
|
||||
|
||||
The RetroDECK pack already contains its own `bios/` folder, so it extracts into `~/retrodeck/` rather than into the BIOS folder.
|
||||
|
||||
\* Archived: the configuration is kept and packs are still built, but upstream is no longer scraped on a schedule.
|
||||
|
||||
## What's included
|
||||
|
||||
BIOS, firmware, and system files for consoles from Atari to PlayStation 3.
|
||||
Every file passes its platform's own verification; where an emulator profile exists, the expected hashes and sizes are read from the emulator's source code (the Source-backed column below).
|
||||
|
||||
- **12 platforms** supported with platform-specific verification
|
||||
- **313 emulators** profiled from source (RetroArch cores + standalone)
|
||||
- **318 emulators** profiled from source (RetroArch cores + standalone)
|
||||
- **396 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
|
||||
- **7,651 files** verified with MD5, SHA1, CRC32 checksums: 2,366 system files, 2,746 arcade ROM sets, 2,539 game and engine data files
|
||||
- **7,651 files** indexed with SHA1, MD5, SHA256 and CRC32 checksums: 2,366 system files, 2,746 arcade ROM sets, 2,539 game and engine data files
|
||||
- **527 files** matched to dump-preservation catalogs (No-Intro, Redump, TOSEC)
|
||||
- **9832 MB** total collection size
|
||||
|
||||
@@ -69,18 +73,18 @@ Full list with per-file details: **[https://abdess.github.io/retrobios/](https:/
|
||||
|
||||
| Platform | Coverage | Verified | Untested | Missing | Source-backed |
|
||||
|----------|----------|----------|----------|---------|---------------|
|
||||
| Batocera | 353/353 (100.0%) | 353 | 0 | 0 | 95/353 (27%) |
|
||||
| Batocera | 353/353 (100.0%) | 353 | 0 | 0 | 96/353 (27%) |
|
||||
| BizHawk | 118/118 (100.0%) | 118 | 0 | 0 | 4/118 (3%) |
|
||||
| EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 | 15/161 (9%) |
|
||||
| Lakka | 530/530 (100.0%) | 530 | 0 | 0 | 118/530 (22%) |
|
||||
| EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 | 16/161 (10%) |
|
||||
| Lakka | 530/530 (100.0%) | 530 | 0 | 0 | 124/530 (23%) |
|
||||
| MiSTer FPGA | 65/65 (100.0%) | 65 | 0 | 0 | - |
|
||||
| ROCKNIX | 38/38 (100.0%) | 38 | 0 | 0 | 29/38 (76%) |
|
||||
| Recalbox | 346/346 (100.0%) | 346 | 0 | 0 | 83/346 (24%) |
|
||||
| RetroArch | 530/530 (100.0%) | 530 | 0 | 0 | 118/530 (22%) |
|
||||
| RetroBat | 341/341 (100.0%) | 341 | 0 | 0 | 81/341 (24%) |
|
||||
| RetroDECK | 2008/2008 (100.0%) | 2008 | 0 | 0 | 104/2008 (5%) |
|
||||
| RetroPie | 530/530 (100.0%) | 530 | 0 | 0 | 118/530 (22%) |
|
||||
| RomM | 374/374 (100.0%) | 374 | 0 | 0 | 84/374 (22%) |
|
||||
| Recalbox | 346/346 (100.0%) | 346 | 0 | 0 | 86/346 (25%) |
|
||||
| RetroArch | 530/530 (100.0%) | 530 | 0 | 0 | 124/530 (23%) |
|
||||
| RetroBat | 341/341 (100.0%) | 341 | 0 | 0 | 84/341 (25%) |
|
||||
| RetroDECK | 2008/2008 (100.0%) | 2008 | 0 | 0 | 121/2008 (6%) |
|
||||
| RetroPie * | 530/530 (100.0%) | 530 | 0 | 0 | 124/530 (23%) |
|
||||
| RomM | 374/374 (100.0%) | 374 | 0 | 0 | 88/374 (24%) |
|
||||
|
||||
Coverage is measured against the file list each platform declares, using that platform's own verification mode.
|
||||
Source-backed counts the files whose content the emulator's own code checks: a size or hash read from its source, reproduced at verification. A dash means no profiled emulator applies to the platform, whose own source is then the only authority.
|
||||
@@ -120,7 +124,7 @@ The [documentation site](https://abdess.github.io/retrobios/) provides:
|
||||
- **Per-emulator profiles** with source code references for every file
|
||||
- **Per-system pages** showing which emulators and platforms cover each console
|
||||
- **Gap analysis** identifying missing files and undeclared core requirements
|
||||
- **Cross-reference** mapping files across 12 platforms and 313 emulators
|
||||
- **Cross-reference** mapping files across 12 platforms and 318 emulators
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -157,4 +161,4 @@ The scripts and tooling are released under the [MIT License](LICENSE).
|
||||
The BIOS and firmware files are not covered by that license: they are third-party system software, preserved and provided for personal backup, archival, and interoperability with emulation software.
|
||||
The legal reasoning is laid out in the [FAQ](https://abdess.github.io/retrobios/wiki/faq/#is-this-legal).
|
||||
|
||||
*Auto-generated on 2026-08-08T00:28:41Z*
|
||||
*Auto-generated on 2026-08-08T02:07:01Z*
|
||||
+52
-6
@@ -1,7 +1,15 @@
|
||||
site_name: RetroBIOS
|
||||
site_description: Source-verified BIOS and firmware packs for RetroArch, Batocera,
|
||||
Recalbox, Lakka, RetroPie, EmuDeck, RetroBat, RetroDECK, RomM, BizHawk, ROCKNIX
|
||||
and MiSTer FPGA.
|
||||
site_url: https://abdess.github.io/retrobios/
|
||||
repo_url: https://github.com/Abdess/retrobios
|
||||
repo_name: Abdess/retrobios
|
||||
# Almost every page is generated from platforms/, emulators/ and database.json,
|
||||
# so a per-page edit link would point at a file that does not exist.
|
||||
edit_uri: ''
|
||||
copyright: MIT for the tooling. BIOS and firmware files are third-party system
|
||||
software, preserved for personal backup, archival and interoperability.
|
||||
theme:
|
||||
name: material
|
||||
palette:
|
||||
@@ -25,24 +33,46 @@ theme:
|
||||
icon:
|
||||
logo: material/chip
|
||||
features:
|
||||
- navigation.instant
|
||||
- navigation.instant.prefetch
|
||||
- navigation.instant.progress
|
||||
- navigation.tabs
|
||||
- navigation.tabs.sticky
|
||||
- navigation.sections
|
||||
- navigation.top
|
||||
- navigation.tracking
|
||||
- navigation.indexes
|
||||
# 400+ pages: pruning keeps the navigation out of every page's HTML.
|
||||
- navigation.prune
|
||||
- navigation.footer
|
||||
- search.suggest
|
||||
- search.highlight
|
||||
- search.share
|
||||
- content.code.copy
|
||||
- content.tabs.link
|
||||
- toc.follow
|
||||
extra_css:
|
||||
- stylesheets/extra.css
|
||||
extra:
|
||||
social:
|
||||
- icon: fontawesome/brands/github
|
||||
link: https://github.com/Abdess/retrobios
|
||||
name: RetroBIOS on GitHub
|
||||
markdown_extensions:
|
||||
- tables
|
||||
- abbr
|
||||
- admonition
|
||||
- attr_list
|
||||
- def_list
|
||||
- footnotes
|
||||
- md_in_html
|
||||
- tables
|
||||
- toc:
|
||||
permalink: true
|
||||
- pymdownx.details
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.keys
|
||||
- pymdownx.superfences:
|
||||
custom_fences:
|
||||
- name: mermaid
|
||||
@@ -52,6 +82,13 @@ markdown_extensions:
|
||||
alternate_style: true
|
||||
plugins:
|
||||
- search
|
||||
# Link rot fails the build: deploy-site.yml runs `mkdocs build --strict`.
|
||||
# omitted_files stays at its default (info) so a stale local docs/ does not
|
||||
# break a local build; CI regenerates docs/ from scratch anyway.
|
||||
validation:
|
||||
absolute_links: warn
|
||||
unrecognized_links: warn
|
||||
anchors: warn
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Download: which-pack.md
|
||||
@@ -211,12 +248,14 @@ nav:
|
||||
- vitaQuakeII: emulators/vitaquake2.md
|
||||
- yabasanshiro: emulators/yabasanshiro.md
|
||||
- Yuzu: emulators/yuzu.md
|
||||
- Community forks (109):
|
||||
- Community forks (111):
|
||||
- EightyOne: emulators/81.md
|
||||
- a5200: emulators/a5200.md
|
||||
- ACE-DL: emulators/ace-dl.md
|
||||
- Anarch: emulators/anarch.md
|
||||
- AppleWin: emulators/applewin.md
|
||||
- Azahar: emulators/azahar.md
|
||||
- AzaharPlus: emulators/azaharplus.md
|
||||
- b2: emulators/b2.md
|
||||
- Beetle Lynx (Mednafen Lynx): emulators/beetle_lynx.md
|
||||
- Beetle NGP (Mednafen Neo Geo Pocket): emulators/beetle_ngp.md
|
||||
@@ -390,7 +429,9 @@ nav:
|
||||
- WASM-4: emulators/wasm4.md
|
||||
- XRick: emulators/xrick.md
|
||||
- Zelda Classic v2.10: emulators/zc210.md
|
||||
- Enhanced forks (14):
|
||||
- Enhanced forks (16):
|
||||
- A7800: emulators/a7800.md
|
||||
- AdvanceMAME: emulators/advancemame.md
|
||||
- bsnes-hd beta: emulators/bsnes_hd_beta.md
|
||||
- bsnes-mercury: emulators/bsnes_mercury.md
|
||||
- DOSBox Pure: emulators/dosbox_pure.md
|
||||
@@ -438,16 +479,20 @@ nav:
|
||||
- Stone Soup: emulators/stonesoup.md
|
||||
- UME 2015: emulators/ume2015.md
|
||||
- VBA-Next: emulators/vba_next.md
|
||||
- Embedded HLE (1):
|
||||
- Embedded HLE (2):
|
||||
- 3dSen: emulators/3dsen.md
|
||||
- PCSX-ReARMed: emulators/pcsx_rearmed.md
|
||||
- Launchers (1):
|
||||
- Launchers (2):
|
||||
- Dolphin Launcher: emulators/dolphin_launcher.md
|
||||
- Other (23):
|
||||
- ES-DE: emulators/es-de.md
|
||||
- Other (25):
|
||||
- ares: emulators/ares.md
|
||||
- Basilisk II: emulators/basiliskii.md
|
||||
- Beetle GBA (Mednafen): emulators/beetle_gba.md
|
||||
- BigPEmu: emulators/bigpemu.md
|
||||
- Cemu: emulators/cemu.md
|
||||
- Clock Signal (CLK): emulators/clk.md
|
||||
- ColEm: emulators/colem.md
|
||||
- Demul: emulators/demul.md
|
||||
- eka2l1: emulators/eka2l1.md
|
||||
- ep128emu-core: emulators/ep128emu.md
|
||||
@@ -468,6 +513,7 @@ nav:
|
||||
- XRoar: emulators/xroar.md
|
||||
- Cross-reference: cross-reference.md
|
||||
- Gap Analysis: gaps.md
|
||||
- Dump provenance: provenance.md
|
||||
- Wiki:
|
||||
- Overview: wiki/index.md
|
||||
- Getting started: wiki/getting-started.md
|
||||
|
||||
@@ -95,3 +95,6 @@ Full libretro docs: `https://docs.libretro.com/library/<core>/`
|
||||
| Recalbox | md5 | multi-hash comma-separated | `es_bios.xml` + `Bios.cpp` |
|
||||
| RetroDECK | md5 | MD5 per file via component manifests | `api_data_processing.sh` |
|
||||
| RomM | md5 | size + any hash (MD5/SHA1/CRC32) | `firmware.py` |
|
||||
| ROCKNIX | md5 | `md5sum()` + `checkInsideZip()` with `altmd5` | `rocknix-systems` |
|
||||
| MiSTer FPGA | md5 | MD5 of the file at its destination | `process_db_index_worker.py` |
|
||||
| BizHawk | sha1 | SHA1 per firmware entry | `FirmwareDatabase.cs` |
|
||||
+74
-18
@@ -23,6 +23,7 @@ from common import (
|
||||
load_database,
|
||||
load_emulator_profiles,
|
||||
load_platform_config,
|
||||
load_platform_registry,
|
||||
unique_emulator_profiles,
|
||||
write_if_changed,
|
||||
)
|
||||
@@ -198,28 +199,50 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
"|----------|-----------|-----------|----------|",
|
||||
]
|
||||
|
||||
# Where the pack itself is extracted, which is not always the BIOS folder:
|
||||
# a pack whose entries already carry their own root (RetroDECK) extracts
|
||||
# one level above it.
|
||||
extract_paths = {
|
||||
"RetroArch": "`system/`",
|
||||
"Lakka": "`system/`",
|
||||
"Lakka": "`/storage/system/`",
|
||||
"Batocera": "`/userdata/bios/`",
|
||||
"BizHawk": "`Firmware/`",
|
||||
"Recalbox": "`/recalbox/share/bios/`",
|
||||
"RetroBat": "`bios/`",
|
||||
"RetroPie": "`BIOS/`",
|
||||
"RetroDECK": "`~/retrodeck/bios/`",
|
||||
"EmuDeck": "`Emulation/bios/`",
|
||||
"RetroPie": "`~/RetroPie/BIOS/`",
|
||||
"RetroDECK": "`~/retrodeck/`",
|
||||
"EmuDeck": "`~/Emulation/bios/`",
|
||||
"RomM": "`bios/{platform_slug}/`",
|
||||
"ROCKNIX": "`/storage/roms/bios/`",
|
||||
"MiSTer FPGA": "`/media/fat/games/`",
|
||||
}
|
||||
archived = {
|
||||
name
|
||||
for name, entry in load_platform_registry(platforms_dir).items()
|
||||
if entry.get("status") == "archived"
|
||||
}
|
||||
|
||||
for name, cov in sorted(coverages.items(), key=lambda x: x[1]["platform"]):
|
||||
display = cov["platform"]
|
||||
path = extract_paths.get(display, "")
|
||||
if name in archived:
|
||||
display = f"{display} *"
|
||||
path = extract_paths.get(cov["platform"], "")
|
||||
lines.append(
|
||||
f"| {display} | {cov['total']} | {path} | [Download]({RELEASE_URL}) |"
|
||||
)
|
||||
|
||||
if archived:
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"The RetroDECK pack already contains its own `bios/` folder, so it"
|
||||
" extracts into `~/retrodeck/` rather than into the BIOS folder.",
|
||||
"",
|
||||
"\\* Archived: the configuration is kept and packs are still built,"
|
||||
" but upstream is no longer scraped on a schedule.",
|
||||
]
|
||||
)
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
@@ -233,7 +256,7 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
f"- **{len(coverages)} platforms** supported with platform-specific verification",
|
||||
f"- **{emulator_count} emulators** profiled from source (RetroArch cores + standalone)",
|
||||
f"- **{len(system_ids)} systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)",
|
||||
f"- **{total_files:,} files** verified with MD5, SHA1, CRC32 checksums:"
|
||||
f"- **{total_files:,} files** indexed with SHA1, MD5, SHA256 and CRC32 checksums:"
|
||||
f" {comp['systems']['files']:,} system files,"
|
||||
f" {comp['arcade']['files']:,} arcade ROM sets,"
|
||||
f" {comp['game_data']['files']:,} game and engine data files",
|
||||
@@ -305,8 +328,9 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
gt_cell = f"{gt['with_validation']}/{gt['total']} ({gt_pct})"
|
||||
else:
|
||||
gt_cell = "0/0"
|
||||
display = f"{cov['platform']} *" if name in archived else cov["platform"]
|
||||
lines.append(
|
||||
f"| {cov['platform']} | {cov['present']}/{cov['total']} ({pct}) | "
|
||||
f"| {display} | {cov['present']}/{cov['total']} ({pct}) | "
|
||||
f"{cov['verified']} | {cov['untested']} | {cov['missing']} | "
|
||||
f"{gt_cell} |"
|
||||
)
|
||||
@@ -428,30 +452,62 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
|
||||
|
||||
def generate_contributing() -> str:
|
||||
return """# Contributing to RetroBIOS
|
||||
return f"""# Contributing to RetroBIOS
|
||||
|
||||
## Add a BIOS file
|
||||
|
||||
1. Fork this repository
|
||||
2. Place the file in `bios/Manufacturer/Console/filename`
|
||||
3. Variants (alternate hashes): `bios/Manufacturer/Console/.variants/`
|
||||
4. Create a Pull Request - checksums are verified automatically
|
||||
3. Variants (alternate hashes for the same file): `bios/Manufacturer/Console/.variants/`
|
||||
4. Open a Pull Request - hashes are verified automatically and reported as a comment
|
||||
|
||||
## Add a new platform
|
||||
The [dump provenance]({SITE_URL}provenance/) page lists catalogued dumps still
|
||||
missing from the collection, with their hashes. A file matching one of those is
|
||||
the most useful contribution.
|
||||
|
||||
1. Write a scraper in `scripts/scraper/`
|
||||
2. Create the platform YAML in `platforms/`
|
||||
3. Register in `platforms/_registry.yml`
|
||||
4. Submit a Pull Request
|
||||
## Add a platform
|
||||
|
||||
Contributors who add platform support are credited in the README,
|
||||
on the documentation site, and in the BIOS packs.
|
||||
1. Write a scraper in `scripts/scraper/` (inherit `BaseScraper`)
|
||||
2. Read the platform's upstream source to determine how it checks BIOS files
|
||||
3. Register it in `platforms/_registry.yml`
|
||||
4. Generate the platform YAML and test: `python scripts/verify.py --platform <name>`
|
||||
|
||||
Full walkthrough: [adding a platform]({SITE_URL}wiki/adding-a-platform/).
|
||||
|
||||
## Add an emulator profile
|
||||
|
||||
1. Clone the emulator's source code, upstream and libretro port
|
||||
2. Trace the file loading from the entry point, not from a keyword grep
|
||||
3. Document every file the code loads, with a `source_ref` line reference
|
||||
4. Write the YAML to `emulators/<name>.yml`
|
||||
5. Test: `python scripts/cross_reference.py --emulator <name>`
|
||||
|
||||
Full walkthrough: [profiling guide]({SITE_URL}wiki/profiling/).
|
||||
|
||||
## File conventions
|
||||
|
||||
- `bios/Manufacturer/Console/filename` for canonical files
|
||||
- `bios/Manufacturer/Console/.variants/filename.sha1prefix` for alternate versions
|
||||
- Files >50 MB go in GitHub release assets (`large-files` release)
|
||||
- RPG Maker and ScummVM directories are excluded from deduplication
|
||||
- See the [documentation site](https://abdess.github.io/retrobios/) for full details
|
||||
- Two paths differing only by case break clones on Windows and macOS;
|
||||
`tests/test_no_case_collisions.py` enforces this
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
```bash
|
||||
python -m unittest discover tests
|
||||
python scripts/pipeline.py --offline
|
||||
```
|
||||
|
||||
## PR validation
|
||||
|
||||
CI computes SHA1/MD5/CRC32 for every new file, checks them against the platform
|
||||
configs, validates the YAML against the schemas, runs the test suite, and posts
|
||||
a report on the PR.
|
||||
|
||||
Contributors who add platform support are credited in the README,
|
||||
on the documentation site, and in the BIOS packs.
|
||||
"""
|
||||
|
||||
|
||||
|
||||
+161
-35
@@ -358,16 +358,23 @@ def generate_home(
|
||||
"",
|
||||
" | Platform | Extract to |",
|
||||
" |----------|-----------|",
|
||||
" | RetroArch / Lakka | `system/` |",
|
||||
" | RetroArch | `system/` |",
|
||||
" | Batocera | `/userdata/bios/` |",
|
||||
" | BizHawk | `Firmware/` |",
|
||||
" | EmuDeck | `Emulation/bios/` |",
|
||||
" | EmuDeck | `~/Emulation/bios/` |",
|
||||
" | Lakka | `/storage/system/` |",
|
||||
" | MiSTer FPGA | `/media/fat/games/` |",
|
||||
" | ROCKNIX | `/storage/roms/bios/` |",
|
||||
" | Recalbox | `/recalbox/share/bios/` |",
|
||||
" | RetroBat | `bios/` |",
|
||||
" | RetroDECK | `~/retrodeck/bios/` |",
|
||||
" | RetroDECK | `~/retrodeck/` |",
|
||||
" | RetroPie | `~/RetroPie/BIOS/` |",
|
||||
" | RomM | `bios/{platform_slug}/` |",
|
||||
"",
|
||||
" The RetroDECK pack already carries its own `bios/` folder, so it "
|
||||
"extracts one level above it. Every other pack extracts straight into "
|
||||
"the BIOS folder. [Full instructions per setup](which-pack.md).",
|
||||
"",
|
||||
]
|
||||
)
|
||||
|
||||
@@ -461,7 +468,7 @@ def generate_stats(stats: dict) -> str:
|
||||
# Platform pages
|
||||
|
||||
|
||||
def generate_platform_index(coverages: dict) -> str:
|
||||
def generate_platform_index(coverages: dict, registry: dict | None = None) -> str:
|
||||
total_files = sum(c["total"] for c in coverages.values())
|
||||
total_present = sum(c["present"] for c in coverages.values())
|
||||
total_verified = sum(c["verified"] for c in coverages.values())
|
||||
@@ -472,8 +479,8 @@ def generate_platform_index(coverages: dict) -> str:
|
||||
f"{len(coverages)} supported platforms with "
|
||||
f"{total_present:,} verified files.",
|
||||
"",
|
||||
"| Platform | Files | Verification | Download |",
|
||||
"|----------|-------|-------------|----------|",
|
||||
"| Platform | Files | Verification | Status | Download |",
|
||||
"|----------|-------|-------------|--------|----------|",
|
||||
]
|
||||
|
||||
mode_labels = {
|
||||
@@ -482,6 +489,7 @@ def generate_platform_index(coverages: dict) -> str:
|
||||
"existence": '<span class="rb-badge rb-badge-info">existence</span>',
|
||||
}
|
||||
|
||||
archived_any = False
|
||||
for name, cov in sorted(coverages.items(), key=lambda x: x[1]["platform"]):
|
||||
display = cov["platform"]
|
||||
|
||||
@@ -489,10 +497,16 @@ def generate_platform_index(coverages: dict) -> str:
|
||||
cov["mode"],
|
||||
f'<span class="rb-badge rb-badge-muted">{cov["mode"]}</span>',
|
||||
)
|
||||
status = (registry or {}).get(name, {}).get("status", "active")
|
||||
if status == "archived":
|
||||
archived_any = True
|
||||
status_html = '<span class="rb-badge rb-badge-muted">archived</span>'
|
||||
else:
|
||||
status_html = '<span class="rb-badge rb-badge-success">active</span>'
|
||||
|
||||
lines.append(
|
||||
f"| [{display}]({name}.md) | "
|
||||
f"{cov['present']:,} | {mode_html} | "
|
||||
f"{cov['present']:,} | {mode_html} | {status_html} | "
|
||||
f"[Pack]({RELEASE_URL}){{ .md-button .md-button--primary }} |"
|
||||
)
|
||||
|
||||
@@ -505,6 +519,15 @@ def generate_platform_index(coverages: dict) -> str:
|
||||
]
|
||||
)
|
||||
|
||||
if archived_any:
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"An archived platform keeps its configuration and still gets a "
|
||||
"pack, but upstream is no longer scraped on a schedule.",
|
||||
]
|
||||
)
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
@@ -547,6 +570,18 @@ def generate_platform_page(
|
||||
logo_md,
|
||||
]
|
||||
|
||||
if (registry or {}).get(name, {}).get("status") == "archived":
|
||||
lines.extend(
|
||||
[
|
||||
'!!! warning "Archived platform"',
|
||||
"",
|
||||
" The configuration is kept and packs are still built, but "
|
||||
"upstream is no longer scraped on a schedule, so the file list "
|
||||
"reflects the last sync rather than today's upstream.",
|
||||
"",
|
||||
]
|
||||
)
|
||||
|
||||
# Stat cards
|
||||
lines.extend(
|
||||
[
|
||||
@@ -2273,37 +2308,56 @@ def generate_contributing() -> str:
|
||||
1. Fork this repository
|
||||
2. Place the file in `bios/Manufacturer/Console/filename`
|
||||
3. Variants (alternate hashes for the same file): place in `bios/Manufacturer/Console/.variants/`
|
||||
4. Create a Pull Request - hashes are verified automatically
|
||||
4. Open a Pull Request - hashes are verified automatically and reported as a comment
|
||||
|
||||
The [dump provenance](provenance.md) page lists catalogued dumps still missing
|
||||
from the collection, with their hashes. A file matching one of those is the
|
||||
most useful contribution.
|
||||
|
||||
## Add a platform
|
||||
|
||||
1. Create a scraper in `scripts/scraper/` (inherit `BaseScraper`)
|
||||
2. Read the platform's upstream source code to understand its BIOS check logic
|
||||
3. Add entry to `platforms/_registry.yml`
|
||||
2. Read the platform's upstream source to determine how it checks BIOS files
|
||||
3. Add an entry to `platforms/_registry.yml`
|
||||
4. Generate the platform YAML config
|
||||
5. Test: `python scripts/verify.py --platform <name>`
|
||||
|
||||
Full walkthrough: [adding a platform](wiki/adding-a-platform.md).
|
||||
|
||||
## Add an emulator profile
|
||||
|
||||
1. Clone the emulator's source code
|
||||
2. Search for BIOS/firmware loading (grep for `bios`, `rom`, `firmware`, `fopen`)
|
||||
3. Document every file the emulator loads with source code references
|
||||
4. Write YAML to `emulators/<name>.yml`
|
||||
1. Clone the emulator's source code, upstream and libretro port
|
||||
2. Trace the file loading from the entry point, not from a keyword grep
|
||||
3. Document every file the code loads, with a `source_ref` line reference
|
||||
4. Write the YAML to `emulators/<name>.yml`
|
||||
5. Test: `python scripts/cross_reference.py --emulator <name>`
|
||||
|
||||
Full walkthrough: [profiling guide](wiki/profiling.md).
|
||||
|
||||
## File conventions
|
||||
|
||||
- `bios/Manufacturer/Console/filename` for canonical files
|
||||
- `bios/Manufacturer/Console/.variants/filename.sha1prefix` for alternate versions
|
||||
- Files >50 MB go in GitHub release assets (`large-files` release)
|
||||
- RPG Maker and ScummVM directories are excluded from deduplication
|
||||
- Two paths differing only by case break clones on Windows and macOS;
|
||||
`tests/test_no_case_collisions.py` enforces this
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
```bash
|
||||
python -m unittest discover tests
|
||||
python scripts/pipeline.py --offline
|
||||
```
|
||||
|
||||
## PR validation
|
||||
|
||||
The CI automatically:
|
||||
- Computes SHA1/MD5/CRC32 of new files
|
||||
- Checks against known hashes in platform configs
|
||||
- Reports coverage impact
|
||||
CI computes SHA1/MD5/CRC32 for every new file, checks them against the platform
|
||||
configs, validates the YAML against the schemas, runs the test suite, and posts
|
||||
a report on the PR.
|
||||
|
||||
Contributors who add platform support are credited in the README, on this site,
|
||||
and in the BIOS packs.
|
||||
"""
|
||||
|
||||
|
||||
@@ -2339,10 +2393,18 @@ def generate_wiki_data_model(db: dict, profiles: dict) -> str:
|
||||
' "md5": "...",',
|
||||
' "sha256": "...",',
|
||||
' "crc32": "...",',
|
||||
' "adler32": "..."',
|
||||
' "adler32": "...",',
|
||||
' "provenance": {',
|
||||
' "redump": {"dat": "...", "name": "...", "description": "..."}',
|
||||
" }",
|
||||
"}",
|
||||
"```",
|
||||
"",
|
||||
"`provenance` maps each catalog that lists the file to the DAT and entry "
|
||||
"it was matched against. It is present only when the file matches a "
|
||||
"snapshot under `provenance/`; the join runs by SHA1 first, then by "
|
||||
"MD5 + size. See [dump provenance](../provenance.md).",
|
||||
"",
|
||||
"### Indexes",
|
||||
"",
|
||||
"| Index | Entries | Purpose |",
|
||||
@@ -2359,13 +2421,18 @@ def generate_wiki_data_model(db: dict, profiles: dict) -> str:
|
||||
"",
|
||||
"1. Path suffix exact match (for regional variants with same filename)",
|
||||
"2. SHA1 exact match",
|
||||
"3. MD5 direct lookup (supports truncated Batocera 29-char MD5)",
|
||||
"4. Name + alias lookup without hash (existence mode)",
|
||||
"5. Name + alias with md5_composite / direct MD5 per candidate",
|
||||
"6. zippedFile content match via inner ROM MD5 index",
|
||||
"7. MAME clone fallback (deduped ZIP mapped to canonical name)",
|
||||
"8. Data directory scan (exact path then case-insensitive basename walk)",
|
||||
"9. Agnostic fallback (size-constrained match under system path prefix)",
|
||||
"3. SHA256 exact match (profiles whose upstream publishes SHA256)",
|
||||
"4. CRC32 + size exact match (CRC-only profiles; size confirms the match)",
|
||||
"5. MD5 direct lookup (supports truncated Batocera 29-char MD5)",
|
||||
"6. Name + alias lookup without hash (existence mode)",
|
||||
"7. Name + alias with md5_composite / direct MD5 per candidate",
|
||||
"8. zippedFile content match via inner ROM MD5 index",
|
||||
"9. MAME clone fallback (deduped ZIP mapped to canonical name)",
|
||||
"10. Data directory scan (exact path then case-insensitive basename walk)",
|
||||
"11. Agnostic fallback (size-constrained match under system path prefix)",
|
||||
"",
|
||||
"The first match wins. Steps and their return codes are described in "
|
||||
"[verification modes](verification-modes.md#file-resolution-chain).",
|
||||
"",
|
||||
"## Platform YAML",
|
||||
"",
|
||||
@@ -2389,9 +2456,15 @@ def generate_wiki_data_model(db: dict, profiles: dict) -> str:
|
||||
"Supports inheritance (`inherits: retroarch`) and shared groups",
|
||||
"(`includes: [group_name]` referencing `_shared.yml`).",
|
||||
"",
|
||||
"`base_destination` is the prefix the pack applies to every entry. It is",
|
||||
"empty when the upstream destinations already carry their own root, which",
|
||||
"is why the RetroDECK pack ships `bios/` and `roms/` at its top level.",
|
||||
"",
|
||||
"## Emulator YAML",
|
||||
"",
|
||||
f"**{len(profiles)}** profiles. Source-verified from emulator code.",
|
||||
f"**{len(profiles)}** profile files, **{len(unique_emulator_profiles(profiles))}** "
|
||||
"distinct emulators once aliases are folded in. Source-verified from "
|
||||
"emulator code.",
|
||||
"",
|
||||
"See the [profiling guide](profiling.md) for the full field reference.",
|
||||
"",
|
||||
@@ -2435,13 +2508,17 @@ def generate_which_pack() -> str:
|
||||
"""Generate the 'Which pack?' decision page."""
|
||||
rel = "https://github.com/Abdess/retrobios/releases"
|
||||
return f"""\
|
||||
# Getting started
|
||||
# Download
|
||||
|
||||
Some retro consoles need firmware files (commonly called BIOS) to run games.
|
||||
Without them, the emulator either refuses to start the game or runs it with
|
||||
reduced accuracy. This project collects and verifies those files so they are
|
||||
ready to use.
|
||||
|
||||
This page picks the right pack for a setup. For BIOS directory paths per
|
||||
platform, verification, and the CLI, see
|
||||
[Getting started](wiki/getting-started.md).
|
||||
|
||||
## Quick install
|
||||
|
||||
The installer detects the platform, finds the BIOS folder, downloads what
|
||||
@@ -2482,7 +2559,7 @@ extract the whole archive directly. To join the parts manually instead:
|
||||
| Setup | What it is | Pack | Extract to |
|
||||
|-------|-----------|------|-----------|
|
||||
| [EmuDeck](https://www.emudeck.com/) | Installs and configures multiple emulators, adds each game to the Steam library | [EmuDeck]({rel}) | `~/Emulation/bios/` |
|
||||
| [RetroDECK](https://retrodeck.net/) | Single Flatpak app, all emulators bundled, one-click install from Discover | [RetroDECK]({rel}) | `~/retrodeck/` |
|
||||
| [RetroDECK](https://retrodeck.net/) | Single Flatpak app, all emulators bundled, one-click install from Discover | [RetroDECK]({rel}) | `~/retrodeck/` (the pack carries its own `bios/`) |
|
||||
| RetroArch standalone | Installed from Discover, Steam, or Flatpak | [RetroArch]({rel}) | Open RetroArch > Settings > Directory > System, that is the folder |
|
||||
|
||||
### Windows
|
||||
@@ -2518,10 +2595,19 @@ extract the whole archive directly. To join the parts manually instead:
|
||||
| [Batocera](https://batocera.org/) | Easy setup, works on Pi 3/4/5 and many other boards (Odroid, etc.) | [Batocera]({rel}) | `/userdata/bios/` |
|
||||
| [Recalbox](https://www.recalbox.com/) | Plug-and-play experience, good for a first build | [Recalbox]({rel}) | `/recalbox/share/bios/` |
|
||||
|
||||
### Android handheld (Retroid Pocket, R36S, Miyoo, etc.)
|
||||
### Handhelds
|
||||
|
||||
Most Android handhelds run RetroArch. Download the [RetroArch pack]({rel})
|
||||
and extract into `RetroArch/system/` on internal storage or SD card.
|
||||
| Setup | What it is | Pack | Extract to |
|
||||
|-------|-----------|------|-----------|
|
||||
| Android (Retroid Pocket, Odin, etc.) | Most Android handhelds run RetroArch | [RetroArch]({rel}) | `RetroArch/system/` on internal storage or SD card |
|
||||
| [ROCKNIX](https://rocknix.org/) | Linux OS for ARM and x86 handhelds (RG35XX, RG552, Deck) | [ROCKNIX]({rel}) | `/storage/roms/bios/`, over SSH or the network share |
|
||||
| [Batocera](https://batocera.org/) | Also images many handhelds | [Batocera]({rel}) | `/userdata/bios/` |
|
||||
|
||||
### FPGA
|
||||
|
||||
| Setup | What it is | Pack | Extract to |
|
||||
|-------|-----------|------|-----------|
|
||||
| [MiSTer FPGA](https://mister-devel.github.io/MkDocs_MiSTer/) | Hardware-level recreation of consoles on a DE10-Nano board | [MiSTer FPGA]({rel}) | `/media/fat/games/`, one subfolder per core |
|
||||
|
||||
### Self-hosted ROM manager
|
||||
|
||||
@@ -2757,7 +2843,8 @@ def main():
|
||||
# Generate platform pages
|
||||
print("Generating platform pages...")
|
||||
write_if_changed(
|
||||
str(docs / "platforms" / "index.md"), generate_platform_index(coverages)
|
||||
str(docs / "platforms" / "index.md"),
|
||||
generate_platform_index(coverages, registry),
|
||||
)
|
||||
for name, cov in coverages.items():
|
||||
write_if_changed(
|
||||
@@ -2836,9 +2923,17 @@ def main():
|
||||
# Rewrite mkdocs.yml entirely (static config + generated nav)
|
||||
mkdocs_static = """\
|
||||
site_name: RetroBIOS
|
||||
site_description: Source-verified BIOS and firmware packs for RetroArch, Batocera,
|
||||
Recalbox, Lakka, RetroPie, EmuDeck, RetroBat, RetroDECK, RomM, BizHawk, ROCKNIX
|
||||
and MiSTer FPGA.
|
||||
site_url: https://abdess.github.io/retrobios/
|
||||
repo_url: https://github.com/Abdess/retrobios
|
||||
repo_name: Abdess/retrobios
|
||||
# Almost every page is generated from platforms/, emulators/ and database.json,
|
||||
# so a per-page edit link would point at a file that does not exist.
|
||||
edit_uri: ''
|
||||
copyright: MIT for the tooling. BIOS and firmware files are third-party system
|
||||
software, preserved for personal backup, archival and interoperability.
|
||||
theme:
|
||||
name: material
|
||||
palette:
|
||||
@@ -2862,24 +2957,46 @@ theme:
|
||||
icon:
|
||||
logo: material/chip
|
||||
features:
|
||||
- navigation.instant
|
||||
- navigation.instant.prefetch
|
||||
- navigation.instant.progress
|
||||
- navigation.tabs
|
||||
- navigation.tabs.sticky
|
||||
- navigation.sections
|
||||
- navigation.top
|
||||
- navigation.tracking
|
||||
- navigation.indexes
|
||||
# 400+ pages: pruning keeps the navigation out of every page's HTML.
|
||||
- navigation.prune
|
||||
- navigation.footer
|
||||
- search.suggest
|
||||
- search.highlight
|
||||
- search.share
|
||||
- content.code.copy
|
||||
- content.tabs.link
|
||||
- toc.follow
|
||||
extra_css:
|
||||
- stylesheets/extra.css
|
||||
extra:
|
||||
social:
|
||||
- icon: fontawesome/brands/github
|
||||
link: https://github.com/Abdess/retrobios
|
||||
name: RetroBIOS on GitHub
|
||||
markdown_extensions:
|
||||
- tables
|
||||
- abbr
|
||||
- admonition
|
||||
- attr_list
|
||||
- def_list
|
||||
- footnotes
|
||||
- md_in_html
|
||||
- tables
|
||||
- toc:
|
||||
permalink: true
|
||||
- pymdownx.details
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.keys
|
||||
- pymdownx.superfences:
|
||||
custom_fences:
|
||||
- name: mermaid
|
||||
@@ -2889,6 +3006,13 @@ markdown_extensions:
|
||||
alternate_style: true
|
||||
plugins:
|
||||
- search
|
||||
# Link rot fails the build: deploy-site.yml runs `mkdocs build --strict`.
|
||||
# omitted_files stays at its default (info) so a stale local docs/ does not
|
||||
# break a local build; CI regenerates docs/ from scratch anyway.
|
||||
validation:
|
||||
absolute_links: warn
|
||||
unrecognized_links: warn
|
||||
anchors: warn
|
||||
"""
|
||||
write_if_changed("mkdocs.yml", mkdocs_static + nav_yaml)
|
||||
|
||||
@@ -2902,7 +3026,9 @@ plugins:
|
||||
+ 1
|
||||
+ len(profiles) # emulator index + detail
|
||||
+ 1 # gap analysis
|
||||
+ 14 # wiki pages (copied from wiki/ + generated data-model)
|
||||
+ 1 # which-pack
|
||||
+ len(list(Path("wiki").glob("*.md"))) # wiki pages copied verbatim
|
||||
+ 1 # generated wiki/data-model
|
||||
+ 1 # contributing
|
||||
)
|
||||
print(f"\nGenerated {total_pages} pages in {args.docs_dir}/")
|
||||
|
||||
@@ -7,6 +7,10 @@ Replicates the exact verification logic of each platform:
|
||||
- Recalbox: MD5 + mandatory/hashMatchMandatory, 3-color severity (Bios.cpp:109-130)
|
||||
- RetroBat: same as Batocera
|
||||
- EmuDeck: MD5 whitelist per system
|
||||
- RetroDECK: MD5 per file via component manifests
|
||||
- RomM: size + any hash, no ZIP inspection (firmware.py verify_file_hashes)
|
||||
- ROCKNIX: MD5 + checkInsideZip with altmd5 (rocknix-systems checkBios)
|
||||
- MiSTer FPGA: MD5 of the file at its destination, no ZIP inspection
|
||||
- BizHawk: SHA1 firmware hash verification
|
||||
|
||||
Cross-references emulator profiles to detect undeclared files used by available cores.
|
||||
|
||||
+54
-11
@@ -117,6 +117,11 @@ This fetches from upstream and prints a summary without writing anything.
|
||||
|
||||
Add an entry to `platforms/_registry.yml` under the `platforms:` key.
|
||||
|
||||
The registry describes *where the data comes from*. How the platform behaves
|
||||
at runtime (`verification_mode`, `base_destination`) belongs to the platform
|
||||
YAML, because that is what `verify.py` and `generate_pack.py` load. Keeping the
|
||||
two apart means a re-scrape can rewrite the YAML without touching the registry.
|
||||
|
||||
### Required fields
|
||||
|
||||
```yaml
|
||||
@@ -124,17 +129,22 @@ platforms:
|
||||
myplatform:
|
||||
config: myplatform.yml # platform YAML filename in platforms/
|
||||
status: active # active or archived
|
||||
scraper: myplatform # matches PLATFORM_NAME in the scraper
|
||||
scraper: myplatform # matches PLATFORM_NAME, null if hand-maintained
|
||||
schedule: weekly # weekly, monthly, or null for no auto-scrape
|
||||
source_url: https://... # upstream data URL
|
||||
source_format: json # json, xml, clrmamepro_dat, python_dict, bash_script+csv, csharp_firmware_database, github_component_manifests
|
||||
source_format: json # json, xml, clrmamepro_dat, python_dict,
|
||||
# bash_script+csv, csharp_firmware_database,
|
||||
# github_component_manifests, mister_downloader_db
|
||||
hash_type: md5 # primary hash in the upstream data
|
||||
verification_mode: md5 # how the platform checks files: existence, md5, sha1
|
||||
base_destination: bios # where files go on disk
|
||||
cores: # which emulator profiles apply
|
||||
- core_a
|
||||
- core_b
|
||||
```
|
||||
|
||||
A platform without a scraper (`scraper: null`, `schedule: null`) keeps its YAML
|
||||
as a hand-maintained file. RetroPie is the example: it inherits RetroArch's
|
||||
file set and has nothing of its own to fetch.
|
||||
|
||||
The `cores` field determines which emulator profiles are resolved for this platform.
|
||||
Three strategies exist:
|
||||
|
||||
@@ -148,11 +158,15 @@ Three strategies exist:
|
||||
|
||||
```yaml
|
||||
logo: https://... # SVG or PNG for UI/docs
|
||||
schedule: weekly # scrape frequency: weekly, monthly, or null
|
||||
inherits_from: retroarch # inherit systems/cores from another platform
|
||||
case_insensitive_fs: true # if the platform runs on case-insensitive filesystems
|
||||
target_scraper: myplatform_targets # hardware target scraper name
|
||||
source_wiki: https://... # human-readable upstream reference
|
||||
target_scraper: myplatform_targets # hardware target scraper name, null if static
|
||||
target_source: https://... # target data source URL
|
||||
contributed_by: # credited in README, site, and packs
|
||||
- username: someone
|
||||
contribution: platform support
|
||||
pr: 50
|
||||
install:
|
||||
detect: # auto-detection for install.py
|
||||
- os: linux
|
||||
@@ -161,6 +175,25 @@ Three strategies exist:
|
||||
parse_key: bios_directory
|
||||
```
|
||||
|
||||
### Platform YAML header
|
||||
|
||||
The runtime behavior lives at the top of `platforms/myplatform.yml`:
|
||||
|
||||
```yaml
|
||||
platform: MyPlatform
|
||||
verification_mode: md5 # existence, md5, or sha1
|
||||
hash_type: md5 # primary hash stored in the file entries
|
||||
base_destination: bios # root directory the pack extracts into
|
||||
inherits: retroarch # optional, merges the parent's systems
|
||||
systems:
|
||||
sony-playstation:
|
||||
files: [...]
|
||||
```
|
||||
|
||||
`base_destination` is what the pack prefixes every entry with. Leave it empty
|
||||
when the upstream destinations already carry their own root, as RetroDECK does
|
||||
with `bios/` and `roms/`.
|
||||
|
||||
### Inheritance
|
||||
|
||||
If the new platform inherits from an existing one (e.g. Lakka inherits RetroArch),
|
||||
@@ -191,14 +224,18 @@ python scripts/verify.py --platform myplatform --verbose
|
||||
## Step 4: Add verification logic
|
||||
|
||||
Check how the platform verifies BIOS files by reading its source code.
|
||||
The `verification_mode` in the registry tells `verify.py` which strategy to use:
|
||||
The `verification_mode` in the platform YAML tells `verify.py` which strategy
|
||||
to use:
|
||||
|
||||
| Mode | Behavior | Example platforms |
|
||||
|------|----------|-------------------|
|
||||
| `existence` | File must exist, no hash check | RetroArch, Lakka, RetroPie |
|
||||
| `md5` | MD5 must match the declared hash | Batocera, Recalbox, RetroBat, EmuDeck, RetroDECK |
|
||||
| `md5` | MD5 must match the declared hash | Batocera, Recalbox, RetroBat, EmuDeck, RetroDECK, RomM, ROCKNIX, MiSTer FPGA |
|
||||
| `sha1` | SHA1 must match | BizHawk |
|
||||
|
||||
Inheriting platforms take the parent's mode: Lakka and RetroPie do not declare
|
||||
one, they get `existence` from RetroArch.
|
||||
|
||||
If the platform has unique verification behavior (e.g. Batocera's `checkInsideZip`,
|
||||
Recalbox's multi-hash comma-separated MD5, RomM's size + any-hash), add the logic
|
||||
to `verify.py` in the platform-specific verification path.
|
||||
@@ -324,13 +361,19 @@ python scripts/pipeline.py --offline
|
||||
This executes in sequence:
|
||||
|
||||
1. `generate_db.py` - rebuild `database.json` from `bios/`
|
||||
1b. `provenance_report.py` - dump-catalog coverage from `provenance/`
|
||||
2. `refresh_data_dirs.py` - update data directories (skipped with `--offline`)
|
||||
3. `verify.py --all` - verify all platforms including the new one
|
||||
4. `generate_pack.py --all` - build ZIP packs + install manifests
|
||||
4. `generate_pack.py --all` - build ZIP packs + install and target manifests
|
||||
5. Consistency check - verify counts match between verify and pack
|
||||
6. Pack integrity - extract ZIPs and verify hashes per platform mode
|
||||
7. `generate_readme.py` - regenerate README
|
||||
8. `generate_site.py` - regenerate documentation site
|
||||
7. `generate_readme.py` - regenerate README and CONTRIBUTING
|
||||
8. `generate_site.py` - regenerate documentation site and `mkdocs.yml`
|
||||
|
||||
A new platform is archived-by-default nowhere: `status: active` puts it in
|
||||
`list_platforms.py`, the CI matrix, and the weekly scrape. Set
|
||||
`status: archived` to keep the configuration without the automation; archived
|
||||
platforms are excluded unless `--all` or `--include-archived` is passed.
|
||||
|
||||
Check the output for:
|
||||
|
||||
|
||||
@@ -182,6 +182,8 @@ normalize them into `BiosRequirement` entries.
|
||||
| C# source | BizHawk `FirmwareDatabase.cs` | Regex for method calls and string literals |
|
||||
| C source | MAME/FBNeo drivers | Use `mame_parser` or `fbneo_parser` (see below) |
|
||||
| JSON (GitHub API) | RetroDECK component manifests | `json.loads()` per manifest file |
|
||||
| JSON in ZIP | MiSTer `bios_db.json.zip` | `zipfile` then `json.loads()` |
|
||||
| Logiqx XML | Redump, No-Intro, TOSEC DATs | Use `logiqx_parser` (see below) |
|
||||
|
||||
### System ID mapping
|
||||
|
||||
@@ -311,6 +313,22 @@ game (
|
||||
Produces `DatRom` dataclass instances with `name`, `size`, `crc32`, `md5`, `sha1`,
|
||||
and `system` fields. The `libretro_scraper` uses this parser.
|
||||
|
||||
### logiqx_parser
|
||||
|
||||
Parses Logiqx XML DATs, the format the dump-preservation catalogs publish:
|
||||
|
||||
```xml
|
||||
<datafile>
|
||||
<header><name>Sony - PlayStation - BIOS Images</name></header>
|
||||
<game name="..."><rom name="..." size="524288" crc="..." md5="..." sha1="..."/></game>
|
||||
</datafile>
|
||||
```
|
||||
|
||||
Used by `redump_dat_scraper` (fetches the four Redump BIOS DATs directly) and
|
||||
by `dat_pack_importer` (reads a locally downloaded No-Intro or TOSEC pack,
|
||||
since neither can be fetched without a browser). Both write a snapshot to
|
||||
`provenance/`; they do not touch platform YAMLs.
|
||||
|
||||
### mame_parser
|
||||
|
||||
Parses MAME C source files to extract BIOS root sets. Handles:
|
||||
|
||||
@@ -57,6 +57,41 @@ For MD5-mode platforms (Batocera), all declared files are treated as required un
|
||||
explicitly marked optional.
|
||||
|
||||
|
||||
## Pack Source Variants
|
||||
|
||||
A pack is built from two file sources: the platform's own declared list
|
||||
(layer 1) and the requirements of the emulator profiles that apply to it
|
||||
(layer 3). `--source` selects which of the two contributes.
|
||||
|
||||
| `--source` | Contents | Name suffix |
|
||||
|-----------|----------|-------------|
|
||||
| `full` (default) | platform baseline plus everything its cores need | none |
|
||||
| `platform` | only what the platform itself declares | `_Platform` |
|
||||
| `truth` | only what the emulator profiles require | `_Truth` |
|
||||
|
||||
`--required-only` crosses with each of them and adds `_Required`:
|
||||
|
||||
```bash
|
||||
python scripts/generate_pack.py --platform retroarch --source platform
|
||||
python scripts/generate_pack.py --platform retroarch --source truth --required-only
|
||||
python scripts/generate_pack.py --all --all-variants --output-dir dist/
|
||||
```
|
||||
|
||||
`--all-variants` builds all six combinations in one run, producing names like
|
||||
`RetroArch_Lakka_v1.22.2_BIOS_Pack.zip`,
|
||||
`RetroArch_Lakka_v1.22.2_Platform_BIOS_Pack.zip` and
|
||||
`RetroArch_Lakka_v1.22.2_Truth_Required_BIOS_Pack.zip`.
|
||||
|
||||
Which one to pick: `full` is the one to ship, since it covers alternate cores
|
||||
and optional firmware. `platform` is much smaller and matches exactly what the
|
||||
frontend checks for, which suits SD cards and handhelds. `truth` is a
|
||||
diagnostic build: it shows what the emulator source code asks for, independent
|
||||
of what the platform declares, and is how a gap between the two becomes
|
||||
visible.
|
||||
|
||||
Variants compose with `--split`, `--target` and `--manifest`.
|
||||
|
||||
|
||||
## Hardware Target Filtering
|
||||
|
||||
### What targets are
|
||||
|
||||
+64
-12
@@ -12,6 +12,7 @@ platforms/ one YAML config per platform (scraped from upstream)
|
||||
_registry.yml platform metadata (logos, scrapers, status, install config)
|
||||
_data_dirs.yml data directory definitions (Dolphin Sys, PPSSPP...)
|
||||
targets/ hardware target configs + _overrides.yml
|
||||
provenance/ dump-catalog snapshots (redump, no-intro, tosec)
|
||||
scripts/ all tooling (Python, pyyaml only dependency)
|
||||
scraper/ upstream scrapers (libretro, batocera, recalbox...)
|
||||
scraper/targets/ hardware target scrapers (retroarch, batocera, emudeck, retropie)
|
||||
@@ -19,13 +20,20 @@ scripts/ all tooling (Python, pyyaml only dependency)
|
||||
install/ JSON install manifests per platform
|
||||
targets/ JSON target manifests per platform (cores per architecture)
|
||||
data/ cached data directories (not BIOS, fetched at build)
|
||||
schemas/ JSON schemas for validation
|
||||
tests/ E2E test suite with synthetic fixtures
|
||||
schemas/ JSON schemas for platform and emulator YAML (checked in CI)
|
||||
tests/ test suite with synthetic fixtures
|
||||
wiki/ hand-written documentation sources
|
||||
docs/ generated MkDocs site (gitignored, rebuilt in CI)
|
||||
_mame_clones.json MAME parent/clone set mappings
|
||||
database.json file index built from bios/ (SHA1 primary key)
|
||||
dist/ generated packs (gitignored)
|
||||
.cache/ hash cache and large file downloads (gitignored)
|
||||
```
|
||||
|
||||
`docs/` is never edited by hand: `generate_site.py` rebuilds it from
|
||||
`platforms/`, `emulators/`, `database.json`, and the `wiki/` sources.
|
||||
Documentation changes belong in `wiki/` or in the generator.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
@@ -35,8 +43,13 @@ Upstream sources Scrapers parse generate_db.py scans
|
||||
es_bios.xml (recalbox) (SHA1 primary key,
|
||||
core-info .info files indexes: by_md5, by_name,
|
||||
FirmwareDatabase.cs by_crc32, by_sha256, by_path_suffix)
|
||||
bios_db.json.zip (MiSTer)
|
||||
MAME/FBNeo source
|
||||
|
||||
provenance/*.json generate_db.py joins database.json entries carry
|
||||
redump, no-intro, by SHA1, then by a provenance field naming
|
||||
tosec snapshots MD5 + size the catalogs that list them
|
||||
|
||||
emulators/*.yml verify.py checks generate_pack.py resolves
|
||||
source-verified platform-native files by hash, builds ZIP
|
||||
from code verification packs per platform
|
||||
@@ -46,13 +59,15 @@ truth.py generates diff_truth.py export_native.py
|
||||
emulator profiles scraped platform (DAT, XML, JSON, Bash)
|
||||
```
|
||||
|
||||
Pipeline runs all steps in sequence: DB, data dirs, MAME/FBNeo hashes,
|
||||
verify, packs, install manifests, target manifests, consistency check,
|
||||
pack integrity, README, site. See [tools](tools.md) for the full pipeline reference.
|
||||
Pipeline runs all steps in sequence: DB, provenance report, data dirs,
|
||||
MAME/FBNeo hashes, verify, packs, install manifests, target manifests,
|
||||
consistency check, pack integrity, README, site. See [tools](tools.md)
|
||||
for the full pipeline reference.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[generate_db] --> B[refresh_data_dirs]
|
||||
A[generate_db] --> A2[provenance report]
|
||||
A2 --> B[refresh_data_dirs]
|
||||
B --> C[MAME/FBNeo hashes]
|
||||
C --> D[verify --all]
|
||||
D --> E[generate_pack --all]
|
||||
@@ -106,6 +121,26 @@ graph TD
|
||||
style ZIP fill:#2d333b,stroke:#adbac7,color:#adbac7
|
||||
```
|
||||
|
||||
## Dump-catalog provenance
|
||||
|
||||
Snapshots of the dump-preservation catalogs (Redump, No-Intro, TOSEC) live in
|
||||
`provenance/` as committed JSON. `generate_db.py` joins them against the
|
||||
collection by SHA1, falling back to MD5 + size, and writes a `provenance` field
|
||||
on each matching database entry. The system pages render it as a verified dump
|
||||
badge.
|
||||
|
||||
Provenance is an annotation, never an authority. It answers "does this file
|
||||
byte-match a catalogued dump", which is a different question from "does the
|
||||
emulator accept it". When the two disagree, pack contents and verification
|
||||
follow the emulator source code.
|
||||
|
||||
`provenance_report.py` reports the reverse direction: catalog entries absent
|
||||
from the collection, which become acquisition targets. A DAT counts as covered
|
||||
when the collection holds at least one of its entries; entries from DATs the
|
||||
collection does not cover are counted but not listed, since No-Intro tags every
|
||||
non-game dump as `[BIOS]`, including tens of thousands of digital-distribution
|
||||
entries that are out of scope here.
|
||||
|
||||
## Pack grouping
|
||||
|
||||
Platforms that produce identical packs are grouped automatically.
|
||||
@@ -174,7 +209,11 @@ graph TD
|
||||
S0 -- yes --> EXACT([exact])
|
||||
S0 -- no --> S1{SHA1<br/>exact match?}
|
||||
S1 -- yes --> EXACT
|
||||
S1 -- no --> S2{MD5 direct<br/>or truncated?}
|
||||
S1 -- no --> S1B{SHA256<br/>exact match?}
|
||||
S1B -- yes --> EXACT
|
||||
S1B -- no --> S1C{CRC32 + size,<br/>no stronger hash?}
|
||||
S1C -- yes --> EXACT
|
||||
S1C -- no --> S2{MD5 direct<br/>or truncated?}
|
||||
S2 -- yes --> MD5([md5_exact])
|
||||
S2 -- no --> S3{name + aliases<br/>no MD5?}
|
||||
S3 -- yes --> EXACT
|
||||
@@ -236,20 +275,33 @@ user's platform, filter files by hardware target, and download with SHA1 verific
|
||||
|
||||
## Tests
|
||||
|
||||
5 test files, 259 tests total:
|
||||
10 test files, 400 tests total:
|
||||
|
||||
| File | Tests | Coverage |
|
||||
|------|-------|----------|
|
||||
| `test_e2e.py` | 196 | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters |
|
||||
| `test_pack_integrity.py` | 8 | extract ZIP packs to disk, verify paths + hashes per platform's native mode |
|
||||
| `test_e2e.py` | 217 | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters |
|
||||
| `test_install.py` | 70 | `install.py` platform detection, config-file parsing, manifest handling |
|
||||
| `test_provenance.py` | 29 | Logiqx/Redump parsers, DAT pack import, provenance join, coverage report |
|
||||
| `test_mame_parser.py` | 22 | BIOS root set detection, ROM block parsing, macro expansion |
|
||||
| `test_hash_merge.py` | 17 | MAME/FBNeo YAML merge, diff detection, formatting preservation |
|
||||
| `test_fbneo_parser.py` | 16 | BIOS set detection, ROM info parsing |
|
||||
| `test_hash_merge.py` | 17 | MAME/FBNeo YAML merge, diff detection |
|
||||
| `test_profile_refs.py` | 12 | `check_profile_refs` pure functions (no network) |
|
||||
| `test_pack_integrity.py` | 8 | extract ZIP packs to disk, verify paths + hashes per platform's native mode |
|
||||
| `test_torrentzip.py` | 8 | TorrentZip builder against real MAME romsets |
|
||||
| `test_no_case_collisions.py` | 1 | guard against case-colliding paths in `bios/` |
|
||||
|
||||
```bash
|
||||
python -m unittest tests.test_e2e -v
|
||||
python -m unittest discover tests -v # full suite
|
||||
python -m unittest tests.test_e2e -v # single module
|
||||
```
|
||||
|
||||
`test_e2e.py`, `test_install.py`, `test_provenance.py`, the parser tests and
|
||||
`test_profile_refs.py` run on synthetic fixtures with no network and no real
|
||||
BIOS files. `test_pack_integrity.py`, `test_torrentzip.py` and
|
||||
`test_no_case_collisions.py` read the working tree and skip when the data they
|
||||
need is absent. See the [testing guide](testing-guide.md) for the fixture
|
||||
pattern and how to add a test.
|
||||
|
||||
## CI workflows
|
||||
|
||||
| Workflow | File | Trigger | Role |
|
||||
|
||||
+22
-12
@@ -8,7 +8,7 @@ Most likely a missing or incorrect BIOS file. Run verification for your platform
|
||||
python scripts/verify.py --platform retroarch
|
||||
```
|
||||
|
||||
Look for MISSING or HASH MISMATCH entries. If a file shows HASH MISMATCH, you have a BIOS file but it's the wrong version or a bad dump. Replace it with one that matches the expected hash.
|
||||
Look for `missing` or `untested` entries. `missing` means the file is not there at all. `untested` means the file is there but its hash is not the expected one, so it is the wrong version or a bad dump: replace it with one matching the hash listed on the system page.
|
||||
|
||||
Some cores also support HLE (see below), so a missing BIOS may not always be the cause. Check the emulator's logs for error messages.
|
||||
|
||||
@@ -57,6 +57,9 @@ absent from the collection are tracked as acquisition targets. When
|
||||
the two views differ, this project follows the code, because that is
|
||||
what decides whether your emulator boots.
|
||||
|
||||
The [dump provenance](../provenance.md) page has the current coverage
|
||||
per catalog and the full list of catalogued dumps still missing.
|
||||
|
||||
## Which MAME version do the arcade BIOS sets match?
|
||||
|
||||
Arcade BIOS sets are coupled to the romset version, so there is one
|
||||
@@ -99,7 +102,7 @@ This is the most contested area. The legal position:
|
||||
|
||||
- **Keys are not copyrightable.** Encryption keys are mathematical values, not creative expression. Copyright protects original works of authorship; a 256-bit number does not meet the threshold of originality. *Bernstein v. DOJ* (1996) established that code and algorithms are protected speech, and the mere publication of numeric values cannot be restricted under copyright.
|
||||
- **DMCA 1201(f) interoperability exemption.** The DMCA prohibits circumvention of technological protection measures, but Section 1201(f) explicitly permits circumvention for the purpose of achieving interoperability between programs. Emulators require these keys to decrypt and run legally purchased game software. The keys enable interoperability, not piracy.
|
||||
- **Library of Congress DMCA exemptions.** The triennial rulemaking process has repeatedly expanded exemptions for video game preservation. The 2024 exemption (37 CFR 201.40) covers circumvention for preservation of software and video games, including when the original hardware is no longer available.
|
||||
- **Library of Congress DMCA exemptions.** The triennial rulemaking process has granted and renewed exemptions for video game and software preservation. The exemptions at 37 CFR 201.40 let an eligible library, archive or museum circumvent access controls to preserve a lawfully acquired video game whose external server support has ended, and to preserve computer programs generally, with access limited to the institution's premises. The Ninth Triennial Proceeding (2024 cycle) renewed those exemptions but declined to extend them to off-premises remote access, so the direction of travel favors preservation without having settled it.
|
||||
- **Keys derived from consumer hardware.** These keys are extracted from retail hardware owned by consumers. Once a product is sold, the manufacturer cannot indefinitely control how the purchaser uses or examines their own property. *Chamberlain v. Skylink* (2004) held that using a product in a way the manufacturer dislikes is not automatically a DMCA violation.
|
||||
- **No trade secret protection.** For keys to qualify as trade secrets, the holder must take reasonable steps to maintain secrecy. Keys embedded in millions of consumer devices and widely published online do not meet this standard.
|
||||
|
||||
@@ -123,23 +126,30 @@ The project tries to archive files while they are still available rather than af
|
||||
|
||||
## What's a hash/checksum?
|
||||
|
||||
A hash is a fixed-length fingerprint computed from a file's contents. If even one byte differs, the hash changes completely. The project uses three types:
|
||||
A hash is a fixed-length fingerprint computed from a file's contents. If even one byte differs, the hash changes completely. Every file in the database carries these:
|
||||
|
||||
| Type | Length | Example |
|
||||
|------|--------|---------|
|
||||
| MD5 | 32 hex chars | `924e392ed05558ffdb115408c263dccf` |
|
||||
| SHA1 | 40 hex chars | `10155d8d6e6e832d8ea1571511e40dfb15fede05` |
|
||||
| CRC32 | 8 hex chars | `2F468B96` |
|
||||
| Type | Length | Example | Used for |
|
||||
|------|--------|---------|----------|
|
||||
| SHA1 | 40 hex chars | `10155d8d6e6e832d8ea1571511e40dfb15fede05` | database primary key, BizHawk, installer downloads |
|
||||
| MD5 | 32 hex chars | `924e392ed05558ffdb115408c263dccf` | most platform verification |
|
||||
| SHA256 | 64 hex chars | `9a1c...` | emulator profiles whose upstream publishes SHA256 |
|
||||
| CRC32 | 8 hex chars | `2F468B96` | ROM-set matching, arcade DATs |
|
||||
|
||||
Different platforms use different hash types for verification. Batocera uses MD5, RetroArch checks existence only, BizHawk uses SHA1, and RomM uses MD5.
|
||||
Emulator profiles add Adler-32 where the code checks it, which is how Dolphin
|
||||
validates its IPL files.
|
||||
|
||||
Verification uses whichever one the platform itself uses: MD5 for Batocera,
|
||||
RetroBat, Recalbox, EmuDeck, RetroDECK, RomM, ROCKNIX and MiSTer FPGA, SHA1 for
|
||||
BizHawk, and nothing at all for RetroArch, Lakka and RetroPie, which only check
|
||||
that the file exists.
|
||||
|
||||
## Why does my verification report say UNTESTED?
|
||||
|
||||
UNTESTED means the file exists on disk but its hash does not match the expected value. This happens on MD5/SHA1-mode platforms (Batocera, Recalbox, BizHawk, etc.) when the file is present but contains different data than what the platform declares.
|
||||
`untested` means the file exists on disk but its hash does not match the expected value. This happens on MD5 and SHA1 platforms (Batocera, Recalbox, BizHawk, ROCKNIX, MiSTer FPGA, and the rest) when the file is present but contains different data than what the platform declares.
|
||||
|
||||
On existence-mode platforms (RetroArch, Lakka, RetroPie), files are never UNTESTED because the platform only checks presence, not content. Those files show as OK if present.
|
||||
On existence-mode platforms (RetroArch, Lakka, RetroPie), files are never `untested` because the platform only checks presence, not content. Those files show as `ok` if present, whatever they contain.
|
||||
|
||||
Running `verify.py --emulator <core> --verbose` shows the emulator-level ground truth, which can confirm whether the file's hash matches what the source code expects.
|
||||
Running `verify.py --emulator <core> --verbose` shows the emulator-level ground truth, which can confirm whether the file's hash matches what the source code expects. On an existence platform, that verbose report is the only thing that can tell you the file is wrong.
|
||||
|
||||
## Can I use BIOS from one platform on another?
|
||||
|
||||
|
||||
+86
-13
@@ -8,27 +8,50 @@ BIOS files are firmware dumps from original console hardware. Emulators need the
|
||||
|
||||
Three ways to get BIOS files in place, from easiest to most manual.
|
||||
|
||||
### Option 1: install.py (recommended)
|
||||
### Option 1: the installer (recommended)
|
||||
|
||||
Self-contained Python script, no dependencies beyond Python 3.10+. Auto-detects your platform and BIOS directory.
|
||||
One command, nothing to clone. It fetches `install.py`, detects the platform
|
||||
and its BIOS directory, downloads only what is missing, and verifies SHA1
|
||||
checksums before writing anything.
|
||||
|
||||
```bash
|
||||
# Linux / macOS / Steam Deck
|
||||
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
|
||||
|
||||
# Windows (PowerShell)
|
||||
irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex
|
||||
```
|
||||
|
||||
Running `install.py` directly works the same way and needs nothing beyond
|
||||
Python 3.10+:
|
||||
|
||||
```bash
|
||||
python install.py
|
||||
```
|
||||
|
||||
Override detection if needed:
|
||||
Override detection when needed:
|
||||
|
||||
```bash
|
||||
python install.py --platform retroarch --dest ~/custom/bios
|
||||
python install.py --check # verify existing files without downloading
|
||||
python install.py --list-platforms # show supported platforms
|
||||
python install.py --target switch # keep only files for that hardware
|
||||
python install.py --check # verify existing files, download nothing
|
||||
python install.py --list-platforms # supported platforms and what was detected
|
||||
python install.py --list-targets # hardware targets for a platform
|
||||
python install.py --jobs 4 # parallel downloads (default 8)
|
||||
python install.py --verbose
|
||||
```
|
||||
|
||||
The installer downloads files from GitHub releases, verifies SHA1 checksums, and places them in the correct directory.
|
||||
Arguments pass through the one-liner too, which is how you target an SD card
|
||||
mounted on another machine:
|
||||
|
||||
### Option 2: download.sh (Linux/macOS)
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh \
|
||||
| sh -s -- --platform retroarch --dest /path/to/sdcard
|
||||
```
|
||||
|
||||
One-liner for systems with `curl` or `wget`:
|
||||
### Option 2: download.sh (Linux/macOS, from a clone)
|
||||
|
||||
Downloads a whole pack rather than the missing files. Needs `curl` and `unzip`:
|
||||
|
||||
```bash
|
||||
bash scripts/download.sh retroarch ~/RetroArch/system/
|
||||
@@ -41,6 +64,11 @@ bash scripts/download.sh --list # show available packs
|
||||
2. Download the ZIP pack for your platform
|
||||
3. Extract to the BIOS directory listed below
|
||||
|
||||
Packs over 2 GB are split into numbered volumes (`.zip.001`, `.zip.002`).
|
||||
Download every part and open the `.001` file with 7-Zip or PeaZip, which
|
||||
extracts the whole set. See [Download](../which-pack.md) for the pack
|
||||
variants and per-setup instructions.
|
||||
|
||||
## BIOS directory by platform
|
||||
|
||||
### RetroArch
|
||||
@@ -49,13 +77,19 @@ RetroArch uses the `system_directory` setting in `retroarch.cfg`. Default locati
|
||||
|
||||
| OS | Default path |
|
||||
|----|-------------|
|
||||
| Windows | `%APPDATA%\RetroArch\system\` |
|
||||
| Windows (installer) | `%APPDATA%\RetroArch\system\` |
|
||||
| Windows (portable .7z) | `system\` next to `retroarch.exe` |
|
||||
| Linux | `~/.config/retroarch/system/` |
|
||||
| Linux (Flatpak) | `~/.var/app/org.libretro.RetroArch/config/retroarch/system/` |
|
||||
| macOS | `~/Library/Application Support/RetroArch/system/` |
|
||||
| Steam Deck | `~/.var/app/org.libretro.RetroArch/config/retroarch/system/` |
|
||||
| Steam Deck (Flatpak) | `~/.var/app/org.libretro.RetroArch/config/retroarch/system/` |
|
||||
| Android | `/storage/emulated/0/RetroArch/system/` |
|
||||
|
||||
Windows has two layouts because the installer stores its data under `%APPDATA%`
|
||||
while the portable archive keeps everything inside the folder you extracted it
|
||||
to. If both exist on the machine, the one RetroArch actually reads is the one
|
||||
shown in the UI.
|
||||
|
||||
To check your actual path: open RetroArch, go to **Settings > Directory > System/BIOS**, or look for `system_directory` in `retroarch.cfg`.
|
||||
|
||||
### Batocera
|
||||
@@ -88,6 +122,13 @@ Relative to the RetroBat installation directory (e.g., `C:\RetroBat\bios\`).
|
||||
~/retrodeck/bios/
|
||||
```
|
||||
|
||||
On a MicroSD install the root moves, e.g. `/run/media/mmcblk0p1/retrodeck/bios/`.
|
||||
|
||||
The RetroDECK pack is the exception to "extract into the BIOS directory": its
|
||||
entries already start with `bios/` (and one with `roms/`), so extract it into
|
||||
`~/retrodeck/` and the files land in the right place. Extracting into
|
||||
`~/retrodeck/bios/` would create `~/retrodeck/bios/bios/`.
|
||||
|
||||
### EmuDeck
|
||||
|
||||
```
|
||||
@@ -110,6 +151,29 @@ Accessible via SSH or Samba.
|
||||
~/RetroPie/BIOS/
|
||||
```
|
||||
|
||||
RetroPie is archived in this project: its configuration is kept and packs are
|
||||
still built, but the upstream data is no longer scraped on a schedule. It
|
||||
inherits RetroArch's file set, so the RetroArch pack applies as well.
|
||||
|
||||
### ROCKNIX
|
||||
|
||||
```
|
||||
/storage/roms/bios/
|
||||
```
|
||||
|
||||
Accessible via SSH or Samba, like Lakka.
|
||||
|
||||
### MiSTer FPGA
|
||||
|
||||
```
|
||||
/media/fat/games/
|
||||
```
|
||||
|
||||
Files go under the per-core subdirectory the MiSTer BIOS database declares
|
||||
(e.g. `/media/fat/games/3DO/boot.rom`). Only the entries the BIOS database
|
||||
lists with a download URL are in scope: the rest ship with the MiSTer
|
||||
distribution and install themselves.
|
||||
|
||||
### BizHawk
|
||||
|
||||
```
|
||||
@@ -120,12 +184,14 @@ Relative to the BizHawk installation directory.
|
||||
|
||||
### RomM
|
||||
|
||||
BIOS files are managed through the RomM web interface. Check the
|
||||
BIOS files live in the RomM library under `bios/{platform_slug}/`, one
|
||||
subfolder per system, and are managed through the web interface. Check the
|
||||
[RomM documentation](https://github.com/rommapp/romm) for setup details.
|
||||
|
||||
## Verifying your setup
|
||||
|
||||
After placing BIOS files, verify that everything is correct:
|
||||
`install.py --check` verifies an existing install without downloading anything.
|
||||
For the full report, run `verify.py` from a clone of the repository:
|
||||
|
||||
```bash
|
||||
python scripts/verify.py --platform retroarch
|
||||
@@ -133,7 +199,14 @@ python scripts/verify.py --platform batocera
|
||||
python scripts/verify.py --platform recalbox
|
||||
```
|
||||
|
||||
The output shows each expected file with its status: OK, MISSING, or HASH MISMATCH. Platforms that verify by MD5 (Batocera, Recalbox, EmuDeck) will catch wrong versions. RetroArch only checks that files exist.
|
||||
The output shows each expected file with its status: `ok`, `missing`, or
|
||||
`untested`. `untested` means the file is there but its hash is not the expected
|
||||
one, which is how a wrong revision or a bad dump shows up.
|
||||
|
||||
Only hash-checking platforms can catch a wrong version: Batocera, RetroBat,
|
||||
Recalbox, EmuDeck, RetroDECK, RomM, ROCKNIX and MiSTer FPGA compare MD5,
|
||||
BizHawk compares SHA1. RetroArch, Lakka and RetroPie only check that the file
|
||||
exists, so add `--verbose` there to compare against the emulator ground truth.
|
||||
|
||||
For a single system:
|
||||
|
||||
|
||||
@@ -2,6 +2,11 @@
|
||||
|
||||
Technical documentation for the RetroBIOS toolchain.
|
||||
|
||||
Pages are grouped by what you came to do. **For users** walks through
|
||||
installing and checking files. **Technical reference** describes how the
|
||||
toolchain behaves, one subject per page. **For contributors** is task-oriented:
|
||||
each page takes one job from start to finish.
|
||||
|
||||
## For users
|
||||
|
||||
- **[Getting started](getting-started.md)** - installation, BIOS directory paths per platform, verification
|
||||
@@ -18,6 +23,9 @@ If you just want to download BIOS packs, see the [home page](../index.md).
|
||||
- **[Data model](data-model.md)** - database.json structure, indexes, file resolution order, YAML formats
|
||||
- **[Troubleshooting](troubleshooting.md)** - diagnosis by symptom: missing BIOS, hash mismatch, pack issues, verify errors
|
||||
|
||||
See also [dump provenance](../provenance.md) for how the collection lines up
|
||||
against the No-Intro, Redump and TOSEC catalogs.
|
||||
|
||||
## For contributors
|
||||
|
||||
- **[Profiling guide](profiling.md)** - create an emulator profile from source code, YAML field reference
|
||||
@@ -53,3 +61,12 @@ See [contributing](../contributing.md) for submission guidelines.
|
||||
- **optional** - a file the core functions without, possibly with reduced accuracy or missing features
|
||||
- **hle_fallback** - flag on a file indicating the core has an HLE path; absence is downgraded to INFO severity
|
||||
- **severity** - the urgency of a verification result: OK (verified), INFO (negligible), WARNING (degraded), CRITICAL (broken)
|
||||
- **status** - the outcome of a single file check: `ok`, `untested` (present, hash not the expected one), or `missing`
|
||||
- **discrepancy** - a file that passes the platform check but fails the emulator's own size or hash validation
|
||||
- **shared group** - a file group in `_shared.yml` that several platforms include, carrying the destination a core expects
|
||||
- **data directory** - a whole directory tree a core needs (Dolphin `Sys/`, PPSSPP assets), cached in `data/`, not indexed in the database
|
||||
- **storage tier** - where a file comes from: `embedded` (in `bios/`), `external` (downloaded at build), `user_provided`
|
||||
- **truth** - platform-shaped data generated from emulator profiles, used to diff against what the platform declares
|
||||
- **dump catalog** - a preservation project (Redump, No-Intro, TOSEC) publishing DATs of verified hardware dumps
|
||||
- **provenance** - the catalogs that list a file, joined into the database by hash; an annotation, never an authority
|
||||
- **manifest** - the JSON file list per platform in `install/`, consumed by `install.py`
|
||||
+46
-11
@@ -214,6 +214,21 @@ A few field conventions that protect the toolchain:
|
||||
```bash
|
||||
python scripts/cross_reference.py --emulator dolphin --json
|
||||
python scripts/verify.py --emulator dolphin
|
||||
python scripts/verify.py --emulator dolphin --verbose # per-core checks + source refs
|
||||
python scripts/check_profile_refs.py --emulator dolphin # do the source_ref lines still hold
|
||||
```
|
||||
|
||||
The profile also has to satisfy `schemas/emulator.schema.json`, which CI checks
|
||||
on every PR touching `emulators/`:
|
||||
|
||||
```bash
|
||||
python -c "
|
||||
import json, yaml, sys
|
||||
from jsonschema import validate
|
||||
schema = json.load(open('schemas/emulator.schema.json'))
|
||||
validate(yaml.safe_load(open('emulators/dolphin.yml')), schema)
|
||||
print('ok')
|
||||
"
|
||||
```
|
||||
|
||||
### Lessons learned
|
||||
@@ -252,26 +267,37 @@ even if documentation mentions it.
|
||||
|
||||
### Profile fields
|
||||
|
||||
The authoritative version of this table is `schemas/emulator.schema.json`,
|
||||
which CI validates every profile against.
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `emulator` | yes | display name |
|
||||
| `type` | yes | `libretro`, `standalone`, `standalone + libretro`, `alias`, `launcher`, `game`, `utility`, `test` |
|
||||
| `core_classification` | no | `pure_libretro`, `official_port`, `community_fork`, `frozen_snapshot`, `enhanced_fork`, `game_engine`, `embedded_hle`, `launcher`, `other` |
|
||||
| `source` | yes | libretro core repository URL |
|
||||
| `core_classification` | no | `pure_libretro`, `official_port`, `community_fork`, `frozen_snapshot`, `enhanced_fork`, `game_engine`, `embedded_hle`, `launcher`, `alias`, `other` |
|
||||
| `source` | yes | libretro core repository URL. A dict when the core ships under several repos |
|
||||
| `upstream` | no | original emulator repository URL |
|
||||
| `profiled_date` | yes | date of source analysis |
|
||||
| `profiled_date` | yes | date of source analysis. Quote it, or YAML parses it as a date object |
|
||||
| `core_version` | yes | version analyzed |
|
||||
| `source_commit` | no | upstream commit SHA the source was read at; anchors every `source_ref` line number |
|
||||
| `source_commit` | no | commit SHA of `source` the code was read at; anchors every `source_ref` line number |
|
||||
| `upstream_commit` | no | same, for `upstream` |
|
||||
| `display_name` | no | full display name (e.g. "Sega - Mega Drive (BlastEm)") |
|
||||
| `logo` | no | image URL used on the emulator page |
|
||||
| `systems` | yes | list of system IDs this core handles |
|
||||
| `cores` | no | list of upstream core names for buildbot/target matching |
|
||||
| `cores` | no | every upstream name the core is known by, for buildbot/target matching |
|
||||
| `alias_of` | for `type: alias` | profile key this one aliases; the alias carries no `files` |
|
||||
| `mode` | no | default mode: `standalone`, `libretro`, or `both` |
|
||||
| `verification` | no | how the core verifies BIOS: `existence` or `md5` |
|
||||
| `files` | yes | list of file entries |
|
||||
| `verification` | no | how the core verifies BIOS: `existence`, `md5`, `sha1`, `crc32` |
|
||||
| `files` | yes, unless `type: alias` | list of file entries |
|
||||
| `data_directories` | no | whole directory trees the core needs, referencing `_data_dirs.yml` keys |
|
||||
| `notes` | no | free-form technical notes |
|
||||
| `note` | no | single-paragraph variant of `notes` |
|
||||
| `exclusion_note` | no | why the profile has no files despite .info declaring firmware |
|
||||
| `analysis` | no | structured per-subsystem analysis (capabilities, supported modes) |
|
||||
| `analysis_date`, `analysis_commit` | no | when and at which commit `analysis` was produced |
|
||||
| `platform_details` | no | per-system platform-specific details (paths, romsets, forced systems) |
|
||||
| `mame_version` | no | MAME romset generation a MAME-family profile targets |
|
||||
| `archive_prefix`, `pack_structure` | no | how arcade entries are laid out inside the pack |
|
||||
|
||||
### File entry fields
|
||||
|
||||
@@ -280,7 +306,8 @@ even if documentation mentions it.
|
||||
| `name` | filename as the core expects it |
|
||||
| `required` | true if the core needs this file to function |
|
||||
| `system` | system ID this file belongs to (for multi-system profiles) |
|
||||
| `size` | expected size in bytes |
|
||||
| `archive` | ROM set the file lives inside, for arcade entries (e.g. `neogeo.zip`) |
|
||||
| `size` | expected size in bytes; a list when the code accepts several exact sizes |
|
||||
| `min_size`, `max_size` | size range when the code accepts a range |
|
||||
| `md5`, `sha1`, `crc32`, `sha256` | expected hashes from source code |
|
||||
| `known_hash_adler32` | expected Adler-32 hash (used by Dolphin IPL files) |
|
||||
@@ -289,13 +316,21 @@ even if documentation mentions it.
|
||||
| `mode` | `libretro`, `standalone`, or `both` |
|
||||
| `hle_fallback` | true if a high-level emulation path exists |
|
||||
| `category` | `bios` (default), `game_data`, `bios_zip` |
|
||||
| `region` | geographic region (e.g. `north-america`, `japan`) |
|
||||
| `source_ref` | source file and line number (e.g. `boot.cpp:42`), read at the profile's `source_commit` when set |
|
||||
| `region` | geographic region (e.g. `north-america`, `japan`); a list when one file covers several |
|
||||
| `region_check` | true if the core refuses to boot a game whose region does not match the BIOS |
|
||||
| `fast_boot` | BIOS generation label the core uses to decide whether fast boot is available |
|
||||
| `priority` | tie-breaker when several BIOS files satisfy the same slot, higher wins |
|
||||
| `embedded` | true if the data is compiled into the binary and the external file is optional |
|
||||
| `bundled` | true if the file ships with the emulator rather than being user-provided |
|
||||
| `has_builtin` | true if the core falls back to a built-in copy when the file is absent |
|
||||
| `config_key` | configuration key the emulator exposes to point at the file |
|
||||
| `load_from` | directory the core reads the file from when it is not the system directory |
|
||||
| `source_ref` | source file and line number (e.g. `boot.cpp:42`), read at the profile's `source_commit` when set. A dict splits `standalone` from `libretro` when they diverge |
|
||||
| `path` | destination path relative to system directory |
|
||||
| `description` | what this file is |
|
||||
| `note` | additional context |
|
||||
| `contents` | structure of files inside a BIOS ZIP (`name`, `description`, `size`, `crc32`) |
|
||||
| `storage` | `large_file` for files > 50 MB stored as release assets |
|
||||
| `storage` | `embedded` (default), `external`, `user_provided`, or `large_file`/`release` for files > 50 MB stored as release assets |
|
||||
| `agnostic` | true if any file under the system path within size constraints satisfies the requirement |
|
||||
| `unsourceable` | reason why the file cannot be sourced (acknowledged gap) |
|
||||
| `destination` | target path within the BIOS directory |
|
||||
|
||||
+72
-25
@@ -14,8 +14,8 @@ Budget target: ~175 minutes/month on the GitHub free tier.
|
||||
| Workflow | File | Trigger |
|
||||
|----------|------|---------|
|
||||
| Build & Release | `build.yml` | Push to `bios/**` or `platforms/**`, manual dispatch |
|
||||
| Deploy Site | `deploy-site.yml` | Push to main (platforms, emulators, wiki, scripts, database.json, mkdocs.yml), manual |
|
||||
| PR Validation | `validate.yml` | PR touching `bios/**` or `platforms/**` |
|
||||
| Deploy Site | `deploy-site.yml` | Push to main (platforms, emulators, provenance, wiki, scripts, database.json, mkdocs.yml), manual |
|
||||
| PR Validation | `validate.yml` | PR touching `bios/**`, `platforms/**` or `emulators/**` |
|
||||
| Weekly Sync | `watch.yml` | Cron Monday 06:00 UTC, manual dispatch |
|
||||
|
||||
## build.yml - Build & Release
|
||||
@@ -37,35 +37,61 @@ validated in production.
|
||||
4. Restore large files from the `large-files` release into `.cache/large/`
|
||||
5. Refresh data directories (`refresh_data_dirs.py`)
|
||||
6. Build packs (`generate_pack.py --all --output-dir dist/`)
|
||||
7. Create GitHub release with tag `v{YYYY.MM.DD}` (appends `.N` suffix if
|
||||
7. Split any pack over 2 GB into `.zip.001`, `.zip.002`, ... volumes. GitHub
|
||||
caps release assets at 2 GB, and the `.001` convention is what 7-Zip and
|
||||
PeaZip open directly (they reject `.partNN` names as corrupt)
|
||||
8. Create GitHub release with tag `v{YYYY.MM.DD}` (appends `.N` suffix if
|
||||
a same-day release already exists)
|
||||
8. Clean up old releases, keeping the 3 most recent plus `large-files`
|
||||
9. Clean up old releases, keeping the 3 most recent plus `large-files`
|
||||
|
||||
**Release notes** include file count, total size, per-pack sizes, and the last
|
||||
15 non-merge commits touching `bios/` or `platforms/`.
|
||||
**Release notes** include file count, total size, per-pack sizes, the extract
|
||||
path per platform, and the last 15 non-merge commits touching `bios/` or
|
||||
`platforms/`.
|
||||
|
||||
**Pack variants.** The workflow builds full packs only. Releases that also
|
||||
carry `*_Platform_BIOS_Pack.zip` had a second `--source platform` run added by
|
||||
hand, as in the manual process below. See
|
||||
[advanced usage](advanced-usage.md#pack-source-variants).
|
||||
|
||||
## deploy-site.yml - Deploy Documentation Site
|
||||
|
||||
**Trigger.** Push to `main` when any of these paths change: `platforms/`,
|
||||
`emulators/`, `wiki/`, `scripts/generate_site.py`, `scripts/generate_readme.py`,
|
||||
`scripts/verify.py`, `scripts/common.py`, `database.json`, `mkdocs.yml`.
|
||||
Also manual dispatch.
|
||||
`emulators/`, `provenance/`, `wiki/`, `scripts/generate_site.py`,
|
||||
`scripts/generate_readme.py`, `scripts/verify.py`, `scripts/common.py`,
|
||||
`database.json`, `mkdocs.yml`. Also manual dispatch.
|
||||
|
||||
The list is the set of inputs the site is generated from. Adding a new input to
|
||||
`generate_site.py` means adding its path here, or the site silently goes stale.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Checkout, Python 3.12
|
||||
2. Install `pyyaml`, `mkdocs-material`, `pymdown-extensions`
|
||||
3. Run `generate_site.py` (converts YAML data into MkDocs pages)
|
||||
4. Run `generate_readme.py` (rebuilds README.md and CONTRIBUTING.md)
|
||||
5. `mkdocs build` to produce the static site
|
||||
6. Upload artifact, deploy to GitHub Pages
|
||||
2. Install `pyyaml`, `mkdocs-material>=9.7.5,<10`, `pymdown-extensions>=10.14`
|
||||
3. Restore large files from the `large-files` release, refresh data directories
|
||||
4. Run `generate_site.py` (converts YAML data into MkDocs pages and rewrites
|
||||
`mkdocs.yml`)
|
||||
5. Run `generate_readme.py` (rebuilds README.md and CONTRIBUTING.md)
|
||||
6. `mkdocs build --strict` to produce the static site
|
||||
7. Upload artifact, deploy to GitHub Pages
|
||||
|
||||
The site is deployed via the `github-pages` environment using the official
|
||||
`actions/deploy-pages` action.
|
||||
`actions/deploy-pages` action. Pages deployments are queued rather than
|
||||
cancelled (`cancel-in-progress: false`): cancelling one mid-flight leaves the
|
||||
deployment stuck and the next runs time out waiting on it.
|
||||
|
||||
`--strict` turns MkDocs warnings into failures, so a broken internal link or a
|
||||
dangling anchor fails the build instead of shipping. The `validation:` block in
|
||||
`mkdocs.yml` is what promotes unrecognized links and missing anchors to
|
||||
warnings in the first place.
|
||||
|
||||
The theme version is pinned on both sides: `>=9.7.5` because that is the
|
||||
release which caps `mkdocs < 2` (MkDocs 2.0 ships without a license), `<10`
|
||||
so a major theme release cannot change the site without a deliberate bump.
|
||||
|
||||
## validate.yml - PR Validation
|
||||
|
||||
**Trigger.** Pull requests that modify `bios/**` or `platforms/**`.
|
||||
**Trigger.** Pull requests that modify `bios/**`, `platforms/**` or
|
||||
`emulators/**`.
|
||||
|
||||
**Concurrency.** Per-PR group, cancel in-progress.
|
||||
|
||||
@@ -75,9 +101,11 @@ Four parallel jobs:
|
||||
`validate_pr.py --markdown` on each, and posts the validation report as a PR
|
||||
comment (hash verification, database match status).
|
||||
|
||||
**validate-configs.** Validates all platform YAML files against
|
||||
`schemas/platform.schema.json` using `jsonschema`. Fails if any config does
|
||||
not match the schema.
|
||||
**validate-configs.** Validates every platform YAML against
|
||||
`schemas/platform.schema.json` and every emulator profile against
|
||||
`schemas/emulator.schema.json`, using `jsonschema`. Fails if any file does not
|
||||
match. `*.old.yml` files are skipped: they are hash-scraper backups, not
|
||||
profiles.
|
||||
|
||||
**run-tests.** Runs `python -m unittest tests.test_e2e -v`. Must pass before
|
||||
merge.
|
||||
@@ -113,8 +141,11 @@ merges.
|
||||
Files larger than 50 MB are stored as assets on a permanent GitHub release
|
||||
named `large-files` (to keep the git repository lightweight).
|
||||
|
||||
Known large files: PS3UPDAT.PUP, PSVUPDAT.PUP, PSP2UPDAT.PUP, dsi_nand.bin,
|
||||
maclc3.zip, Firmware.19.0.0.zip (Switch).
|
||||
Examples: PS3UPDAT.PUP, PSVUPDAT.PUP, PSP2UPDAT.PUP, the DSi NAND images,
|
||||
maclc3.zip, Firmware.19.0.0.zip (Switch), the QEMU EDK2 firmware, the ScummVM
|
||||
data bundle, the EasyRPG soundfont, the Dolphin/Ishiiruka SD card images, and
|
||||
the arcade sets over 100 MB. `.gitignore` is the authoritative list: every
|
||||
`bios/` path listed there is a release asset.
|
||||
|
||||
**Storage.** Listed in `.gitignore` so they stay out of git history. The
|
||||
`large-files` release is excluded from cleanup (the build workflow only
|
||||
@@ -138,21 +169,37 @@ from the release and caches in `.cache/large/` for subsequent runs.
|
||||
When `build.yml` is disabled, build and release manually:
|
||||
|
||||
```bash
|
||||
# Run the full pipeline (DB + verify + packs + consistency check)
|
||||
python scripts/pipeline.py --offline
|
||||
# Run the full pipeline (DB + verify + packs + manifests + integrity + docs)
|
||||
python scripts/pipeline.py
|
||||
|
||||
# Or step by step:
|
||||
python scripts/generate_db.py --force --bios-dir bios --output database.json
|
||||
python scripts/verify.py --all
|
||||
python scripts/generate_pack.py --all --output-dir dist/
|
||||
python scripts/generate_pack.py --all --output-dir dist/ # full packs
|
||||
python scripts/generate_pack.py --all --source platform --output-dir dist/ # platform packs
|
||||
python scripts/generate_pack.py --all --verify-packs --output-dir dist/
|
||||
|
||||
# Split anything over 2 GB (GitHub asset cap)
|
||||
for f in dist/*.zip; do
|
||||
[ "$(stat -c%s "$f")" -gt 2000000000 ] || continue
|
||||
split --bytes=1900M --numeric-suffixes=1 --suffix-length=3 "$f" "$f." && rm "$f"
|
||||
done
|
||||
|
||||
# Create the release
|
||||
DATE=$(date +%Y.%m.%d)
|
||||
gh release create "v${DATE}" dist/*.zip \
|
||||
gh release create "v${DATE}" dist/*.zip* \
|
||||
--title "BIOS Pack v${DATE}" \
|
||||
--notes "Release notes here" \
|
||||
--latest
|
||||
```
|
||||
|
||||
The two `generate_pack.py` runs are what puts both `*_BIOS_Pack.zip` and
|
||||
`*_Platform_BIOS_Pack.zip` on the release. `--all-variants` builds all six
|
||||
combinations instead, which is more than a release needs.
|
||||
|
||||
Run the pipeline online for a release: `--offline` skips the data directory
|
||||
refresh and the MAME/FBNeo hash refresh, so the packs would ship stale data
|
||||
directories.
|
||||
|
||||
To re-enable automated releases, remove the `if: false` guard from the
|
||||
`release` job in `build.yml`.
|
||||
+82
-18
@@ -2,30 +2,51 @@
|
||||
|
||||
This page covers how to run, understand, and extend the test suite.
|
||||
|
||||
All tests use synthetic fixtures. No real BIOS files, platform configs, or
|
||||
network access required.
|
||||
10 modules, 400 tests. No network access anywhere. Most modules build synthetic
|
||||
fixtures in a temp directory; the three that read the working tree skip cleanly
|
||||
when the data they need is absent.
|
||||
|
||||
## Running tests
|
||||
|
||||
Run a single test module:
|
||||
|
||||
```bash
|
||||
python -m unittest tests.test_e2e -v
|
||||
python -m unittest tests.test_pack_integrity -v
|
||||
python -m unittest tests.test_mame_parser -v
|
||||
python -m unittest tests.test_fbneo_parser -v
|
||||
python -m unittest tests.test_hash_merge -v
|
||||
```
|
||||
|
||||
Run the full suite:
|
||||
|
||||
```bash
|
||||
python -m unittest discover tests -v
|
||||
```
|
||||
|
||||
Run a single module:
|
||||
|
||||
```bash
|
||||
python -m unittest tests.test_e2e -v
|
||||
python -m unittest tests.test_install -v
|
||||
python -m unittest tests.test_provenance -v
|
||||
python -m unittest tests.test_mame_parser -v
|
||||
python -m unittest tests.test_hash_merge -v
|
||||
python -m unittest tests.test_fbneo_parser -v
|
||||
python -m unittest tests.test_profile_refs -v
|
||||
python -m unittest tests.test_pack_integrity -v
|
||||
python -m unittest tests.test_torrentzip -v
|
||||
python -m unittest tests.test_no_case_collisions -v
|
||||
```
|
||||
|
||||
The only dependency is `pyyaml`. No test framework beyond the standard
|
||||
library `unittest` module.
|
||||
|
||||
## Modules at a glance
|
||||
|
||||
| Module | Tests | Fixtures | What it covers |
|
||||
|--------|-------|----------|----------------|
|
||||
| `test_e2e.py` | 217 | synthetic | resolution, verification, packs, cross-reference, targets, truth |
|
||||
| `test_install.py` | 70 | synthetic | `install.py` detection, config parsing, manifest handling |
|
||||
| `test_provenance.py` | 29 | synthetic | Logiqx/Redump parsing, DAT import, provenance join, coverage report |
|
||||
| `test_mame_parser.py` | 22 | inline C | BIOS root sets, ROM blocks, macro expansion |
|
||||
| `test_hash_merge.py` | 17 | synthetic | YAML hash merge, diff, formatting preservation |
|
||||
| `test_fbneo_parser.py` | 16 | inline C | `BDF_BOARDROM` sets, ROM info parsing |
|
||||
| `test_profile_refs.py` | 12 | synthetic | `check_profile_refs` pure functions, no network |
|
||||
| `test_pack_integrity.py` | 8 | real packs | extract each ZIP, verify paths and hashes |
|
||||
| `test_torrentzip.py` | 8 | real romsets | TorrentZip builder byte-for-byte |
|
||||
| `test_no_case_collisions.py` | 1 | real `bios/` | no case-colliding paths on Windows/macOS clones |
|
||||
|
||||
## Test architecture
|
||||
|
||||
### test_e2e.py
|
||||
@@ -49,8 +70,16 @@ logic.
|
||||
|-------|----------|
|
||||
| `test_01`--`test_14` | File resolution (SHA1, MD5, name, alias, truncated MD5, composite, zip contents, variants, hash mismatch) |
|
||||
| `test_20`--`test_31` | Verification (existence mode, MD5 mode, required/optional severity, zipped file, multi-hash) |
|
||||
| `test_40`--`test_47` | Cross-reference (undeclared files, standalone skip, alias profiles, data dir suppression, exclusion notes) |
|
||||
| `test_50`+ | Platform config (inheritance, shared groups, data directories, grouping, core resolution, target filtering, ground truth) |
|
||||
| `test_40`--`test_51` | Cross-reference and platform grouping (undeclared files, standalone skip, alias profiles, data dir suppression, exclusion notes) |
|
||||
| `test_60`--`test_61` | Storage tiers (external, user-provided) |
|
||||
| `test_70`--`test_84` | Emulator-level validation (index build, size, CRC32, MD5, SHA1, crypto) |
|
||||
| `test_90`--`test_125` | Per-emulator and per-system verification, `dest_hint` resolution, registry metadata |
|
||||
| `test_130`--`test_183` | Pack generation (required-only, split, `--from-md5`, path conflicts, archive extras, truth generation, exporters) |
|
||||
| `test_200`--`test_227` | Pack source variants, deterministic ZIPs, SHA256/CRC32 resolution, MiSTer scraper, manifests |
|
||||
|
||||
Numbers are stable anchors, not an execution order. When a range fills up, a
|
||||
letter suffix keeps a new test next to the behavior it covers
|
||||
(`test_130b_existence_pack_reports_hash_mismatch`).
|
||||
|
||||
Each test calls the same functions that `verify.py` and `generate_pack.py` use
|
||||
in production, against the synthetic fixtures.
|
||||
@@ -82,6 +111,38 @@ upstream BIOS hashes into emulator profiles. Covers:
|
||||
Fixtures are programmatically generated YAML/JSON files written to a temp
|
||||
directory.
|
||||
|
||||
### Provenance and installer tests
|
||||
|
||||
**test_provenance.** Covers the dump-catalog pipeline end to end on synthetic
|
||||
DATs: the Logiqx XML parser, the Redump scraper's parsing path, the DAT pack
|
||||
importer, the SHA1-then-MD5+size join performed by `generate_db.py`, and the
|
||||
coverage report's covered/uncovered DAT accounting.
|
||||
|
||||
**test_install.** Covers `install.py` without touching the network: OS
|
||||
detection, each registry detection method (`config_file`, `path_exists`,
|
||||
`file_exists`), config-file key parsing, manifest loading, target filtering,
|
||||
and destination resolution.
|
||||
|
||||
**test_profile_refs.** Covers the pure functions of `check_profile_refs`
|
||||
(anchor matching, line-window search, hash extraction). The GitHub fetching
|
||||
path is not exercised, so the module runs offline.
|
||||
|
||||
### Tests that read the working tree
|
||||
|
||||
Three modules assert on real repository data instead of fixtures. Each skips
|
||||
when its input is missing, so a partial checkout still runs green.
|
||||
|
||||
**test_torrentzip.** Rebuilds real MAME romsets through the TorrentZip builder
|
||||
and asserts the output is byte-identical, which is what keeps arcade ZIP hashes
|
||||
stable across rebuilds.
|
||||
|
||||
**test_no_case_collisions.** Walks `bios/` and fails on two paths that differ
|
||||
only by case. On Windows and macOS, git can only check out one of them, which
|
||||
silently corrupts the clone. `.variants/` is exempt: those names are
|
||||
disambiguated by a hash suffix on purpose.
|
||||
|
||||
**test_pack_integrity.** Described below; needs packs in `dist/`.
|
||||
|
||||
## How to add a test
|
||||
|
||||
1. **Pick the right category.** Find the number range that matches the
|
||||
@@ -162,13 +223,16 @@ Ideally, tests, code, and documentation ship together. When profiles and platfor
|
||||
|
||||
The `validate.yml` workflow runs `test_e2e` on every pull request that touches
|
||||
`bios/` or `platforms/` files. The test job (`run-tests`) runs in parallel
|
||||
with BIOS validation, schema validation, and auto-labeling.
|
||||
with BIOS validation, schema validation, and auto-labeling. `build.yml` runs
|
||||
the same module before building release packs.
|
||||
|
||||
CI runs `test_e2e` only, not the whole suite: the other modules either need
|
||||
build artifacts (`test_pack_integrity` needs packs in `dist/`) or duplicate
|
||||
work the workflow already does. Run `python -m unittest discover tests` locally
|
||||
before pushing.
|
||||
|
||||
Tests must pass before merge. If a test fails in CI, reproduce locally with:
|
||||
|
||||
```bash
|
||||
python -m unittest tests.test_e2e -v 2>&1 | head -50
|
||||
```
|
||||
|
||||
The `build.yml` workflow also runs the test suite before building release
|
||||
packs.
|
||||
+69
-6
@@ -13,7 +13,8 @@ python scripts/pipeline.py --offline --skip-docs # skip readme + site generati
|
||||
python scripts/pipeline.py --offline --target switch # filter by hardware target
|
||||
python scripts/pipeline.py --offline --with-truth # include truth generation + diff
|
||||
python scripts/pipeline.py --offline --with-export # include native format export
|
||||
python scripts/pipeline.py --check-buildbot # check buildbot data freshness
|
||||
python scripts/pipeline.py --offline --include-archived # include archived platforms
|
||||
python scripts/pipeline.py --check-buildbot # online run + buildbot freshness check
|
||||
```
|
||||
|
||||
Pipeline steps:
|
||||
@@ -21,10 +22,11 @@ Pipeline steps:
|
||||
| Step | Description | Skipped by |
|
||||
|------|-------------|------------|
|
||||
| 1/8 | Generate database | - |
|
||||
| 1b | Dump-catalog coverage report | - (offline, reads `provenance/`) |
|
||||
| 2/8 | Refresh data directories | `--offline` |
|
||||
| 2a | Refresh MAME BIOS hashes | `--offline` |
|
||||
| 2a2 | Refresh FBNeo BIOS hashes | `--offline` |
|
||||
| 2b | Check buildbot staleness | only with `--check-buildbot` |
|
||||
| 2b | Check buildbot staleness | only with `--check-buildbot`, skipped by `--offline` |
|
||||
| 2c | Generate truth YAMLs | only with `--with-truth` / `--with-export` |
|
||||
| 2d | Diff truth vs scraped | only with `--with-truth` / `--with-export` |
|
||||
| 2e | Export native formats | only with `--with-export` |
|
||||
@@ -37,6 +39,25 @@ Pipeline steps:
|
||||
| 7/8 | Generate README | `--skip-docs` |
|
||||
| 8/8 | Generate site | `--skip-docs` |
|
||||
|
||||
Pipeline flags:
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
| `--offline` | skip every step that needs the network (2, 2a, 2a2, 2b) |
|
||||
| `--skip-packs` | skip steps 4, 4b, 4c, 6 and the consistency check |
|
||||
| `--skip-docs` | skip steps 7 and 8 |
|
||||
| `--include-archived` | include archived platforms in verify, packs, truth and export |
|
||||
| `--target TARGET` | filter verify, packs and truth by hardware target |
|
||||
| `--source {platform,truth,full}` | pack file source, passed through to `generate_pack.py` |
|
||||
| `--all-variants` | build the 6 source x required combinations |
|
||||
| `--check-buildbot` | run step 2b; additive, and ignored under `--offline` |
|
||||
| `--with-truth` | run steps 2c and 2d |
|
||||
| `--with-export` | run steps 2c, 2d and 2e |
|
||||
| `--output-dir DIR` | pack output directory (default `dist/`) |
|
||||
|
||||
`--check-buildbot` runs the whole pipeline online. For the freshness check
|
||||
alone, run `python scripts/check_buildbot_system.py`.
|
||||
|
||||
## Individual tools
|
||||
|
||||
### generate_db.py
|
||||
@@ -71,12 +92,16 @@ Verification modes per platform:
|
||||
| Platform | Mode | Logic |
|
||||
|----------|------|-------|
|
||||
| RetroArch, Lakka, RetroPie | existence | file present = OK |
|
||||
| Batocera, RetroBat | md5 | MD5 hash match |
|
||||
| Recalbox | md5 | MD5 multi-hash, 3 severity levels |
|
||||
| Batocera, RetroBat | md5 | MD5 hash match, plus `checkInsideZip` for `zippedFile` entries |
|
||||
| Recalbox | md5 | MD5 multi-hash, `md5_composite` for ZIPs, 3 severity levels |
|
||||
| EmuDeck | md5 | MD5 whitelist per system |
|
||||
| RetroDECK | md5 | MD5 per file via component manifests |
|
||||
| RomM | md5 | size + any hash (MD5/SHA1/CRC32) |
|
||||
| BizHawk | sha1 | SHA1 per firmware from FirmwareDatabase.cs |
|
||||
| RomM | md5 | size + any hash (MD5/SHA1/CRC32), no ZIP inspection |
|
||||
| ROCKNIX | md5 | MD5 hash match, same shape as Batocera |
|
||||
| MiSTer FPGA | md5 | MD5 of the file at its destination, no ZIP inspection |
|
||||
| BizHawk | sha1 | SHA1 per firmware from `FirmwareDatabase.cs` |
|
||||
|
||||
Full details and severity mapping: [verification modes](verification-modes.md).
|
||||
|
||||
### generate_pack.py
|
||||
|
||||
@@ -111,8 +136,16 @@ python scripts/generate_pack.py --platform retroarch --source full # both (
|
||||
python scripts/generate_pack.py --all --all-variants --output-dir dist/ # all 6 combinations
|
||||
python scripts/generate_pack.py --all --all-variants --verify-packs --output-dir dist/
|
||||
|
||||
# Listing and integrity
|
||||
python scripts/generate_pack.py --list # list available platforms
|
||||
python scripts/generate_pack.py --list-emulators
|
||||
python scripts/generate_pack.py --list-systems
|
||||
python scripts/generate_pack.py --platform retroarch --list-targets
|
||||
python scripts/generate_pack.py --all --verify-packs --output-dir dist/ # extract + hash
|
||||
|
||||
# Data refresh
|
||||
python scripts/generate_pack.py --all --refresh-data # force re-download data dirs
|
||||
python scripts/generate_pack.py --all --offline # skip the refresh entirely
|
||||
|
||||
# Install manifests (consumed by install.py)
|
||||
python scripts/generate_pack.py --all --manifest --output-dir install/
|
||||
@@ -224,6 +257,8 @@ python scripts/refresh_data_dirs.py --registry path/to/_data_dirs.yml
|
||||
| `auto_fetch.py` | Fetch missing BIOS files from known sources (4-step pipeline) |
|
||||
| `list_platforms.py` | List active platforms (`--all` includes archived, used by CI) |
|
||||
| `download.py` | Download packs from GitHub releases (Python, multi-threaded) |
|
||||
| `download.sh` | Same, as a shell one-liner (`curl` + `unzip`) |
|
||||
| `provenance_report.py` | Dump-catalog coverage and acquisition targets (see above) |
|
||||
| `generate_readme.py` | Generate README.md and CONTRIBUTING.md from database |
|
||||
| `generate_site.py` | Generate all MkDocs site pages (this documentation) |
|
||||
| `deterministic_zip.py` | Rebuild MAME BIOS ZIPs deterministically (same ROMs = same hash) |
|
||||
@@ -284,6 +319,7 @@ Located in `scripts/scraper/`. Each inherits `BaseScraper` and implements `fetch
|
||||
|
||||
Internal modules: `base_scraper.py` (abstract base with `_fetch_raw()` caching
|
||||
and shared CLI), `dat_parser.py` (clrmamepro DAT format parser),
|
||||
`logiqx_parser.py` (Logiqx XML DAT parser, used by the dump catalogs),
|
||||
`mame_parser.py` (MAME C source BIOS root set parser),
|
||||
`fbneo_parser.py` (FBNeo C source BIOS set parser),
|
||||
`_hash_merge.py` (text-based YAML patching that preserves formatting).
|
||||
@@ -291,6 +327,33 @@ and shared CLI), `dat_parser.py` (clrmamepro DAT format parser),
|
||||
Adding a scraper: inherit `BaseScraper`, implement `fetch_requirements()`,
|
||||
call `scraper_cli(YourScraper)` in `__main__`.
|
||||
|
||||
## Dump-catalog provenance
|
||||
|
||||
Snapshots of the preservation catalogs are committed to `provenance/` and
|
||||
refreshed manually. `generate_db.py` joins them into `database.json`;
|
||||
`provenance_report.py` reports the gaps.
|
||||
|
||||
```bash
|
||||
python -m scripts.scraper.redump_dat_scraper --dry-run
|
||||
python -m scripts.scraper.redump_dat_scraper --output provenance/redump.json
|
||||
python -m scripts.scraper.dat_pack_importer --source no-intro --pack no-intro.zip
|
||||
python -m scripts.scraper.dat_pack_importer --source tosec --pack TOSEC-v2025.zip
|
||||
python scripts/provenance_report.py # coverage summary (pipeline step 1b)
|
||||
python scripts/provenance_report.py --missing # list acquisition targets
|
||||
python scripts/provenance_report.py --json # full report
|
||||
```
|
||||
|
||||
| Catalog | Acquisition | Notes |
|
||||
|---------|------------|-------|
|
||||
| Redump | direct fetch from redump.info `/static/bios` | 4 BIOS DATs. redump.org serves stale DATs |
|
||||
| No-Intro | local pack, imported with `dat_pack_importer` | Dat-o-Matic blocks automation; the `hugo19941994/auto-datfile-generator` mirror rebuilds daily |
|
||||
| TOSEC | local pack, imported with `dat_pack_importer` | annual pack from tosecdev.org, 138 Firmware DATs |
|
||||
|
||||
The join is by SHA1, then by MD5 + size. A match sets the `provenance` field
|
||||
on the database entry and renders a verified dump badge on the system pages.
|
||||
It never overrides the emulator source code: see
|
||||
[architecture](architecture.md#dump-catalog-provenance).
|
||||
|
||||
## Target scrapers
|
||||
|
||||
Located in `scripts/scraper/targets/`. Each inherits `BaseTargetScraper` and implements `fetch_targets()`.
|
||||
|
||||
+21
-9
@@ -43,10 +43,17 @@ The file exists on disk, but the emulator reports it as missing.
|
||||
|
||||
Each platform expects BIOS files in a specific base directory:
|
||||
|
||||
- RetroArch, Lakka: `system/` inside the RetroArch directory
|
||||
- RetroArch: `system/` inside the RetroArch directory
|
||||
- Lakka: `/storage/system/`
|
||||
- Batocera: `/userdata/bios/`
|
||||
- Recalbox: `/recalbox/share/bios/`
|
||||
- RetroPie: `~/RetroPie/BIOS/`
|
||||
- ROCKNIX: `/storage/roms/bios/`
|
||||
- MiSTer FPGA: `/media/fat/games/`
|
||||
- BizHawk: `Firmware/` inside the BizHawk directory
|
||||
- EmuDeck: `~/Emulation/bios/`
|
||||
- RetroDECK: `~/retrodeck/bios/`
|
||||
- RetroBat: `bios/` inside the RetroBat directory
|
||||
|
||||
Some cores expect files in subdirectories (e.g. `dc/` for Dreamcast, `pcsx2/bios/`
|
||||
for PlayStation 2). Check the `path:` field in the emulator profile for the exact
|
||||
@@ -81,12 +88,14 @@ or emulator expects. The reason field shows the expected vs actual hash prefix.
|
||||
To find the correct version, check the system page on the site. It lists every
|
||||
known BIOS file with its expected MD5 and SHA1.
|
||||
|
||||
**UNTESTED:**
|
||||
**Existence-mode platforms never report UNTESTED:**
|
||||
|
||||
On existence-only platforms (RetroArch, Lakka, RetroPie), the file is present
|
||||
but its hash was not verified against a known value. The platform itself only
|
||||
checks that the file exists. The `--verbose` flag shows ground truth data from
|
||||
emulator profiles, which can confirm whether the file's hash is actually correct.
|
||||
On RetroArch, Lakka and RetroPie, verification only asks whether the file is
|
||||
there, so the only outcomes are OK and MISSING. A wrong-region or corrupt file
|
||||
still shows as OK. To find out whether the content is right on those platforms,
|
||||
run `verify.py --verbose`: it adds the emulator ground truth from the profiles
|
||||
and flags a `DISCREPANCY` when the file passes the platform check but fails the
|
||||
emulator's own size or hash validation.
|
||||
|
||||
**The .variants/ directory:**
|
||||
|
||||
@@ -109,11 +118,14 @@ important:
|
||||
|
||||
| Severity | Meaning | Action needed |
|
||||
|----------|---------|---------------|
|
||||
| CRITICAL | Required file missing or hash mismatch on MD5 platforms | Must fix. Core won't function. |
|
||||
| WARNING | Optional file missing, or hash mismatch on existence platforms | Core works but with reduced functionality. |
|
||||
| INFO | Optional file missing on existence-only platforms, or HLE fallback available | Core works fine, BIOS improves accuracy. |
|
||||
| CRITICAL | Required file missing on an MD5 or SHA1 platform | Must fix. Core won't function. |
|
||||
| WARNING | Optional file missing on an MD5 or SHA1 platform, hash mismatch (UNTESTED) on any hash platform, or a required file missing on an existence platform | Core works but with reduced functionality, or runs on content nobody verified. |
|
||||
| INFO | Optional file missing on an existence-only platform, or a missing file the core covers with an HLE fallback | Core works fine, BIOS improves accuracy. |
|
||||
| OK | File present and verified | No action needed. |
|
||||
|
||||
The full mapping, including how `hle_fallback` overrides the platform mode, is
|
||||
in [verification modes](verification-modes.md#severity-matrix).
|
||||
|
||||
Focus on CRITICAL issues first. WARNING files improve the experience but aren't
|
||||
strictly necessary. INFO files are nice to have.
|
||||
|
||||
|
||||
@@ -28,7 +28,7 @@ Lakka and RetroPie inherit this behavior through platform config inheritance
|
||||
|
||||
## MD5 Mode
|
||||
|
||||
**Platforms**: Batocera, RetroBat, Recalbox, EmuDeck, RetroDECK, RomM
|
||||
**Platforms**: Batocera, RetroBat, Recalbox, EmuDeck, RetroDECK, RomM, ROCKNIX, MiSTer FPGA
|
||||
|
||||
All MD5-mode platforms compute a hash of the file and compare it against an expected value.
|
||||
The details vary by platform.
|
||||
@@ -98,6 +98,29 @@ SHA1, MD5, and CRC32 for reference, but `verify.py` checks only the MD5 field,
|
||||
matching the platform's runtime behavior. ZIP files are not opened; only the
|
||||
container is checked.
|
||||
|
||||
### ROCKNIX verification
|
||||
|
||||
**Source**: `rocknix-systems`, function `checkBios`
|
||||
|
||||
ROCKNIX derives its BIOS checker from Batocera's, so the logic has the same
|
||||
shape: `md5sum()` on the file, `checkInsideZip()` for archive entries, and an
|
||||
`altmd5` field that accepts a second hash for the same entry. Every entry the
|
||||
platform declares is mandatory in its own data, so the platform YAML marks all
|
||||
38 files `required: true` and none of them use `zippedFile`.
|
||||
|
||||
### MiSTer FPGA verification
|
||||
|
||||
**Source**: `Downloader_MiSTer`, `jobs/process_db_index_worker.py`
|
||||
|
||||
The MiSTer downloader hashes the file at its destination path and compares it
|
||||
against the MD5 recorded in the BIOS database. There is no ZIP inspection and
|
||||
no required/optional distinction: an entry either matches or it does not.
|
||||
|
||||
Scope note: only the BiosDB entries are in scope. They carry an explicit `url`
|
||||
because MiSTer cannot host them, which is exactly what the user has to supply.
|
||||
The 208 `games/` files shipped by the `Distribution_MiSTer` repository carry no
|
||||
URL and install themselves, so they are not part of the pack.
|
||||
|
||||
|
||||
## SHA1 Mode
|
||||
|
||||
@@ -201,9 +224,11 @@ The function tries these steps in order, returning the first match:
|
||||
| Step | Method | Returns | When it applies |
|
||||
|------|--------|---------|-----------------|
|
||||
| 0 | Path suffix exact | `exact` | `dest_hint` matches `by_path_suffix` index (regional variants with same filename, e.g., `GC/USA/IPL.bin` vs `GC/EUR/IPL.bin`) |
|
||||
| 1 | SHA1 exact | `exact` | SHA1 present in the file entry and found in database |
|
||||
| 2 | MD5 direct lookup | `md5_exact` | MD5 present, not a `zipped_file` entry, name matches (prevents cross-contamination from unrelated files sharing an MD5) |
|
||||
| 3 | Name/alias existence | `exact` | No MD5 in entry; any file with matching name or alias exists. Prefers primary over `.variants/` |
|
||||
| 1 | SHA1 exact | `exact` | SHA1 present in the file entry and found in database. A list-valued `sha1` from a profile is accepted |
|
||||
| 1b | SHA256 exact | `exact` | SHA256 present, for profiles read from sources that publish SHA256 (e.g. MesenCE) |
|
||||
| 1c | CRC32 + size exact | `exact` | Only when no stronger hash is declared (CRC-only profiles such as FBNeo and Clock Signal). CRC32 alone is weak, so the declared size has to confirm the match |
|
||||
| 2 | MD5 direct lookup | `md5_exact` | MD5 present, not a `zipped_file` entry. A full 32-char MD5 is trusted on its own; a truncated MD5 also has to match the name, which prevents cross-contamination from unrelated files sharing a prefix |
|
||||
| 3 | Name/alias existence | `exact` | No MD5 in entry; any file with matching name or alias exists. Prefers primary over `.variants/`, with a case-insensitive fallback for upstreams that disagree on casing |
|
||||
| 4 | Name + md5_composite/MD5 | `exact` or `hash_mismatch` | Name matches, checks md5_composite for ZIPs and direct MD5 per candidate. Falls back to hash_mismatch if name matches but no hash does |
|
||||
| 5 | ZIP contents index | `zip_exact` | `zipped_file` with MD5; searches inner ROM MD5 across all ZIPs when name-based resolution failed |
|
||||
| 6 | MAME clone fallback | `mame_clone` | File was deduped; resolves via canonical set name (up to 3 levels deep) |
|
||||
|
||||
Reference in new issue
Block a user