feat: add archive_prefix for core-specific BIOS subdirectories

Closes #43

FBNeo and Kronos expect BIOS archives in core-specific subdirectories
(system/fbneo/, system/kronos/). RetroArch firmware check uses .info
paths which include these prefixes, so files at root show as Missing.

Add archive_prefix field to emulator profiles. The pack code now places
archive copies in the prefixed subdirectory while keeping root copies
for cores that expect them there (e.g. Geolith for neogeo.zip).
This commit is contained in:
Abdessamad Derraz
2026-03-31 09:17:54 +02:00
parent 40ff2b5307
commit b56f8dd05f
17 changed files with 10826 additions and 1421 deletions

View File

@@ -1,109 +1,14 @@
# Contributing to RetroBIOS # Contributing to RetroBIOS
## Types of contributions ## Add a BIOS file
- **Add a BIOS file** - a great way to get started. Fork, add the file, open a PR. 1. Fork this repository
- **Create an emulator profile** - document what a core actually loads from source code. See the [profiling guide](https://abdess.github.io/retrobios/wiki/profiling/). 2. Place the file in `bios/Manufacturer/Console/filename`
- **Add a platform** - integrate a new frontend (scraper + YAML config). See [adding a platform](https://abdess.github.io/retrobios/wiki/adding-a-platform/). 3. Variants (alternate hashes): `bios/Manufacturer/Console/.variants/`
- **Add or fix a scraper** - parse upstream sources for BIOS requirements. See [adding a scraper](https://abdess.github.io/retrobios/wiki/adding-a-scraper/). 4. Create a Pull Request - checksums are verified automatically
- **Fix a bug or improve tooling** - Python scripts in `scripts/`, single dependency (`pyyaml`).
## Local setup ## File conventions
```bash - Files >50 MB go in GitHub release assets (`large-files` release)
git clone https://github.com/Abdess/retrobios.git - RPG Maker and ScummVM directories are excluded from deduplication
cd retrobios - See the [documentation site](https://abdess.github.io/retrobios/) for full details
pip install pyyaml
# run tests
python -m unittest tests.test_e2e -v
# run full pipeline (DB + verify + packs + consistency check)
python scripts/pipeline.py --offline
```
Requires Python 3.10 or later.
## Adding a BIOS file
1. Place the file in `bios/Manufacturer/Console/filename`.
2. Alternate versions (different hash, same purpose) go in `bios/Manufacturer/Console/.variants/`.
3. Files over 50 MB go as assets on the `large-files` GitHub release (git handles them better that way).
4. RPG Maker and ScummVM directories are excluded from deduplication - please keep their structure as-is.
5. Open a pull request. CI validates checksums automatically and posts a report.
## Commit conventions
Format: `type: description` (50 characters max, lowercase start).
Allowed types: `feat`, `refactor`, `chore`, `docs`, `fix`.
```
feat: add panasonic 3do bios files
docs: update architecture diagram
fix: resolve truncated md5 matching
chore: remove unused test fixtures
refactor: extract hash logic to common.py
```
Keep messages factual. No marketing language, no superfluous adjectives.
## Code and documentation quality
The codebase runs on Python 3.10+ with a single dependency (`pyyaml`). All modules
include `from __future__ import annotations` at the top. Type hints on every function
signature, `pathlib` instead of `os.path`, and dataclasses where a plain class would
just hold attributes.
On performance: O(1) or O(n) algorithms are preferred. If something needs O(n^2), a
comment explaining why helps future readers. List comprehensions over explicit loops,
generators when iterating large datasets, and standard default arguments
(`def f(items=None)` over `def f(items=[])`).
File I/O uses context managers. ZIP extraction goes through `safe_extract_zip()` in
`common.py`, which prevents zip-slip path traversal.
The code stays lean. Comments that describe *why* age better than comments that
restate *what*. Unused variables can be deleted rather than renamed with an underscore.
The same spirit applies to documentation and emulator profiles. Straightforward
language, honest labels ("untested" when something is untested).
When a bug is spotted while working on something else, fixing it in the same PR
keeps things tidy. Features ideally ship complete in one pass: code, tests, a
passing pipeline run, and any documentation updates together.
## Pull request process
CI runs four checks on every PR:
| Check | What it does |
|-------|-------------|
| `validate-bios` | hashes changed BIOS files against the database, posts a validation report |
| `validate-configs` | schema-validates platform YAML configs |
| `run-tests` | runs the full E2E test suite |
| `label-pr` | auto-labels the PR based on changed file paths |
All four checks must pass. One maintainer approval is required before merge.
## Project philosophy
This repository is an archive. The goal is a complete collection of every file a
user needs to run every supported console in an emulator, with zero friction. That
scope goes beyond BIOS ROMs: firmware updates, system fonts, data files, encryption
keys, hiscore databases, anything the emulator loads from disk rather than generating
itself. In a hundred years the pack should still work out of the box.
Licensing considerations are secondary to completeness. If an emulator needs a
file to function, the file belongs in the collection. The project's legal position is
documented in the FAQ section of the site.
Integrity matters. Every file, hash, and metadata field is cross-checked
against the emulator's source code. Upstream references like System.dat,
`.info` files, and wiki pages are valuable and generally accurate, though
they can occasionally fall out of date. When an upstream source and the
code disagree, the code at runtime is the tiebreaker.
## Documentation
Full reference docs, profiling guides, and architecture details are on the [documentation site](https://abdess.github.io/retrobios/).

View File

@@ -2,7 +2,7 @@
Complete BIOS and firmware packs for Batocera, BizHawk, EmuDeck, Lakka, Recalbox, RetroArch, RetroBat, RetroDECK, RetroPie, and RomM. Complete BIOS and firmware packs for Batocera, BizHawk, EmuDeck, Lakka, Recalbox, RetroArch, RetroBat, RetroDECK, RetroPie, and RomM.
**7,241** verified files across **396** systems, ready to extract into your emulator's BIOS directory. **7,293** verified files across **396** systems, ready to extract into your emulator's BIOS directory.
## Quick Install ## Quick Install
@@ -46,8 +46,8 @@ Each file is checked against the emulator's source code to match what the code a
- **10 platforms** supported with platform-specific verification - **10 platforms** supported with platform-specific verification
- **329 emulators** profiled from source (RetroArch cores + standalone) - **329 emulators** profiled from source (RetroArch cores + standalone)
- **396 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...) - **396 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
- **7,241 files** verified with MD5, SHA1, CRC32 checksums - **7,293 files** verified with MD5, SHA1, CRC32 checksums
- **8144 MB** total collection size - **8710 MB** total collection size
## Supported systems ## Supported systems
@@ -59,15 +59,15 @@ Full list with per-file details: **[https://abdess.github.io/retrobios/](https:/
| Platform | Coverage | Verified | Untested | Missing | | Platform | Coverage | Verified | Untested | Missing |
|----------|----------|----------|----------|---------| |----------|----------|----------|----------|---------|
| Batocera | 356/362 (98.3%) | 349 | 7 | 6 | | Batocera | 361/362 (99.7%) | 354 | 7 | 1 |
| BizHawk | 118/118 (100.0%) | 118 | 0 | 0 | | BizHawk | 118/118 (100.0%) | 118 | 0 | 0 |
| EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 | | EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 |
| Lakka | 442/448 (98.7%) | 442 | 0 | 6 | | Lakka | 443/448 (98.9%) | 443 | 0 | 5 |
| Recalbox | 277/346 (80.1%) | 274 | 3 | 69 | | Recalbox | 277/346 (80.1%) | 274 | 3 | 69 |
| RetroArch | 442/448 (98.7%) | 442 | 0 | 6 | | RetroArch | 443/448 (98.9%) | 443 | 0 | 5 |
| RetroBat | 339/339 (100.0%) | 335 | 4 | 0 | | RetroBat | 339/339 (100.0%) | 335 | 4 | 0 |
| RetroDECK | 1960/2006 (97.7%) | 1934 | 26 | 46 | | RetroDECK | 1960/2006 (97.7%) | 1934 | 26 | 46 |
| RetroPie | 442/448 (98.7%) | 442 | 0 | 6 | | RetroPie | 443/448 (98.9%) | 443 | 0 | 5 |
| RomM | 372/374 (99.5%) | 372 | 0 | 2 | | RomM | 372/374 (99.5%) | 372 | 0 | 2 |
## Build your own pack ## Build your own pack
@@ -130,4 +130,4 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
This repository provides BIOS files for personal backup and archival purposes. This repository provides BIOS files for personal backup and archival purposes.
*Auto-generated on 2026-03-30T20:16:27Z* *Auto-generated on 2026-03-30T23:36:52Z*

File diff suppressed because it is too large Load Diff

View File

@@ -33,6 +33,8 @@ systems:
- taito-cchip - taito-cchip
- ym2608 - ym2608
archive_prefix: fbneo
data_directories: data_directories:
- ref: fbneo-cheats - ref: fbneo-cheats
destination: fbneo/cheats destination: fbneo/cheats

View File

@@ -32,6 +32,8 @@ notes: |
need_fullpath=false, extensions=zip|7z, savestate=deterministic. need_fullpath=false, extensions=zip|7z, savestate=deterministic.
archive_prefix: fbneo
files: files:
- name: "hiscore.dat" - name: "hiscore.dat"
path: "fbneo/hiscore.dat" path: "fbneo/hiscore.dat"

View File

@@ -38,6 +38,8 @@ notes: |
need_fullpath=false, extensions=zip|7z|cue|ccd, savestate=deterministic. need_fullpath=false, extensions=zip|7z|cue|ccd, savestate=deterministic.
archive_prefix: fbneo
files: files:
# ------------------------------------------------------- # -------------------------------------------------------
# Neo Geo MVS/AES (neogeo.zip) — 68K BIOS ROMs # Neo Geo MVS/AES (neogeo.zip) — 68K BIOS ROMs

View File

@@ -38,11 +38,14 @@ notes: |
Standalone supports MPEG card ROM loading (Video CD card); disabled in Standalone supports MPEG card ROM loading (Video CD card); disabled in
libretro port (mpegpath = NULL in libretro.c:1578). libretro port (mpegpath = NULL in libretro.c:1578).
archive_prefix: kronos
files: files:
# ----------------------------------------------------------- # -----------------------------------------------------------
# Saturn BIOS - primary (any region) # Saturn BIOS - primary (any region)
# ----------------------------------------------------------- # -----------------------------------------------------------
- name: "saturn_bios.bin" - name: "saturn_bios.bin"
path: "kronos/saturn_bios.bin"
system: sega-saturn system: sega-saturn
required: true required: true
size: 524288 size: 524288

File diff suppressed because it is too large Load Diff

View File

@@ -3,7 +3,7 @@
"platform": "bizhawk", "platform": "bizhawk",
"display_name": "BizHawk", "display_name": "BizHawk",
"version": "1.0", "version": "1.0",
"generated": "2026-03-30T09:46:23Z", "generated": "2026-03-30T22:08:44Z",
"base_destination": "Firmware", "base_destination": "Firmware",
"detect": [ "detect": [
{ {
@@ -18,8 +18,8 @@
} }
], ],
"standalone_copies": [], "standalone_copies": [],
"total_files": 437, "total_files": 456,
"total_size": 1790314370, "total_size": 1805641545,
"files": [ "files": [
{ {
"dest": "panafz1.bin", "dest": "panafz1.bin",
@@ -2623,6 +2623,177 @@
"MAME" "MAME"
] ]
}, },
{
"dest": "ekara.zip",
"sha1": "86665ff4bce0f27c1ffd1d0459708885b82983a2",
"size": 630644,
"repo_path": "bios/Arcade/Arcade/ekara.zip",
"cores": [
"MAME"
]
},
{
"dest": "ekaraa.zip",
"sha1": "98080e5a3d352e04ed8b50e6a04af456518aa66e",
"size": 629642,
"repo_path": "bios/Arcade/Arcade/ekaraa.zip",
"cores": [
"MAME"
]
},
{
"dest": "ekaraj.zip",
"sha1": "d4fa61d730b6aaf354bbec5e997c0db30efc85d0",
"size": 629853,
"repo_path": "bios/Arcade/Arcade/ekaraj.zip",
"cores": [
"MAME"
]
},
{
"dest": "ekarag.zip",
"sha1": "39e589aa0158b48d33648413c89778f8e8cc0d58",
"size": 795612,
"repo_path": "bios/Arcade/Arcade/ekarag.zip",
"cores": [
"MAME"
]
},
{
"dest": "ekaras.zip",
"sha1": "ab288761b8cd5a02fc7b3d12acbb1e3371214b69",
"size": 813756,
"repo_path": "bios/Arcade/Arcade/ekaras.zip",
"cores": [
"MAME"
]
},
{
"dest": "isinger.zip",
"sha1": "28c6f8828b6820c072832fa7027beb7be9aad020",
"size": 556765,
"repo_path": "bios/Arcade/Arcade/isinger.zip",
"cores": [
"MAME"
]
},
{
"dest": "ekaraphs.zip",
"sha1": "31199ff06972ba2a1a67b1b403119ee3d821efc7",
"size": 798457,
"repo_path": "bios/Arcade/Arcade/ekaraphs.zip",
"cores": [
"MAME"
]
},
{
"dest": "epitch.zip",
"sha1": "d4fa61d730b6aaf354bbec5e997c0db30efc85d0",
"size": 629853,
"repo_path": "bios/Arcade/Arcade/ekaraj.zip",
"cores": [
"MAME"
]
},
{
"dest": "ekaramix.zip",
"sha1": "08cea726163f490471d88e4c640b8385ee065836",
"size": 663402,
"repo_path": "bios/Arcade/Arcade/ekaramix.zip",
"cores": [
"MAME"
]
},
{
"dest": "ddrfammt.zip",
"sha1": "ec9a6c1bf8f33f5717d51588ffe87239313b2a06",
"size": 883352,
"repo_path": "bios/Arcade/Arcade/ddrfammt.zip",
"cores": [
"MAME"
]
},
{
"dest": "popira.zip",
"sha1": "5fb387eef5d254797413c9d0ea342b64b7eeb5bb",
"size": 654918,
"repo_path": "bios/Arcade/Arcade/popira.zip",
"cores": [
"MAME"
]
},
{
"dest": "popirak.zip",
"sha1": "9801ee035decbb5e45aa0a20ca4d26323e0ac126",
"size": 639838,
"repo_path": "bios/Arcade/Arcade/popirak.zip",
"cores": [
"MAME"
]
},
{
"dest": "popira2.zip",
"sha1": "5143c86ac93607223cafb5e529ea08221518e64a",
"size": 1124630,
"repo_path": "bios/Arcade/Arcade/popira2.zip",
"cores": [
"MAME"
]
},
{
"dest": "taikodp.zip",
"sha1": "446013455ab02be7fc3e27bf6ff680293c8657b7",
"size": 1141771,
"repo_path": "bios/Arcade/Arcade/taikodp.zip",
"cores": [
"MAME"
]
},
{
"dest": "jpopira.zip",
"sha1": "500a2402fcdf856d127128153e62e51c3c2f7bdc",
"size": 1116085,
"repo_path": "bios/Arcade/Arcade/jpopira.zip",
"cores": [
"MAME"
]
},
{
"dest": "evio.zip",
"sha1": "96ac6cc92b40e57f04be7215703e532394bead55",
"size": 1292911,
"repo_path": "bios/Arcade/Arcade/evio.zip",
"cores": [
"MAME"
]
},
{
"dest": "tak_daig.zip",
"sha1": "d3c641bdde6c6f681abd3bdd36463d1e7264b6e7",
"size": 951997,
"repo_path": "bios/Arcade/Arcade/tak_daig.zip",
"cores": [
"MAME"
]
},
{
"dest": "gcslottv.zip",
"sha1": "4ea3ec9c41ab767907167b5022bfcfd1a05795c8",
"size": 737452,
"repo_path": "bios/Arcade/Arcade/gcslottv.zip",
"cores": [
"MAME"
]
},
{
"dest": "hikara.zip",
"sha1": "dbfdea3057a5fcc0e0d243deace87fd6840a2322",
"size": 636237,
"repo_path": "bios/Arcade/Arcade/hikara.zip",
"cores": [
"MAME"
]
},
{ {
"dest": "bios9.bin", "dest": "bios9.bin",
"sha1": "bfaac75f101c135e32e2aaf541de6b1be4c8c62d", "sha1": "bfaac75f101c135e32e2aaf541de6b1be4c8c62d",

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -337,6 +337,8 @@ def _collect_emulator_extras(
from common import resolve_platform_cores from common import resolve_platform_cores
from verify import find_undeclared_files from verify import find_undeclared_files
profiles = emu_profiles if emu_profiles is not None else load_emulator_profiles(emulators_dir)
undeclared = find_undeclared_files(config, emulators_dir, db, emu_profiles, target_cores=target_cores) undeclared = find_undeclared_files(config, emulators_dir, db, emu_profiles, target_cores=target_cores)
extras = [] extras = []
seen_dests: set[str] = set(seen) seen_dests: set[str] = set(seen)
@@ -364,7 +366,6 @@ def _collect_emulator_extras(
# different path by another core (e.g. neocd/ vs root, same_cdi/bios/ vs root). # different path by another core (e.g. neocd/ vs root, same_cdi/bios/ vs root).
# Only adds a copy when the file is ALREADY covered at a different path - # Only adds a copy when the file is ALREADY covered at a different path -
# never introduces a file that wasn't selected by the first pass. # never introduces a file that wasn't selected by the first pass.
profiles = emu_profiles if emu_profiles is not None else load_emulator_profiles(emulators_dir)
relevant = resolve_platform_cores(config, profiles, target_cores=target_cores) relevant = resolve_platform_cores(config, profiles, target_cores=target_cores)
standalone_set = {str(c) for c in config.get("standalone_cores", [])} standalone_set = {str(c) for c in config.get("standalone_cores", [])}
by_name = db.get("indexes", {}).get("by_name", {}) by_name = db.get("indexes", {}).get("by_name", {})
@@ -422,6 +423,41 @@ def _collect_emulator_extras(
"source_emulator": profile.get("emulator", emu_name), "source_emulator": profile.get("emulator", emu_name),
}) })
# Archive prefix pass: cores that store BIOS archives in a subdirectory
# (e.g. system/fbneo/neogeo.zip). When the archive is already covered at
# the root, add a copy at the prefixed path so the core's .info firmware
# check finds it.
for emu_name, profile in sorted(profiles.items()):
if profile.get("type") in ("launcher", "alias"):
continue
if emu_name not in relevant:
continue
prefix = profile.get("archive_prefix", "")
if not prefix:
continue
profile_archives: set[str] = set()
for f in profile.get("files", []):
archive = f.get("archive", "")
if archive:
profile_archives.add(archive)
for archive_name in sorted(profile_archives):
if archive_name not in covered_names:
continue
dest = f"{prefix}/{archive_name}"
full_dest = f"{base_dest}/{dest}" if base_dest else dest
if full_dest in seen_dests:
continue
if not by_name.get(archive_name):
continue
seen_dests.add(full_dest)
extras.append({
"name": archive_name,
"destination": dest,
"required": True,
"hle_fallback": False,
"source_emulator": profile.get("emulator", emu_name),
})
# Third pass: agnostic scan — for filename-agnostic cores, include all # Third pass: agnostic scan — for filename-agnostic cores, include all
# DB files matching the system path prefix and size criteria. # DB files matching the system path prefix and size criteria.
files_db = db.get("files", {}) files_db = db.get("files", {})
@@ -1066,9 +1102,10 @@ def generate_pack(
if _has_path_conflict(full_dest, seen_destinations, seen_parents): if _has_path_conflict(full_dest, seen_destinations, seen_parents):
continue continue
dest_hint = fe.get("destination", "")
local_path, status = resolve_file( local_path, status = resolve_file(
fe, db, bios_dir, zip_contents, fe, db, bios_dir, zip_contents,
data_dir_registry=data_registry, dest_hint=dest_hint, data_dir_registry=data_registry,
) )
if status in ("not_found", "external", "user_provided"): if status in ("not_found", "external", "user_provided"):
continue continue
@@ -1181,6 +1218,9 @@ def _normalize_zip_for_pack(source_zip: str, dest_path: str, target_zf: zipfile.
try: try:
rebuild_zip_deterministic(source_zip, tmp_path) rebuild_zip_deterministic(source_zip, tmp_path)
target_zf.write(tmp_path, dest_path) target_zf.write(tmp_path, dest_path)
except zipfile.BadZipFile:
# Corrupt source ZIP: copy as-is (will be flagged by verify)
target_zf.write(source_zip, dest_path)
finally: finally:
os.unlink(tmp_path) os.unlink(tmp_path)
@@ -1318,8 +1358,11 @@ def generate_emulator_pack(
archives.add(archive) archives.add(archive)
# Pack archives as units # Pack archives as units
archive_prefix = profile.get("archive_prefix", "")
for archive_name in sorted(archives): for archive_name in sorted(archives):
archive_dest = _sanitize_path(archive_name) archive_dest = _sanitize_path(archive_name)
if archive_prefix:
archive_dest = f"{archive_prefix}/{archive_dest}"
if pack_structure: if pack_structure:
mode_key = "standalone" if standalone else "libretro" mode_key = "standalone" if standalone else "libretro"
prefix = pack_structure.get(mode_key, "") prefix = pack_structure.get(mode_key, "")
@@ -2262,7 +2305,9 @@ def generate_manifest(
if _has_path_conflict(full_dest, seen_destinations, seen_parents): if _has_path_conflict(full_dest, seen_destinations, seen_parents):
continue continue
local_path, status = resolve_file(fe, db, bios_dir, zip_contents) dest_hint = fe.get("destination", "")
local_path, status = resolve_file(fe, db, bios_dir, zip_contents,
dest_hint=dest_hint)
if status in ("not_found", "external", "user_provided"): if status in ("not_found", "external", "user_provided"):
continue continue