diff --git a/README.md b/README.md index 57ac75dc..5c10fccf 100644 --- a/README.md +++ b/README.md @@ -32,20 +32,20 @@ The script auto-detects your platform, downloads only missing files, and verifie Pick your platform, download the ZIP, extract to the BIOS path. -| Platform | BIOS files | Extract to | Download | -|----------|-----------|-----------|----------| -| Batocera | 353 | `/userdata/bios/` | [Download](../../releases/latest) | -| BizHawk | 118 | `Firmware/` | [Download](../../releases/latest) | -| EmuDeck | 161 | `~/Emulation/bios/` | [Download](../../releases/latest) | -| Lakka | 530 | `/storage/system/` | [Download](../../releases/latest) | -| MiSTer FPGA | 65 | `/media/fat/games/` | [Download](../../releases/latest) | -| ROCKNIX | 38 | `/storage/roms/bios/` | [Download](../../releases/latest) | -| Recalbox | 346 | `/recalbox/share/bios/` | [Download](../../releases/latest) | -| RetroArch | 530 | `system/` | [Download](../../releases/latest) | -| RetroBat | 341 | `bios/` | [Download](../../releases/latest) | -| RetroDECK | 2008 | `~/retrodeck/` | [Download](../../releases/latest) | -| RetroPie * | 530 | `~/RetroPie/BIOS/` | [Download](../../releases/latest) | -| RomM | 374 | `bios/{platform_slug}/` | [Download](../../releases/latest) | +| Platform | Size | Extract to | Download | +|----------|------|-----------|----------| +| Batocera | 2.7 GB | `/userdata/bios/` | [Download](../../releases/latest) | +| BizHawk | 2.4 GB | `Firmware/` | [Download](../../releases/latest) | +| EmuDeck | 1.6 GB | `~/Emulation/bios/` | [Download](../../releases/latest) | +| Lakka | 3.9 GB | `/storage/system/` | [Download](../../releases/latest) | +| MiSTer FPGA | 23 MB | `/media/fat/games/` | [Download](../../releases/latest) | +| ROCKNIX | 3.8 GB | `/storage/roms/bios/` | [Download](../../releases/latest) | +| Recalbox | 2.3 GB | `/recalbox/share/bios/` | [Download](../../releases/latest) | +| RetroArch | 3.9 GB | `system/` | [Download](../../releases/latest) | +| RetroBat | 3.1 GB | `bios/` | [Download](../../releases/latest) | +| RetroDECK | 4.6 GB | `~/retrodeck/` | [Download](../../releases/latest) | +| RetroPie * | - | `~/RetroPie/BIOS/` | [Download](../../releases/latest) | +| RomM | 1.4 GB | `bios/{platform_slug}/` | [Download](../../releases/latest) | The RetroDECK pack already contains its own `bios/` folder, so it extracts into `~/retrodeck/` rather than into the BIOS folder. @@ -57,7 +57,7 @@ BIOS, firmware, and system files for consoles from Atari to PlayStation 3. Every file passes its platform's own verification; where an emulator profile exists, the expected hashes and sizes are read from the emulator's source code (the Source-backed column below). - **12 platforms** supported with platform-specific verification -- **318 emulators** profiled from source (RetroArch cores + standalone) +- **320 emulators** profiled from source (RetroArch cores + standalone) - **396 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...) - **7,651 files** indexed with SHA1, MD5, SHA256 and CRC32 checksums: 2,366 system files, 2,746 arcade ROM sets, 2,539 game and engine data files - **527 files** matched to dump-preservation catalogs (No-Intro, Redump, TOSEC) @@ -71,24 +71,24 @@ Full list with per-file details: **[https://abdess.github.io/retrobios/](https:/ ## Coverage -| Platform | Coverage | Verified | Untested | Missing | Source-backed | -|----------|----------|----------|----------|---------|---------------| -| Batocera | 353/353 (100.0%) | 353 | 0 | 0 | 96/353 (27%) | -| BizHawk | 118/118 (100.0%) | 118 | 0 | 0 | 4/118 (3%) | -| EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 | 16/161 (10%) | -| Lakka | 530/530 (100.0%) | 530 | 0 | 0 | 124/530 (23%) | -| MiSTer FPGA | 65/65 (100.0%) | 65 | 0 | 0 | - | -| ROCKNIX | 38/38 (100.0%) | 38 | 0 | 0 | 29/38 (76%) | -| Recalbox | 346/346 (100.0%) | 346 | 0 | 0 | 86/346 (25%) | -| RetroArch | 530/530 (100.0%) | 530 | 0 | 0 | 124/530 (23%) | -| RetroBat | 341/341 (100.0%) | 341 | 0 | 0 | 84/341 (25%) | -| RetroDECK | 2008/2008 (100.0%) | 2008 | 0 | 0 | 121/2008 (6%) | -| RetroPie * | 530/530 (100.0%) | 530 | 0 | 0 | 124/530 (23%) | -| RomM | 374/374 (100.0%) | 374 | 0 | 0 | 88/374 (24%) | +| Platform | Files in pack | Still missing | Verified by | +|----------|--------------:|--------------:|-------------| +| Batocera | 1,601 | 4 | MD5 hash | +| BizHawk | 539 | 5 | SHA1 hash | +| EmuDeck | 525 | 0 | MD5 hash | +| Lakka | 1,631 | 4 | file presence | +| MiSTer FPGA | 65 | 0 | MD5 hash | +| ROCKNIX | 1,496 | 5 | MD5 hash | +| Recalbox | 1,184 | 6 | MD5 hash | +| RetroArch | 1,631 | 4 | file presence | +| RetroBat | 1,235 | 4 | MD5 hash | +| RetroDECK | 3,266 | 4 | MD5 hash | +| RetroPie * | 1,678 | 4 | file presence | +| RomM | 543 | 4 | MD5 hash | -Coverage is measured against the file list each platform declares, using that platform's own verification mode. -Source-backed counts the files whose content the emulator's own code checks: a size or hash read from its source, reproduced at verification. A dash means no profiled emulator applies to the platform, whose own source is then the only authority. -The [gap analysis](https://abdess.github.io/retrobios/gaps/) page counts separately the files a profile documents without a content check, and tracks where platform lists and emulator source code disagree. +A pack carries what the platform declares plus what its emulators load without the platform listing it, so it holds more files than the platform's own list. +Still missing counts files an emulator needs that are not in the collection yet; verified by is the check the platform itself runs on them. +The [gap analysis](https://abdess.github.io/retrobios/gaps/) page names those missing files and details how far each platform's files are corroborated against emulator source code. ## Build your own pack @@ -124,7 +124,7 @@ The [documentation site](https://abdess.github.io/retrobios/) provides: - **Per-emulator profiles** with source code references for every file - **Per-system pages** showing which emulators and platforms cover each console - **Gap analysis** identifying missing files and undeclared core requirements -- **Cross-reference** mapping files across 12 platforms and 318 emulators +- **Cross-reference** mapping files across 12 platforms and 320 emulators ## How it works @@ -161,4 +161,4 @@ The scripts and tooling are released under the [MIT License](LICENSE). The BIOS and firmware files are not covered by that license: they are third-party system software, preserved and provided for personal backup, archival, and interoperability with emulation software. The legal reasoning is laid out in the [FAQ](https://abdess.github.io/retrobios/wiki/faq/#is-this-legal). -*Auto-generated on 2026-08-08T02:07:01Z* +*Auto-generated on 2026-08-08T03:39:12Z* diff --git a/scripts/generate_readme.py b/scripts/generate_readme.py index f541a0ca..3a171ed2 100644 --- a/scripts/generate_readme.py +++ b/scripts/generate_readme.py @@ -51,6 +51,9 @@ def compute_coverage( total = result["total_files"] present = ok + untested pct = (present / total * 100) if total > 0 else 0 + undeclared = result.get("undeclared_files", []) + core_present = sum(1 for u in undeclared if u.get("in_repo")) + core_missing = len(undeclared) - core_present return { "platform": config.get("platform", platform_name), "total": total, @@ -59,6 +62,10 @@ def compute_coverage( "missing": missing, "present": present, "percentage": pct, + "core_present": core_present, + "core_missing": core_missing, + "pack_files": present + core_present, + "total_missing": missing + core_missing, "mode": config.get("verification_mode", "existence"), "details": result["details"], "config": config, @@ -69,6 +76,33 @@ def compute_coverage( } +def manifest_totals( + platform_name: str, install_dir: str = "install" +) -> tuple[int | None, int | None]: + """Files and bytes a platform's pack ships, from its install manifest. + + The manifest is written when packs are built, so it reflects the real + pack contents (platform list, core complement and data directories). + Returns (None, None) when no manifest exists yet. + """ + path = os.path.join(install_dir, f"{platform_name}.json") + if not os.path.exists(path): + return None, None + try: + with open(path) as f: + manifest = json.load(f) + except (json.JSONDecodeError, OSError): + return None, None + return manifest.get("total_files"), manifest.get("total_size") + + +def format_size(size: int) -> str: + """Human-readable pack size.""" + if size >= 1024**3: + return f"{size / 1024**3:.1f} GB" + return f"{size / 1024**2:.0f} MB" + + SITE_URL = "https://abdess.github.io/retrobios/" RELEASE_URL = "../../releases/latest" REPO = "Abdess/retrobios" @@ -195,8 +229,8 @@ def generate_readme(db: dict, platforms_dir: str) -> str: "", "Pick your platform, download the ZIP, extract to the BIOS path.", "", - "| Platform | BIOS files | Extract to | Download |", - "|----------|-----------|-----------|----------|", + "| Platform | Size | Extract to | Download |", + "|----------|------|-----------|----------|", ] # Where the pack itself is extracted, which is not always the BIOS folder: @@ -227,8 +261,10 @@ def generate_readme(db: dict, platforms_dir: str) -> str: if name in archived: display = f"{display} *" path = extract_paths.get(cov["platform"], "") + _, size = manifest_totals(name) + size_cell = format_size(size) if size else "-" lines.append( - f"| {display} | {cov['total']} | {path} | [Download]({RELEASE_URL}) |" + f"| {display} | {size_cell} | {path} | [Download]({RELEASE_URL}) |" ) if archived: @@ -313,40 +349,37 @@ def generate_readme(db: dict, platforms_dir: str) -> str: "", "## Coverage", "", - "| Platform | Coverage | Verified | Untested | Missing | Source-backed |", - "|----------|----------|----------|----------|---------|---------------|", + "| Platform | Files in pack | Still missing | Verified by |", + "|----------|--------------:|--------------:|-------------|", ] ) + mode_labels = { + "md5": "MD5 hash", + "sha1": "SHA1 hash", + "existence": "file presence", + } + for name, cov in sorted(coverages.items(), key=lambda x: x[1]["platform"]): - pct = f"{cov['percentage']:.1f}%" - gt = cov["ground_truth"] - if not gt.get("applicable", True): - gt_cell = "-" - elif gt["total"]: - gt_pct = f"{gt['with_validation'] / gt['total'] * 100:.0f}%" - gt_cell = f"{gt['with_validation']}/{gt['total']} ({gt_pct})" - else: - gt_cell = "0/0" display = f"{cov['platform']} *" if name in archived else cov["platform"] + files = manifest_totals(name)[0] or cov["pack_files"] + checked = mode_labels.get(cov["mode"], cov["mode"]) lines.append( - f"| {display} | {cov['present']}/{cov['total']} ({pct}) | " - f"{cov['verified']} | {cov['untested']} | {cov['missing']} | " - f"{gt_cell} |" + f"| {display} | {files:,} | {cov['total_missing']} | {checked} |" ) lines.extend( [ "", - "Coverage is measured against the file list each platform declares," - " using that platform's own verification mode.", - "Source-backed counts the files whose content the emulator's own" - " code checks: a size or hash read from its source, reproduced at" - " verification. A dash means no profiled emulator applies to the" - " platform, whose own source is then the only authority.", - f"The [gap analysis]({SITE_URL}gaps/) page counts separately the" - " files a profile documents without a content check, and tracks" - " where platform lists and emulator source code disagree.", + "A pack carries what the platform declares plus what its emulators" + " load without the platform listing it, so it holds more files than" + " the platform's own list.", + "Still missing counts files an emulator needs that are not in the" + " collection yet; verified by is the check the platform itself runs" + " on them.", + f"The [gap analysis]({SITE_URL}gaps/) page names those missing files" + " and details how far each platform's files are corroborated against" + " emulator source code.", "", "## Build your own pack", "", diff --git a/scripts/generate_site.py b/scripts/generate_site.py index ef24692c..9187b3e2 100644 --- a/scripts/generate_site.py +++ b/scripts/generate_site.py @@ -32,7 +32,7 @@ from common import ( ) yaml = require_yaml() -from generate_readme import compute_coverage +from generate_readme import compute_coverage, manifest_totals from provenance_report import build_report DOCS_DIR = "docs" @@ -1731,58 +1731,77 @@ def generate_gap_analysis( # ---- Verification per platform ---- lines.extend([ - "## Verification by Platform", + "## What Each Pack Contains", "", - "| Platform | Files | Verified | Untested | Missing | Mode | Profiled | Source-backed |", - "|----------|------:|---------:|---------:|--------:|------|---------:|--------------:|", + "| Platform | Files in pack | Declared by platform | Still missing | Verified by |", + "|----------|--------------:|---------------------:|--------------:|-------------|", ]) + mode_labels = { + "md5": "MD5 hash", + "sha1": "SHA1 hash", + "existence": "file presence", + } + for pname, cov in sorted(coverages.items(), key=lambda x: x[1]["platform"]): display = cov["platform"] - m = cov["missing"] - u = cov["untested"] + gone = cov["total_missing"] missing_str = ( - f'{m}' - if m > 0 + f'{gone}' + if gone > 0 else '0' ) - untested_str = ( - f'{u}' - if u > 0 - else str(u) - ) - gt = cov["ground_truth"] - if not gt.get("applicable", True): - gt_cell = prof_cell = "-" - elif gt["total"]: - gt_pct = f"{gt['with_validation'] / gt['total'] * 100:.0f}%" - gt_cell = f"{gt['with_validation']}/{gt['total']} ({gt_pct})" - prof_pct = f"{gt.get('with_profile', 0) / gt['total'] * 100:.0f}%" - prof_cell = f"{gt.get('with_profile', 0)}/{gt['total']} ({prof_pct})" - else: - gt_cell = prof_cell = "0/0" + files = manifest_totals(pname)[0] or cov["pack_files"] lines.append( f"| [{display}](platforms/{pname}.md) " - f"| {cov['total']} " - f"| {cov['verified']} " - f"| {untested_str} " + f"| {files:,} " + f"| {cov['present']:,} " f"| {missing_str} " - f"| {cov['mode']} " - f"| {prof_cell} " - f"| {gt_cell} |" + f"| {mode_labels.get(cov['mode'], cov['mode'])} |" ) lines.extend([ "", - "Verification follows each platform's own runtime check " - "([how each mode works](wiki/verification-modes.md)): the counts " - "measure the repository against the file list each platform declares. " - "Profiled counts the files documented in an emulator profile (name or " - "alias read from source). Source-backed is stricter: the emulator's " - "own code checks the file's content (a size or hash read from its " - "source, reproduced at verification). A file can be profiled without " - "being source-backed when the code loads it but never checks it. " - "A dash means no profiled emulator applies to the platform, whose " - "own source is then the only authority.", + "A pack carries the platform's own file list plus every file its " + "emulators load without that list mentioning them, which is why it " + "ships several times what the platform declares. Still missing counts " + "files an emulator needs that are not in the collection yet, named in " + "the sections below. Verified by is the check the platform runs at " + "runtime, replicated here " + "([how each mode works](wiki/verification-modes.md)).", + "", + "## Corroboration Against Emulator Source", + "", + "| Platform | Declared by platform | Profiled | Content checked by emulator code |", + "|----------|---------------------:|---------:|---------------------------------:|", + ]) + + for pname, cov in sorted(coverages.items(), key=lambda x: x[1]["platform"]): + gt = cov["ground_truth"] + if not gt.get("applicable", True): + prof_cell = gt_cell = "-" + elif gt["total"]: + prof_pct = f"{gt.get('with_profile', 0) / gt['total'] * 100:.0f}%" + prof_cell = f"{gt.get('with_profile', 0)} ({prof_pct})" + gt_pct = f"{gt['with_validation'] / gt['total'] * 100:.0f}%" + gt_cell = f"{gt['with_validation']} ({gt_pct})" + else: + prof_cell = gt_cell = "0" + lines.append( + f"| [{cov['platform']}](platforms/{pname}.md) " + f"| {gt['total']:,} | {prof_cell} | {gt_cell} |" + ) + + lines.extend([ + "", + "Both columns count the files a platform declares. Profiled means the " + "file is documented in an emulator profile read from source. Content " + "checked is stricter: the emulator's code verifies a size or hash for " + "it, and this tool reproduces that check. The gap between the two is " + "not a defect, it is what the emulator code does: many emulators load " + "a file without ever checking its content, and no amount of profiling " + "can invent a check the code does not perform. A dash means no " + "profiled emulator applies to the platform, whose own source is then " + "the only authority.", "", ])