Merge branch 'main' of https://github.com/Abdess/retrobios

9 new wiki pages: getting-started, faq, troubleshooting,
advanced-usage, verification-modes, adding-a-platform,
adding-a-scraper, testing-guide, release-process.

Updated architecture.md with mermaid diagrams, tools.md with
full pipeline and target/exporter sections, profiling.md with
missing fields, index.md with glossary and nav links.

Expanded CONTRIBUTING.md from stub to full contributor guide.

Filter .old.yml from load_emulator_profiles, generate_db alias
collection, and generate_readme counts. Fix BizHawk sha1 mode
in tools.md, fix RetroPie path, fix export_truth.py typos.

.github/workflows/build.yml

@@ -18,13 +18,11 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  regenerate:
+  release:
+    if: false  # disabled until pack generation is validated in production
     runs-on: ubuntu-latest
     permissions:
       contents: write
-      pages: write
-    outputs:
-      changed: ${{ steps.commit.outputs.changed }}
     steps:
       - uses: actions/checkout@v6
 
@@ -34,57 +32,9 @@ jobs:
 
       - run: pip install pyyaml
 
-      - name: Restore large files from release
-        run: |
-          mkdir -p .cache/large
-          gh release download large-files -D .cache/large/ 2>/dev/null || true
-          # Copy large files to their bios/ paths so generate_db sees them
-          for f in .cache/large/*; do
-            [ -f "$f" ] || continue
-            name=$(basename "$f")
-            # Match against .gitignore entries to find target path
-            target=$(grep "$name" .gitignore | head -1)
-            if [ -n "$target" ] && [ ! -f "$target" ]; then
-              mkdir -p "$(dirname "$target")"
-              cp "$f" "$target"
-              echo "Restored: $target"
-            fi
-          done
-        env:
-          GH_TOKEN: ${{ github.token }}
-
-      - name: Regenerate database, readme, and site
-        run: |
-          pip install mkdocs-material
-          python scripts/generate_db.py --force --bios-dir bios --output database.json
-          python scripts/generate_readme.py --db database.json --platforms-dir platforms
-          python scripts/generate_site.py
-
       - name: Run tests
         run: python -m unittest tests.test_e2e -v
 
-      - name: Deploy site to GitHub Pages
-        if: github.ref == 'refs/heads/main'
-        run: mkdocs gh-deploy --force --clean
-
-      - name: Commit if changed
-        id: commit
-        run: |
-          git diff --quiet database.json README.md CONTRIBUTING.md && echo "changed=false" >> "$GITHUB_OUTPUT" && exit 0
-          git config user.name "github-actions[bot]"
-          git config user.email "github-actions[bot]@users.noreply.github.com"
-          git add database.json README.md CONTRIBUTING.md
-          git commit -m "regenerate database and docs"
-          git push
-          echo "changed=true" >> "$GITHUB_OUTPUT"
-
-  release:
-    needs: regenerate
-    if: (needs.regenerate.outputs.changed == 'true' || github.event.inputs.force_release == 'true')
-    runs-on: ubuntu-latest
-    permissions:
-      contents: write
-    steps:
       - name: Rate limit
         if: github.event.inputs.force_release != 'true'
         id: rate
@@ -103,20 +53,6 @@ jobs:
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
-      - uses: actions/checkout@v6
-        if: steps.rate.outputs.skip != 'true'
-        with:
-          ref: main
-
-      - uses: actions/setup-python@v6
-        if: steps.rate.outputs.skip != 'true'
-        with:
-          python-version: "3.12"
-
-      - name: Install dependencies
-        if: steps.rate.outputs.skip != 'true'
-        run: pip install pyyaml
-
       - name: Restore large files from release
         if: steps.rate.outputs.skip != 'true'
         run: |
@@ -142,9 +78,7 @@ jobs:
 
       - name: Build packs
         if: steps.rate.outputs.skip != 'true'
-        run: |
-          python scripts/generate_db.py --bios-dir bios --output database.json
-          python scripts/generate_pack.py --all --output-dir dist/
+        run: python scripts/generate_pack.py --all --output-dir dist/
 
       - name: Release
         if: steps.rate.outputs.skip != 'true'
@@ -168,14 +102,17 @@ jobs:
           ${PACKS}
 
           ### Install
-          Download, extract to your emulator's BIOS directory.
+          Download the pack matching your frontend, extract to the BIOS directory.
 
-          | Platform | Path |
-          |----------|------|
-          | RetroArch / Lakka | system/ |
-          | Batocera | /userdata/bios/ |
-          | Recalbox | /recalbox/share/bios/ |
-          | RetroBat | bios/ |
+          | Platform | Pack | Path |
+          |----------|------|------|
+          | RetroArch / Lakka | RetroArch_Lakka_BIOS_Pack.zip | system/ |
+          | Batocera | Batocera_BIOS_Pack.zip | /userdata/bios/ |
+          | Recalbox | Recalbox_BIOS_Pack.zip | /recalbox/share/bios/ |
+          | RetroBat | RetroBat_BIOS_Pack.zip | bios/ |
+          | RetroDECK | RetroDECK_BIOS_Pack.zip | ~/retrodeck/bios/ |
+          | EmuDeck | EmuDeck_BIOS_Pack.zip | Emulation/bios/ |
+          | RomM | RomM_BIOS_Pack.zip | bios/{platform_slug}/ |
 
           ### Changes
           ${CHANGES}

.github/workflows/deploy-site.yml

@@ -0,0 +1,59 @@
+name: Deploy Site
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "platforms/**"
+      - "emulators/**"
+      - "wiki/**"
+      - "scripts/generate_site.py"
+      - "scripts/generate_readme.py"
+      - "scripts/verify.py"
+      - "scripts/common.py"
+      - "database.json"
+      - "mkdocs.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: deploy-site
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+
+      - run: pip install pyyaml mkdocs-material pymdown-extensions
+
+      - name: Generate site
+        run: |
+          python scripts/generate_site.py
+          python scripts/generate_readme.py --db database.json --platforms-dir platforms
+          mkdocs build
+
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v4
+        with:
+          path: site/
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

.gitignore

@@ -11,13 +11,8 @@ site/
 *.log
 node_modules/
 
-# Generated site pages (built in CI)
-docs/index.md
-docs/platforms/
-docs/systems/
-docs/emulators/
-docs/contributing.md
-docs/gaps.md
+# Generated site pages (built by generate_site.py, deployed in CI)
+docs/
 
 # Large files stored as GitHub Release assets (> 50MB)
 bios/Arcade/Arcade/Firmware.19.0.0.zip
@@ -35,3 +30,43 @@ data/
 bios/Arcade/MAME/artwork/snspell.zip
 bios/Arcade/MAME/MAME 0.174 Arcade XML.dat
 bios/Sony/PlayStation Vita/.variants/PSP2UPDAT.PUP
+bios/Nintendo/DS/DSi_Nand_JPN.bin
+bios/Nintendo/DS/DSi_Nand_EUR.bin
+bios/Nintendo/DS/DSi_Nand_USA.bin
+bios/Nintendo/DS/DSi_Nand_AUS.bin
+bios/Nintendo/DS/DSi_Nand_CHN.bin
+bios/Nintendo/DS/DSi_Nand_KOR.bin
+
+# QEMU EDK2 firmware (64MB each)
+bios/QEMU/edk2-aarch64-code.fd
+bios/QEMU/edk2-arm-code.fd
+bios/QEMU/edk2-arm-vars.fd
+
+# ScummVM data bundle (79MB, fetched from large-files release)
+bios/ScummVM/scummvm.zip
+
+# Arcade BIOS ZIPs > 100MB
+bios/Arcade/Arcade/svcpcb.zip
+bios/Arcade/Arcade/ps2.zip
+bios/Arcade/Arcade/kf2k3pcb.zip
+bios/Arcade/Arcade/ms5pcb.zip
+
+# Dolphin/Ishiiruka SD card images (128MB each)
+bios/Nintendo/Wii/WiiSD.raw
+bios/Nintendo/Wii/sd.raw
+
+# EasyRPG soundfont (142MB)
+bios/RPG Maker/easyrpg/easyrpg.soundfont
+
+# Temporary files
+tmp/
+
+# FBA2012 Toaplan samples (>100MB)
+bios/Arcade/fba2012/samples/fireshrk.zip
+bios/Arcade/fba2012/samples/ghox.zip
+bios/Arcade/fba2012/samples/samesame.zip
+bios/Arcade/fba2012/samples/tekipaki.zip
+bios/Arcade/fba2012/samples/vimana.zip
+bios/Arcade/fba2012/samples/fireshrk.zip
+bios/Arcade/fba2012/samples/samesame.zip
+bios/Arcade/fba2012/samples/vimana.zip

CONTRIBUTING.md

@@ -1,14 +1,109 @@
 # Contributing to RetroBIOS
 
-## Add a BIOS file
+## Types of contributions
 
-1. Fork this repository
-2. Place the file in `bios/Manufacturer/Console/filename`
-3. Variants (alternate hashes): `bios/Manufacturer/Console/.variants/`
-4. Create a Pull Request - checksums are verified automatically
+- **Add a BIOS file** - a great way to get started. Fork, add the file, open a PR.
+- **Create an emulator profile** - document what a core actually loads from source code. See the [profiling guide](https://abdess.github.io/retrobios/wiki/profiling/).
+- **Add a platform** - integrate a new frontend (scraper + YAML config). See [adding a platform](https://abdess.github.io/retrobios/wiki/adding-a-platform/).
+- **Add or fix a scraper** - parse upstream sources for BIOS requirements. See [adding a scraper](https://abdess.github.io/retrobios/wiki/adding-a-scraper/).
+- **Fix a bug or improve tooling** - Python scripts in `scripts/`, single dependency (`pyyaml`).
 
-## File conventions
+## Local setup
 
-- Files >50 MB go in GitHub release assets (`large-files` release)
-- RPG Maker and ScummVM directories are excluded from deduplication
-- See the [documentation site](https://abdess.github.io/retrobios/) for full details
+```bash
+git clone https://github.com/Abdess/retrobios.git
+cd retrobios
+pip install pyyaml
+
+# run tests
+python -m unittest tests.test_e2e -v
+
+# run full pipeline (DB + verify + packs + consistency check)
+python scripts/pipeline.py --offline
+```
+
+Requires Python 3.10 or later.
+
+## Adding a BIOS file
+
+1. Place the file in `bios/Manufacturer/Console/filename`.
+2. Alternate versions (different hash, same purpose) go in `bios/Manufacturer/Console/.variants/`.
+3. Files over 50 MB go as assets on the `large-files` GitHub release (git handles them better that way).
+4. RPG Maker and ScummVM directories are excluded from deduplication - please keep their structure as-is.
+5. Open a pull request. CI validates checksums automatically and posts a report.
+
+## Commit conventions
+
+Format: `type: description` (50 characters max, lowercase start).
+
+Allowed types: `feat`, `refactor`, `chore`, `docs`, `fix`.
+
+```
+feat: add panasonic 3do bios files
+docs: update architecture diagram
+fix: resolve truncated md5 matching
+chore: remove unused test fixtures
+refactor: extract hash logic to common.py
+```
+
+Keep messages factual. No marketing language, no superfluous adjectives.
+
+## Code and documentation quality
+
+The codebase runs on Python 3.10+ with a single dependency (`pyyaml`). All modules
+include `from __future__ import annotations` at the top. Type hints on every function
+signature, `pathlib` instead of `os.path`, and dataclasses where a plain class would
+just hold attributes.
+
+On performance: O(1) or O(n) algorithms are preferred. If something needs O(n^2), a
+comment explaining why helps future readers. List comprehensions over explicit loops,
+generators when iterating large datasets, and standard default arguments
+(`def f(items=None)` over `def f(items=[])`).
+
+File I/O uses context managers. ZIP extraction goes through `safe_extract_zip()` in
+`common.py`, which prevents zip-slip path traversal.
+
+The code stays lean. Comments that describe *why* age better than comments that
+restate *what*. Unused variables can be deleted rather than renamed with an underscore.
+
+The same spirit applies to documentation and emulator profiles. Straightforward
+language, honest labels ("untested" when something is untested).
+
+When a bug is spotted while working on something else, fixing it in the same PR
+keeps things tidy. Features ideally ship complete in one pass: code, tests, a
+passing pipeline run, and any documentation updates together.
+
+## Pull request process
+
+CI runs four checks on every PR:
+
+| Check | What it does |
+|-------|-------------|
+| `validate-bios` | hashes changed BIOS files against the database, posts a validation report |
+| `validate-configs` | schema-validates platform YAML configs |
+| `run-tests` | runs the full E2E test suite |
+| `label-pr` | auto-labels the PR based on changed file paths |
+
+All four checks must pass. One maintainer approval is required before merge.
+
+## Project philosophy
+
+This repository is an archive. The goal is a complete collection of every file a
+user needs to run every supported console in an emulator, with zero friction. That
+scope goes beyond BIOS ROMs: firmware updates, system fonts, data files, encryption
+keys, hiscore databases, anything the emulator loads from disk rather than generating
+itself. In a hundred years the pack should still work out of the box.
+
+Licensing considerations are secondary to completeness. If an emulator needs a
+file to function, the file belongs in the collection. The project's legal position is
+documented in the FAQ section of the site.
+
+Integrity matters. Every file, hash, and metadata field is cross-checked
+against the emulator's source code. Upstream references like System.dat,
+`.info` files, and wiki pages are valuable and generally accurate, though
+they can occasionally fall out of date. When an upstream source and the
+code disagree, the code at runtime is the tiebreaker.
+
+## Documentation
+
+Full reference docs, profiling guides, and architecture details are on the [documentation site](https://abdess.github.io/retrobios/).

README.md

@@ -1,41 +1,124 @@
-# Retrogaming BIOS & Firmware Collection
+# RetroBIOS
 
-Complete, verified collection of BIOS, firmware, and system files for retrogaming emulators.
+Complete BIOS and firmware packs for Batocera, BizHawk, EmuDeck, Lakka, Recalbox, RetroArch, RetroBat, RetroDECK, RetroPie, and RomM.
 
-> **6724** files | **5013.3 MB** | **8** platforms | **275** emulator profiles
+**7,241** verified files across **396** systems, ready to extract into your emulator's BIOS directory.
 
-## Download
+## Quick Install
 
-| Platform | Files | Verification | Pack |
-|----------|-------|-------------|------|
-| Batocera | 359 | md5 | [Download](../../releases/latest) |
-| EmuDeck | 161 | md5 | [Download](../../releases/latest) |
-| Lakka | 448 | existence | [Download](../../releases/latest) |
-| Recalbox | 346 | md5 | [Download](../../releases/latest) |
-| RetroArch | 448 | existence | [Download](../../releases/latest) |
-| RetroBat | 331 | md5 | [Download](../../releases/latest) |
-| RetroDECK | 2007 | md5 | [Download](../../releases/latest) |
-| RetroPie | 448 | existence | [Download](../../releases/latest) |
+Copy one command into your terminal:
+
+```bash
+# Linux / macOS / Steam Deck
+curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
+
+# Windows (PowerShell)
+irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex
+
+# Handheld (SD card mounted on PC)
+curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform retroarch --dest /path/to/sdcard
+```
+
+The script auto-detects your platform, downloads only missing files, and verifies checksums.
+
+## Download BIOS packs
+
+Pick your platform, download the ZIP, extract to the BIOS path.
+
+| Platform | BIOS files | Extract to | Download |
+|----------|-----------|-----------|----------|
+| Batocera | 362 | `/userdata/bios/` | [Download](../../releases/latest) |
+| BizHawk | 118 | `Firmware/` | [Download](../../releases/latest) |
+| EmuDeck | 161 | `Emulation/bios/` | [Download](../../releases/latest) |
+| Lakka | 448 | `system/` | [Download](../../releases/latest) |
+| Recalbox | 346 | `/recalbox/share/bios/` | [Download](../../releases/latest) |
+| RetroArch | 448 | `system/` | [Download](../../releases/latest) |
+| RetroBat | 339 | `bios/` | [Download](../../releases/latest) |
+| RetroDECK | 2006 | `~/retrodeck/bios/` | [Download](../../releases/latest) |
+| RetroPie | 448 | `BIOS/` | [Download](../../releases/latest) |
+| RomM | 374 | `bios/{platform_slug}/` | [Download](../../releases/latest) |
+
+## What's included
+
+BIOS, firmware, and system files for consoles from Atari to PlayStation 3.
+Each file is checked against the emulator's source code to match what the code actually loads at runtime.
+
+- **10 platforms** supported with platform-specific verification
+- **329 emulators** profiled from source (RetroArch cores + standalone)
+- **396 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
+- **7,241 files** verified with MD5, SHA1, CRC32 checksums
+- **8144 MB** total collection size
+
+## Supported systems
+
+NES, SNES, Nintendo 64, GameCube, Wii, Game Boy, Game Boy Advance, Nintendo DS, Nintendo 3DS, Switch, PlayStation, PlayStation 2, PlayStation 3, PSP, PS Vita, Mega Drive, Saturn, Dreamcast, Game Gear, Master System, Neo Geo, Atari 2600, Atari 7800, Atari Lynx, Atari ST, MSX, PC Engine, TurboGrafx-16, ColecoVision, Intellivision, Commodore 64, Amiga, ZX Spectrum, Arcade (MAME), and 362+ more.
+
+Full list with per-file details: **[https://abdess.github.io/retrobios/](https://abdess.github.io/retrobios/)**
 
 ## Coverage
 
 | Platform | Coverage | Verified | Untested | Missing |
 |----------|----------|----------|----------|---------|
-| Batocera | 359/359 (100.0%) | 358 | 1 | 0 |
+| Batocera | 356/362 (98.3%) | 349 | 7 | 6 |
+| BizHawk | 118/118 (100.0%) | 118 | 0 | 0 |
 | EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 |
-| Lakka | 448/448 (100.0%) | 448 | 0 | 0 |
-| Recalbox | 346/346 (100.0%) | 346 | 0 | 0 |
-| RetroArch | 448/448 (100.0%) | 448 | 0 | 0 |
-| RetroBat | 331/331 (100.0%) | 331 | 0 | 0 |
-| RetroDECK | 2007/2007 (100.0%) | 1698 | 309 | 0 |
-| RetroPie | 448/448 (100.0%) | 448 | 0 | 0 |
+| Lakka | 442/448 (98.7%) | 442 | 0 | 6 |
+| Recalbox | 277/346 (80.1%) | 274 | 3 | 69 |
+| RetroArch | 442/448 (98.7%) | 442 | 0 | 6 |
+| RetroBat | 339/339 (100.0%) | 335 | 4 | 0 |
+| RetroDECK | 1960/2006 (97.7%) | 1934 | 26 | 46 |
+| RetroPie | 442/448 (98.7%) | 442 | 0 | 6 |
+| RomM | 372/374 (99.5%) | 372 | 0 | 2 |
 
-## Documentation
+## Build your own pack
 
-Full file listings, platform coverage, emulator profiles, and gap analysis: **[https://abdess.github.io/retrobios/](https://abdess.github.io/retrobios/)**
+Clone the repo and generate packs for any platform, emulator, or system:
+
+```bash
+# Full platform pack
+python scripts/generate_pack.py --platform retroarch --output-dir dist/
+python scripts/generate_pack.py --platform batocera --output-dir dist/
+
+# Single emulator or system
+python scripts/generate_pack.py --emulator dolphin
+python scripts/generate_pack.py --system sony-playstation-2
+
+# List available emulators and systems
+python scripts/generate_pack.py --list-emulators
+python scripts/generate_pack.py --list-systems
+
+# Verify your BIOS collection
+python scripts/verify.py --all
+python scripts/verify.py --platform batocera
+python scripts/verify.py --emulator flycast
+python scripts/verify.py --platform retroarch --verbose  # emulator ground truth
+```
+
+Only dependency: Python 3 + `pyyaml`.
+
+## Documentation site
+
+The [documentation site](https://abdess.github.io/retrobios/) provides:
+
+- **Per-platform pages** with file-by-file verification status and hashes
+- **Per-emulator profiles** with source code references for every file
+- **Per-system pages** showing which emulators and platforms cover each console
+- **Gap analysis** identifying missing files and undeclared core requirements
+- **Cross-reference** mapping files across 10 platforms and 329 emulators
+
+## How it works
+
+Documentation and metadata can drift from what emulators actually load.
+To keep packs accurate, each file is checked against the emulator's source code.
+
+1. **Read emulator source code** - trace every file the code loads, its expected hash and size
+2. **Cross-reference with platforms** - match against what each platform declares
+3. **Build packs** - include baseline files plus what each platform's cores need
+4. **Verify** - run platform-native checks and emulator-level validation
 
 ## Contributors
 
+<a href="https://github.com/PixNyb"><img src="https://avatars.githubusercontent.com/u/40770831?v=4" width="50" title="PixNyb"></a>
 <a href="https://github.com/monster-penguin"><img src="https://avatars.githubusercontent.com/u/266009589?v=4" width="50" title="monster-penguin"></a>
 
 
@@ -47,4 +130,4 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 
 This repository provides BIOS files for personal backup and archival purposes.
 
-*Auto-generated on 2026-03-20T19:10:25Z*
+*Auto-generated on 2026-03-30T20:16:27Z*