fix: sha1-based large file restore, fix broken data dir urls

Replace grep-based restore with SHA1 matching via database.json. The old grep heuristic failed for assets with renamed basenames (dsi_nand_batocera42.bin) or special characters (MAME dots vs spaces), and only restored to the first .gitignore match when multiple paths shared a basename. Fix 3 broken data directory sources: - opentyrian: buildbot URL 404, use release asset - syobonaction: invalid git_subtree URL, use GitHub archive - stonesoup: same fix, adds 532 game data files
feat: restore large files and data dirs in site deploy
2026-04-18 23:02:33 -05:00 · 2026-04-02 18:46:44 +02:00 · 2026-04-02 18:25:02 +02:00 · 2026-04-02 17:55:23 +02:00 · 2026-04-02 17:13:59 +02:00 · 2026-04-02 16:26:12 +02:00
2911 changed files with 2323951 additions and 88660 deletions
@@ -58,16 +58,32 @@ jobs:
        run: |
          mkdir -p .cache/large
          gh release download large-files -D .cache/large/ 2>/dev/null || true
-          for f in .cache/large/*; do
-            [ -f "$f" ] || continue
-            name=$(basename "$f")
-            target=$(grep "$name" .gitignore | head -1)
-            if [ -n "$target" ] && [ ! -f "$target" ]; then
-              mkdir -p "$(dirname "$target")"
-              cp "$f" "$target"
-              echo "Restored: $target"
-            fi
-          done
+          python3 -c "
+          import hashlib, json, os, shutil
+          db = json.load(open('database.json'))
+          with open('.gitignore') as f:
+              ignored = {l.strip() for l in f if l.strip().startswith('bios/')}
+          cache = '.cache/large'
+          if not os.path.isdir(cache):
+              exit(0)
+          idx = {}
+          for fn in os.listdir(cache):
+              fp = os.path.join(cache, fn)
+              if os.path.isfile(fp):
+                  h = hashlib.sha1(open(fp, 'rb').read()).hexdigest()
+                  idx[h] = fp
+          restored = 0
+          for sha1, entry in db['files'].items():
+              path = entry['path']
+              if path in ignored and not os.path.exists(path):
+                  src = idx.get(sha1)
+                  if src:
+                      os.makedirs(os.path.dirname(path), exist_ok=True)
+                      shutil.copy2(src, path)
+                      print(f'Restored: {path}')
+                      restored += 1
+          print(f'Total: {restored} files restored')
+          "
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

@@ -36,6 +36,43 @@ jobs:

      - run: pip install pyyaml mkdocs-material pymdown-extensions

+      - name: Restore large files from release
+        run: |
+          mkdir -p .cache/large
+          gh release download large-files -D .cache/large/ 2>/dev/null || true
+          python3 -c "
+          import hashlib, json, os, shutil
+          db = json.load(open('database.json'))
+          with open('.gitignore') as f:
+              ignored = {l.strip() for l in f if l.strip().startswith('bios/')}
+          cache = '.cache/large'
+          if not os.path.isdir(cache):
+              exit(0)
+          idx = {}
+          for fn in os.listdir(cache):
+              fp = os.path.join(cache, fn)
+              if os.path.isfile(fp):
+                  h = hashlib.sha1(open(fp, 'rb').read()).hexdigest()
+                  idx[h] = fp
+          restored = 0
+          for sha1, entry in db['files'].items():
+              path = entry['path']
+              if path in ignored and not os.path.exists(path):
+                  src = idx.get(sha1)
+                  if src:
+                      os.makedirs(os.path.dirname(path), exist_ok=True)
+                      shutil.copy2(src, path)
+                      print(f'Restored: {path}')
+                      restored += 1
+          print(f'Total: {restored} files restored')
+          "
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Refresh data directories
+        run: python scripts/refresh_data_dirs.py
+        continue-on-error: true
+
      - name: Generate site
        run: |
          python scripts/generate_site.py
@@ -29,4 +29,45 @@ data/
 # Large files stored as GitHub Release assets (additional)
 bios/Arcade/MAME/artwork/snspell.zip
 bios/Arcade/MAME/MAME 0.174 Arcade XML.dat
-bios/Sony/PlayStation Vita/.variants/PSP2UPDAT.PUP
+bios/Sony/PlayStation Vita/.variants/PSP2UPDAT.PUP.3ae832c9
+bios/Nintendo/DS/DSi_Nand_JPN.bin
+bios/Nintendo/DS/DSi_Nand_EUR.bin
+bios/Nintendo/DS/DSi_Nand_USA.bin
+bios/Nintendo/DS/DSi_Nand_AUS.bin
+bios/Nintendo/DS/DSi_Nand_CHN.bin
+bios/Nintendo/DS/DSi_Nand_KOR.bin
+bios/Nintendo/DS/dsi_nand.bin
+
+# QEMU EDK2 firmware (64MB each)
+bios/QEMU/edk2-aarch64-code.fd
+bios/QEMU/edk2-arm-code.fd
+bios/QEMU/edk2-arm-vars.fd
+
+# ScummVM data bundle (79MB, fetched from large-files release)
+bios/ScummVM/scummvm.zip
+
+# Arcade BIOS ZIPs > 100MB
+bios/Arcade/Arcade/svcpcb.zip
+bios/Arcade/Arcade/ps2.zip
+bios/Arcade/Arcade/kf2k3pcb.zip
+bios/Arcade/Arcade/ms5pcb.zip
+
+# Dolphin/Ishiiruka SD card images (128MB each)
+bios/Nintendo/Wii/WiiSD.raw
+bios/Nintendo/Wii/sd.raw
+
+# EasyRPG soundfont (142MB)
+bios/RPG Maker/easyrpg/easyrpg.soundfont
+
+# Temporary files
+tmp/
+
+# FBA2012 Toaplan samples (>100MB)
+bios/Arcade/fba2012/samples/fireshrk.zip
+bios/Arcade/fba2012/samples/ghox.zip
+bios/Arcade/fba2012/samples/samesame.zip
+bios/Arcade/fba2012/samples/tekipaki.zip
+bios/Arcade/fba2012/samples/vimana.zip
+bios/Arcade/fba2012/samples/fireshrk.zip
+bios/Arcade/fba2012/samples/samesame.zip
+bios/Arcade/fba2012/samples/vimana.zip
@@ -7,6 +7,16 @@
 3. Variants (alternate hashes): `bios/Manufacturer/Console/.variants/`
 4. Create a Pull Request - checksums are verified automatically

+## Add a new platform
+
+1. Write a scraper in `scripts/scraper/`
+2. Create the platform YAML in `platforms/`
+3. Register in `platforms/_registry.yml`
+4. Submit a Pull Request
+
+Contributors who add platform support are credited in the README,
+on the documentation site, and in the BIOS packs.
+
 ## File conventions

 - Files >50 MB go in GitHub release assets (`large-files` release)
@@ -1,8 +1,25 @@
 # RetroBIOS

-Complete BIOS and firmware packs for Batocera, EmuDeck, Lakka, Recalbox, RetroArch, RetroBat, RetroDECK, RetroPie, and RomM.
+Complete BIOS and firmware packs for Batocera, BizHawk, EmuDeck, Lakka, Recalbox, RetroArch, RetroBat, RetroDECK, RetroPie, and RomM.

-**6,748** verified files across **352** systems, ready to extract into your emulator's BIOS directory.
+**7,302** verified files across **396** systems, ready to extract into your emulator's BIOS directory.
+
+## Quick Install
+
+Copy one command into your terminal:
+
+```bash
+# Linux / macOS / Steam Deck
+curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh
+
+# Windows (PowerShell)
+irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex
+
+# Handheld (SD card mounted on PC)
+curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform retroarch --dest /path/to/sdcard
+```
+
+The script auto-detects your platform, downloads only missing files, and verifies checksums.

 ## Download BIOS packs

@@ -10,13 +27,14 @@ Pick your platform, download the ZIP, extract to the BIOS path.

 | Platform | BIOS files | Extract to | Download |
 |----------|-----------|-----------|----------|
-| Batocera | 359 | `/userdata/bios/` | [Download](../../releases/latest) |
+| Batocera | 361 | `/userdata/bios/` | [Download](../../releases/latest) |
+| BizHawk | 118 | `Firmware/` | [Download](../../releases/latest) |
 | EmuDeck | 161 | `Emulation/bios/` | [Download](../../releases/latest) |
 | Lakka | 448 | `system/` | [Download](../../releases/latest) |
 | Recalbox | 346 | `/recalbox/share/bios/` | [Download](../../releases/latest) |
 | RetroArch | 448 | `system/` | [Download](../../releases/latest) |
-| RetroBat | 331 | `bios/` | [Download](../../releases/latest) |
-| RetroDECK | 2007 | `~/retrodeck/bios/` | [Download](../../releases/latest) |
+| RetroBat | 339 | `bios/` | [Download](../../releases/latest) |
+| RetroDECK | 2006 | `~/retrodeck/bios/` | [Download](../../releases/latest) |
 | RetroPie | 448 | `BIOS/` | [Download](../../releases/latest) |
 | RomM | 374 | `bios/{platform_slug}/` | [Download](../../releases/latest) |

@@ -25,15 +43,15 @@ Pick your platform, download the ZIP, extract to the BIOS path.
 BIOS, firmware, and system files for consoles from Atari to PlayStation 3.
 Each file is checked against the emulator's source code to match what the code actually loads at runtime.

- **9 platforms** supported with platform-specific verification
- **328 emulators** profiled from source (RetroArch cores + standalone)
- **352 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
- **6,748 files** verified with MD5, SHA1, CRC32 checksums
- **5251 MB** total collection size
+- **10 platforms** supported with platform-specific verification
+- **329 emulators** profiled from source (RetroArch cores + standalone)
+- **396 systems** covered (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
+- **7,302 files** verified with MD5, SHA1, CRC32 checksums
+- **8765 MB** total collection size

 ## Supported systems

-NES, SNES, Nintendo 64, GameCube, Wii, Game Boy, Game Boy Advance, Nintendo DS, Nintendo 3DS, Switch, PlayStation, PlayStation 2, PlayStation 3, PSP, PS Vita, Mega Drive, Saturn, Dreamcast, Game Gear, Master System, Neo Geo, Atari 2600, Atari 7800, Atari Lynx, Atari ST, MSX, PC Engine, TurboGrafx-16, ColecoVision, Intellivision, Commodore 64, Amiga, ZX Spectrum, Arcade (MAME), and 318+ more.
+NES, SNES, Nintendo 64, GameCube, Wii, Game Boy, Game Boy Advance, Nintendo DS, Nintendo 3DS, Switch, PlayStation, PlayStation 2, PlayStation 3, PSP, PS Vita, Mega Drive, Saturn, Dreamcast, Game Gear, Master System, Neo Geo, Atari 2600, Atari 7800, Atari Lynx, Atari ST, MSX, PC Engine, TurboGrafx-16, ColecoVision, Intellivision, Commodore 64, Amiga, ZX Spectrum, Arcade (MAME), and 362+ more.

 Full list with per-file details: **[https://abdess.github.io/retrobios/](https://abdess.github.io/retrobios/)**

@@ -41,15 +59,16 @@ Full list with per-file details: **[https://abdess.github.io/retrobios/](https:/

 | Platform | Coverage | Verified | Untested | Missing |
 |----------|----------|----------|----------|---------|
-| Batocera | 359/359 (100.0%) | 358 | 1 | 0 |
+| Batocera | 361/361 (100.0%) | 361 | 0 | 0 |
+| BizHawk | 118/118 (100.0%) | 118 | 0 | 0 |
 | EmuDeck | 161/161 (100.0%) | 161 | 0 | 0 |
 | Lakka | 448/448 (100.0%) | 448 | 0 | 0 |
 | Recalbox | 346/346 (100.0%) | 346 | 0 | 0 |
 | RetroArch | 448/448 (100.0%) | 448 | 0 | 0 |
-| RetroBat | 331/331 (100.0%) | 331 | 0 | 0 |
-| RetroDECK | 2007/2007 (100.0%) | 2007 | 0 | 0 |
+| RetroBat | 339/339 (100.0%) | 339 | 0 | 0 |
+| RetroDECK | 2006/2006 (100.0%) | 2006 | 0 | 0 |
 | RetroPie | 448/448 (100.0%) | 448 | 0 | 0 |
-| RomM | 374/374 (100.0%) | 359 | 15 | 0 |
+| RomM | 374/374 (100.0%) | 374 | 0 | 0 |

 ## Build your own pack

@@ -72,6 +91,7 @@ python scripts/generate_pack.py --list-systems
 python scripts/verify.py --all
 python scripts/verify.py --platform batocera
 python scripts/verify.py --emulator flycast
+python scripts/verify.py --platform retroarch --verbose  # emulator ground truth
 ```

 Only dependency: Python 3 + `pyyaml`.
@@ -84,7 +104,7 @@ The [documentation site](https://abdess.github.io/retrobios/) provides:
 - **Per-emulator profiles** with source code references for every file
 - **Per-system pages** showing which emulators and platforms cover each console
 - **Gap analysis** identifying missing files and undeclared core requirements
- **Cross-reference** mapping files across 9 platforms and 328 emulators
+- **Cross-reference** mapping files across 10 platforms and 329 emulators

 ## How it works

@@ -110,4 +130,4 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

 This repository provides BIOS files for personal backup and archival purposes.

-*Auto-generated on 2026-03-26T12:17:35Z*
+*Auto-generated on 2026-04-02T13:38:26Z*
@@ -0,0 +1,65 @@
+ИИ�Ш
+
+
+
+�Щ�Щ�Ы�Ы�ШШ
+
+
+
+�Щ�Щ�Ы�ЫИ=ИИ
+
+
+
+�й�й�л�л�н�и
+
+
+
+�й�й�л�лИИИИ
+
+
+
+ЈшЈшЈшЈшЈшЈш
+
+
+
+ЈшЈшЈшЈшЙ§Иј
+
+
+
+ИјИјИјИјЙ§Pј
+
+
+
+ИјИјИјИјИИHИ
+
+
+
+H(H(H(H(H(H(
+
+
+
+H(H(H(H(ИЙИИ
+
+
+
+X8X8X8X8	ЩX8
+
+
+
+X8X8X8X8ИИИИ
+
+
+
+hhhhhh
+
+
+
+hhhh�Нxp
+
+
+
+xxxx
+-xp
+
+
+
@@ -0,0 +1,65 @@
+Ш
+
+
+
+99;;8(
+
+
+
+99;;-Ш8H
+
+
+
+(H(H(H(H-H8H
+
+
+
+(H(H(H(HШШШШ
+
+
+
+XxXxXxXxXxXx
+
+
+
+XxXxXxXxШШШШ
+
+
+
+hh€hh€h€h€
+
+
+
+hh€hh€ШНШШ
+
+
+
+�№�№�»�»�Ѕ�ё
+
+
+
+�№�№�»�»ШЩШШ
+
+
+
+ЁИЁИЁИЁИ)YЁИ
+
+
+
+ЁИЁИЁИЁИЩэШш
+
+
+
+ШшШшШшШшЩэ ш
+
+
+
+ШшШшШшШшШЭиа
+
+
+
+и€ии€иMиа
+
+
+
+и€ии€и
--- a/Show More
+++ b/Show More