regenerate database and docs

Batocera upstream has a truncated 29-char MD5 for zx48.rom.
The scraper now resolves truncated hashes via prefix match
against database.json, preventing schema validation failures.

.github/workflows/build.yml

@@ -13,6 +13,7 @@ on:
 
 permissions:
   contents: write
+  pages: write
 
 concurrency:
   group: build
@@ -30,10 +31,35 @@ jobs:
 
       - run: pip install pyyaml
 
-      - name: Regenerate database and docs
+      - name: Restore large files from release
         run: |
-          python scripts/generate_db.py --bios-dir bios --output database.json
+          mkdir -p .cache/large
+          gh release download large-files -D .cache/large/ 2>/dev/null || true
+          # Copy large files to their bios/ paths so generate_db sees them
+          for f in .cache/large/*; do
+            [ -f "$f" ] || continue
+            name=$(basename "$f")
+            # Match against .gitignore entries to find target path
+            target=$(grep "$name" .gitignore | head -1)
+            if [ -n "$target" ] && [ ! -f "$target" ]; then
+              mkdir -p "$(dirname "$target")"
+              cp "$f" "$target"
+              echo "Restored: $target"
+            fi
+          done
+        env:
+          GH_TOKEN: ${{ github.token }}
+
+      - name: Regenerate database, readme, and site
+        run: |
+          pip install mkdocs-material
+          python scripts/generate_db.py --force --bios-dir bios --output database.json
           python scripts/generate_readme.py --db database.json --platforms-dir platforms
+          python scripts/generate_site.py
+
+      - name: Deploy site to GitHub Pages
+        if: github.ref == 'refs/heads/main'
+        run: mkdocs gh-deploy --force --clean
 
       - name: Commit if changed
         id: commit
@@ -79,6 +105,11 @@ jobs:
         with:
           python-version: "3.12"
 
+      - name: Refresh data directories
+        if: steps.rate.outputs.skip != 'true'
+        run: python scripts/refresh_data_dirs.py
+        continue-on-error: true
+
       - name: Build packs
         if: steps.rate.outputs.skip != 'true'
         run: |

.github/workflows/watch.yml

@@ -60,6 +60,9 @@ jobs:
           python scripts/generate_db.py --bios-dir bios --output database.json
           python scripts/auto_fetch.py --all 2>&1 | tee /tmp/fetch_report.txt || true
 
+      - name: Refresh data directories
+        run: python scripts/refresh_data_dirs.py --force
+
       - name: Deduplicate BIOS files
         run: python scripts/dedup.py
 

.gitignore

@@ -6,10 +6,19 @@ __pycache__/
 *.pyc
 .cache/
 dist/
+site/
 *.tmp
 *.log
 node_modules/
 
+# Generated site pages (built in CI)
+docs/index.md
+docs/platforms/
+docs/systems/
+docs/emulators/
+docs/contributing.md
+docs/gaps.md
+
 # Large files stored as GitHub Release assets (> 50MB)
 bios/Arcade/Arcade/Firmware.19.0.0.zip
 bios/Arcade/Arcade/maclc3.zip
@@ -19,6 +28,9 @@ bios/Sony/PlayStation 3/PS3UPDAT.PUP
 bios/Sony/PlayStation Vita/PSVUPDAT.PUP
 bios/Sony/PlayStation Vita/PSP2UPDAT.PUP
 
+# Data directories (fetched at build time)
+data/
+
 # Large files stored as GitHub Release assets (additional)
 bios/Arcade/MAME/artwork/snspell.zip
 bios/Arcade/MAME/MAME 0.174 Arcade XML.dat

CONTRIBUTING.md

@@ -1,51 +1,14 @@
-# Contributing BIOS Files
+# Contributing to RetroBIOS
 
-Thank you for helping expand the BIOS collection!
+## Add a BIOS file
 
-## How to Contribute
+1. Fork this repository
+2. Place the file in `bios/Manufacturer/Console/filename`
+3. Variants (alternate hashes): `bios/Manufacturer/Console/.variants/`
+4. Create a Pull Request - checksums are verified automatically
 
-1. **Fork** this repository
-2. **Add** your BIOS file to the correct directory under `bios/Manufacturer/Console/`
-3. **Create a Pull Request**
+## File conventions
 
-## File Placement
-
-Place files in the correct manufacturer/console directory:
-```
-bios/
-├── Sony/
-│   └── PlayStation/
-│       └── scph5501.bin
-├── Nintendo/
-│   └── Game Boy Advance/
-│       └── gba_bios.bin
-└── Sega/
-    └── Dreamcast/
-        └── dc_boot.bin
-```
-
-## Verification
-
-All submitted BIOS files are automatically verified against known checksums:
-
-1. **Hash verification** - SHA1/MD5 checked against known databases
-2. **Size verification** - File size matches expected value
-3. **Platform reference** - File must be referenced in at least one platform config
-4. **Duplicate detection** - Existing files are flagged to avoid duplication
-
-## What We Accept
-
-- **Verified BIOS dumps** with matching checksums from known databases
-- **System firmware** required by emulators
-- **New variants** of existing BIOS files (different regions, versions)
-
-## What We Don't Accept
-
-- Game ROMs or ISOs
-- Modified/patched BIOS files
-- Files without verifiable checksums
-- Executable files (.exe, .bat, .sh)
-
-## Questions?
-
-Open an [Issue](../../issues) if you're unsure about a file.
+- Files >50 MB go in GitHub release assets (`large-files` release)
+- RPG Maker and ScummVM directories are excluded from deduplication
+- See the [documentation site](https://abdess.github.io/retrobios/) for full details