Commit Graph

62 Commits

Author SHA1 Message Date
Abdessamad Derraz
040ea9f217 fix: resolve_file skips MD5 lookup for zipped_file entries
Same guard as verify.py: when zipped_file is set, the md5 is for the
inner ROM, not the container ZIP. Direct MD5 lookup resolved to the
standalone ROM file instead of the ZIP parent.

Fixes: ep64.zip/ep128.zip (Enterprise) written as raw ROM data instead
of ZIP archives in Batocera pack. Also fixes any other zipped_file entry
where the inner ROM MD5 matched a standalone file in the database.

Also: update Dinothawr.zip SHA1 in retroarch.yml to match actual file.
2026-03-18 06:27:35 +01:00
Abdessamad Derraz
84ab0ea6d3 fix: revert verify dedup (breaks counts), optimize pack generation
verify.py: removed destination dedup - verify counts ALL platform
entries (398 for RetroArch). Pack deduplicates at generation (395).
The delta (3 files: c52/g7400/jopac.bin) is correct behavior.

generate_pack.py: skip build_zip_contents_index() when no zipped_file
entries exist. RetroArch pack: 20s -> 11s (no ZIP scan needed).
2026-03-18 06:27:35 +01:00
Abdessamad Derraz
4faae161b4 feat: implement --include-extras with hybrid core detection
generate_pack.py now merges Tier 2 emulator files into platform packs:
- Auto-detects cores from platform YAML "core:" fields (31 for RetroArch)
- Also reads manual "emulators:" list from _registry.yml (for Batocera etc)
- Union of both sources = complete emulator coverage per platform
- Files already in platform pack are skipped (Tier 1 wins)

Results with --include-extras:
  RetroArch: 395 -> 654 files (+259 emulator extras)
  Batocera:  359 -> 632 files (+273 emulator extras)

Pack naming: BIOS_Pack.zip (normal) vs Complete_Pack.zip (with extras)
2026-03-18 05:39:13 +01:00
Abdessamad Derraz
9052a6b750 feat: add emulator profiles and cross-reference engine (tier 2)
New two-tier architecture:
- Tier 1: Platform configs (what the UI checks) - unchanged
- Tier 2: Emulator profiles (what the code actually loads)

11 emulator profiles from source code analysis:
  cemu, citra, dolphin, duckstation, flycast,
  melonds, pcsx2, ppsspp, rpcs3, vita3k, xemu

Each profile documents every file the emulator loads with
source code references (file:line), hashes, and notes.

New scripts/cross_reference.py computes gaps between what
platforms declare and what emulators need.

Current gap: 200 undeclared files, 24 already in repo.
DuckStation alone recognizes 105 PS1/PS2 BIOS variants.

generate_pack.py gains --include-extras flag (future use).
_registry.yml maps platforms to their emulators.
2026-03-17 20:08:27 +01:00
Abdessamad Derraz
8d81aee235 refactor: quality audit fixes, honest verification reporting
- batocera_scraper: fix OrderedDict parsing for ast.literal_eval
- auto_fetch: fix TypeError when sha1/md5 is None
- verify: filter non-ZIP files for zipped_file entries (F2)
- verify: distinguish ZIP read errors from hash mismatches (F5)
- generate_pack: track seen_destinations with source hash (F7)

Batocera ep64/ep128.zip now correctly reported as MISSING
instead of false UNTESTED (resolved to .rom instead of .zip)
2026-03-17 15:35:30 +01:00
Abdessamad Derraz
5ab82a7898 refactor: security hardening + mame arcade bios updates
Security fixes:
- Zip-slip protection in _extract_zip_to_archive (sanitize paths)
- Hash verification for large file downloads (cache + post-download)
- Sanitize YAML destination fields against path traversal
- Size limit on ZIP entry reads (512MB cap, prevents zip bombs)
- Download size limits in auto_fetch (100MB cap)
- Reject hashless external downloads
- Sanitize filenames in place_file with basename()

MAME arcade updates from Batocera v38 pack:
- Updated naomi, naomi2, naomigd, awbios, airlbios, hod2bios, hikaru
- Old versions preserved in .variants/ for RetroBat compatibility

Batocera 675/680 (+9), all other platforms unchanged at 0 missing
2026-03-17 15:32:14 +01:00
Abdessamad Derraz
af74fffa14 refactor: fix code review findings across all scripts
Critical: stream large file downloads (OOM fix), fix basename match
in auto_fetch, include hashes in pack grouping fingerprint, handle
not_in_zip status in verify, fix escaped quotes in batocera parser.

Important: deduplicate shared group includes, catch coreinfo network
errors, fix NODEDUP path component match, fix CI word splitting on
spaces, replace bare except Exception in 3 files.

Minor: argparse in list_platforms, specific exceptions in download.py.
2026-03-17 15:16:51 +01:00
Abdessamad Derraz
0ffb8cbd0d feat: complete retrobat coverage, fix large file resolution 2026-03-17 13:03:57 +01:00
Abdessamad Derraz
3453f89d9d refactor: consolidate CI pipeline, remove third-party deps 2026-03-17 12:33:10 +01:00
Abdessamad Derraz
851a14e49a add retrobat platform support (scraper, config, verify) 2026-03-17 11:38:52 +01:00
Abdessamad Derraz
c23c565c6d update repo references after rename to retrobios 2026-03-17 11:16:37 +01:00
Abdessamad Derraz
13c561888d v2: automated BIOS platform with full pipeline
Reorganized 6 branches into bios/Manufacturer/Console/.
Scrapers for RetroArch, Batocera, Recalbox, and libretro core-info.
Platform-aware verification replicating native logic per platform.
Pack generation with dedup, alias resolution, variant support.
CI/CD: weekly auto-scrape, auto-release, PR validation.
Large files (>50MB) stored as GitHub Release assets, auto-fetched at build time.
2026-03-17 10:54:39 +01:00