Commit Graph

38 Commits

Author SHA1 Message Date
Abdessamad Derraz
94000bdaef fix: align verify and pack validation, pipeline 100% consistent
generate_pack.py now applies emulator-level validation (crc32, sha1,
adler32) matching verify.py behavior. existence mode: validation is
informational (file present = OK). md5 mode: validation downgrades
to UNTESTED. clone resolution moved to common.py resolve_local_file.
all 6 platforms pass consistency check.
2026-03-24 22:21:47 +01:00
Abdessamad Derraz
ae4846550f fix: clone resolution in common.py, move clone map to root
moved _mame_clones.json out of bios/ (was indexed by generate_db.py
as BIOS file). clone resolution now in common.py resolve_local_file
so all tools (verify, pack, cross_reference) resolve clones
transparently. removed duplicate clone code from generate_pack.py.
added error handling on os.remove in dedup.py. consistency check
now passes for Batocera/EmuDeck/Lakka/RetroArch (4/6 platforms).
2026-03-24 21:57:49 +01:00
Abdessamad Derraz
fb1007496d chore: deduplicate bios/ — remove 427 files, save 227 MB
true duplicates (same file in multiple dirs): removed copies, kept
canonical. MAME device clones (different names, identical content):
removed copies, created _mame_clones.json mapping for pack-time
assembly via deterministic ZIP rebuild. generate_pack.py resolves
clones transparently. 95 canonical ZIPs serve 392 clone names.
2026-03-24 21:35:50 +01:00
Abdessamad Derraz
d4849681a7 feat: add 3DS signature/crypto verification to verify.py
pure python RSA-2048 PKCS1v15 SHA256 for SecureInfo_A,
LocalFriendCodeSeed_B, movable.sed. AES-128-CBC + SHA256 for otp.bin.
keys extracted from azahar default_keys.h, added RSA/ECC sections
to aes_keys.txt. sect233r1 ECC not reproducible (binary field curve).
2026-03-24 11:36:29 +01:00
Abdessamad Derraz
1d350f0578 feat: add emulator/system pack generation, validation checks, path resolution
add --emulator, --system, --standalone, --list-emulators, --list-systems
to verify.py and generate_pack.py. packs are RTU with data directories,
regional BIOS variants, and archive support.

validation: field per file (size, crc32, md5, sha1) with conflict
detection. by_path_suffix index in database.json for regional variant
resolution via dest_hint. restructure GameCube IPL to regional subdirs.

66 E2E tests, full pipeline verified.
2026-03-22 14:02:20 +01:00
Abdessamad Derraz
74f17694c2 feat: add category field to emulator profiles, source missing BIOS
Add category: game_data to sdlpal, nxengine, opentyrian, easyrpg,
mkxp_z profiles. verify.py separates game_data from bios in core
gap metrics for cleaner coverage numbers.

New BIOS files: Cemu fonts (4), QEMU bios-256k + vgabios-stdvga,
GAM4980 ROMs (2), SC-3000 Export variant.
2026-03-21 07:37:22 +01:00
Abdessamad Derraz
27df5c8fb5 fix: resolve case collisions on case-insensitive filesystems
Move 17 case-colliding BIOS variants to .variants/ so Windows
and macOS clones no longer lose files. Existence-based platforms
keep the primary, MD5-based platforms resolve from .variants/.

Also fix resolve_local_file zipped_file resolution: when multiple
ZIPs share a name, verify the inner ROM exists before accepting
a candidate. Fixes adam_fdc.zip resolving to the wrong archive.
2026-03-20 20:02:42 +01:00
Abdessamad Derraz
59ee9088fa chore: add Atari 1450XLD OS Rev 3 (1984-03-27) 2026-03-20 12:52:34 +01:00
Abdessamad Derraz
1c9261d304 chore: add Atari 1450XLD, Arabic 65XE, compiled OS 2026-03-20 10:04:06 +01:00
Abdessamad Derraz
4a76c5ca15 chore: add Atari OS-A NTSC, XEGS, doc format ares 2026-03-20 09:50:59 +01:00
Abdessamad Derraz
7b712ca089 chore: add Game Master, MSX, Atari BIOS files 2026-03-20 07:06:11 +01:00
Abdessamad Derraz
789ff0a32b chore: add MAME and RetroDECK ROM sets 2026-03-20 06:52:57 +01:00
Abdessamad Derraz
6ee162f8fb chore: add MAME and RetroDECK ROM sets 2026-03-19 23:26:49 +01:00
Abdessamad Derraz
fbb2079f9b feat: complete 81 profile, archive 27 standalone ROMs 2026-03-18 17:52:12 +01:00
Abdessamad Derraz
846640dd7c feat: emulator mode field, archive ZX81 standalone ROMs
emulator profiles support mode: standalone | libretro | both.
cross_reference.py skips standalone-only files for libretro platforms.
81.yml: type standalone + libretro, upstream ref added, files listed
with mode: standalone and source_refs to both codebases.
bios/Sinclair/ZX 81/: zx81.rom (8K) and dkchr.rom (4K) archived.
2026-03-18 17:37:01 +01:00
Abdessamad Derraz
0451ff6b67 fix: restore dolphin DSP/font firmware to bios/, integration test pass
dsp_coef.bin, dsp_rom.bin, font_western.bin, font_japanese.bin copied
back to bios/Nintendo/GameCube/ (needed by verify.py, were moved to
data/ with the Sys folder). Pack includes 2537 files with Dolphin Sys.
2026-03-18 16:13:00 +01:00
Abdessamad Derraz
e07d85fc63 refactor: move dolphin Sys to data/, reduce DB by 2500 entries 2026-03-18 15:59:02 +01:00
Abdessamad Derraz
bb307aa250 feat: archive full dolphin-emu/Sys, add DSP/font/IPL paths to pack
dolphin-emu/Sys/ folder (2562 files) from libretro/dolphin Data/Sys.
retroarch.yml: DSP firmware (dsp_coef.bin, dsp_rom.bin), fonts
(font_western.bin, font_japanese.bin) at dolphin-emu/Sys/GC/ paths.
ref: DolphinLibretro/Boot.cpp:72-73, HW/DSPLLE/DSPHost.cpp,
HW/EXI/EXI_DeviceIPL.cpp. pack: 452 files, 0 missing.
2026-03-18 15:16:20 +01:00
Abdessamad Derraz
d920231c50 feat: add bioscv.rom, GameIndex.yaml, enrich mu.yml with MD5s 2026-03-18 14:22:19 +01:00
Abdessamad Derraz
c71378a71c feat: shared groups, scraper subdir prefixes, arcade + emulator profiles
shared groups in _shared.yml: np2kai, keropi, quasi88, kronos, ep128emu
with source references for each subdirectory requirement.

libretro_scraper: CORE_SUBDIR_MAP applies subdirectory prefixes at
generation time (np2kai/, keropi/, dc/). EXTRA_SYSTEMS adds QUASI88.
SYSTEM_SHARED_GROUPS injects includes for kronos/np2kai/keropi.

new BIOS: CPS3 (23 ZIPs), Cannonball OutRun (40 ROMs), PCem PC BIOS
(73 files), VICE Commodore ROMs, Spectrum ZIPs, dc_bios.bin, X1 fonts.

new emulator profiles: redream, melonds_ds, lrps2 with doc vs source
notes. platforms/README.md documents shared groups architecture.
2026-03-18 13:49:59 +01:00
Abdessamad Derraz
7653d5d108 feat: add 19 BIOS files, fix cross_reference resolution
New files: OpenTyrian data (11), Cave Story (2), SeaBIOS,
VGA BIOS, OpenSBI, Cromwell, xbox_hdd, Sega CD Model 2 (3),
NGP Color BIOS, Pentagon 128p-1.rom, X1 font, BK TERAK.
cross_reference.py: basename + case-insensitive lookup.
2026-03-18 12:50:55 +01:00
Abdessamad Derraz
dd9e59c8e3 chore: remove nested bios/Philips/CD-i/bios/ duplicate + empty TOS ghosts
Nested bios/ directory inside CD-i was an agent artifact.
tos100uk.img and tos206us.img were 0-byte ghost files from failed git show.
2026-03-18 08:25:56 +01:00
Abdessamad Derraz
0c367ca7c6 feat: restore deleted TOS UK images, regenerate database
tos102uk.img, tos104uk.img, tos106uk.img were accidentally deleted
by a background agent. Restored from git history.
2026-03-18 07:47:25 +01:00
Abdessamad Derraz
e82a498124 feat: update sc3000 to basic iii v1.1, regenerate db and readme
Replace sc3000.rom with BASIC Level III v1.1 (Japan) from MAME 0.270+
software list (SHA1 737a06b3, closer to Batocera target).

Batocera 679/680, Recalbox 346/346, all others 100%.
2026-03-17 17:51:11 +01:00
Abdessamad Derraz
1257653c9b feat: batocera 679/680, fix variant indexing, add hikaru + segaboot
Fix variant name indexing: files in .variants/ now indexed under
canonical name (naomi2.zip instead of naomi2.zip.da79eca4).
Fix .zip detection for variant paths in verify.py.
Add composite MD5 matching in resolver for ZIP variants.

Add hikaru.zip (MAME 0.285, 6 ROMs) and segaboot.gcm (Triforce)
from archive.org. Both match Batocera expected MD5s.

Batocera 679/680 (1 untested: sc3000 private dump)
Recalbox 346/346 (100%)
2026-03-17 17:06:02 +01:00
Abdessamad Derraz
36a17b315a feat: batocera 678/680, add 6 naomi2 roms from mame 0.278
Add missing NAOMI 2 ROMs (315-6146, 315-6215, sp5001/a/b, sp5002-a)
from MAME 0.278 non-merged set. Clean up naomi2 variant files.

Recalbox composite naomi2.zip kept as variant (.da79eca4).
Batocera 678/680 (2 untested: sc3000, triforce)
Recalbox 345/346 (1 untested: naomi2 composite mismatch)
2026-03-17 16:19:21 +01:00
Abdessamad Derraz
bb1855d3f7 feat: recalbox 346/346 via md5_composite, add mame variants
Add md5_composite() to verify.py replicating Recalbox Zip::Md5Composite
(sorted filenames, sequential content hash). Independent of ZIP
compression level, resolves all 9 MAME arcade untested entries.

Add Recalbox-specific MAME ZIP variants from Recalbox 10 pack.
Batocera 671/680 (9 untested MAME-specific), all others 100%.
2026-03-17 16:08:39 +01:00
Abdessamad Derraz
4bf62b8126 feat: fix enterprise zips, restore deleted files, add sms aliases
- Create ep64.zip/ep128.zip MAME sets from existing ROMs
- Restore Complex_4627v1.03.bin, N64DD IPL, PS2 SCPH-70004 files
  (accidentally deleted by background agent)
- Add Xbox Complex_4627.bin alias for Batocera
- Add SMS BIOS aliases for RetroBat naming convention

All platforms at 0 missing:
  Batocera 677/680, Recalbox 337/346, all others 100%
2026-03-17 15:44:12 +01:00
Abdessamad Derraz
5ab82a7898 refactor: security hardening + mame arcade bios updates
Security fixes:
- Zip-slip protection in _extract_zip_to_archive (sanitize paths)
- Hash verification for large file downloads (cache + post-download)
- Sanitize YAML destination fields against path traversal
- Size limit on ZIP entry reads (512MB cap, prevents zip bombs)
- Download size limits in auto_fetch (100MB cap)
- Reject hashless external downloads
- Sanitize filenames in place_file with basename()

MAME arcade updates from Batocera v38 pack:
- Updated naomi, naomi2, naomigd, awbios, airlbios, hod2bios, hikaru
- Old versions preserved in .variants/ for RetroBat compatibility

Batocera 675/680 (+9), all other platforms unchanged at 0 missing
2026-03-17 15:32:14 +01:00
Abdessamad Derraz
ff38eebde7 feat: fix retro computer bios, retrobat ecwolf/freej2me variants
Add missing ROMs to Dragon 32/64, TRS-80, TRS-80 M3/M4 ZIPs.
Add RetroBat-specific ecwolf.pk3 and freej2me-lr.jar variants.

Batocera 666/680 (+5), RetroBat 343/343 (100%), EmuDeck 164/164
2026-03-17 15:13:32 +01:00
Abdessamad Derraz
7ea649da7c feat: emudeck 164/164, complete coverage all platforms
Add SCPH-1000R (PlayStation Classic JP) romJP.bin from archive.org
NAND dump. Last missing file across all 7 platforms.

All platforms at 0 missing:
  RetroArch  398/398, Lakka 398/398, RetroPie 398/398
  RetroBat   341/343 (2 untested)
  Recalbox   336/346 (10 untested)
  Batocera   589/680 (91 untested, MAME version diffs)
  EmuDeck    164/164
2026-03-17 14:44:10 +01:00
Abdessamad Derraz
936cec07cd feat: emudeck at 163/164, add mega cd + ps1_rom variants
Add 3 rare BIOS variants from TOSEC/archive.org:
- Mega CD v1.00s JP (a3ddcc8483) from Myrient TOSEC
- Sega CD Model 2 v2.00 US (310a9081d2) from Myrient TOSEC
- ps1_rom.bin 4MB PS3-extracted (fbb5f59ec3) from archive.org

EmuDeck: 163/164 (99.4%), 1 unknown hash remains (476d68a9)
All other platforms: 0 missing
2026-03-17 14:37:58 +01:00
Abdessamad Derraz
617f5a3654 feat: expand bios collection, emudeck at 97.6%
Add 36 new files from multiple sources:
- PSX rare variants (DTL-H2000, SCPH-5903 VCD, SCPH-7000/7000W, DTL-H1100)
- Sega Mega CD 9 region variants (EU/JP/US firmware revisions)
- Saturn HiSaturn GameNavi v1.03
- N64DD IPL (US/JP/DEV), PS2 SCPH-70004, Xbox Complex BIOS
- PPSSPP system pack, 3DS/Switch/Wii U keys, Nokia N-Gage
- Apple Macintosh mac755.chd

Platform coverage:
  RetroArch  398/398 (100%)
  Lakka      398/398 (100%)
  RetroPie   398/398 (100%)
  Batocera   589/680 (86.6%, 91 untested MAME versions)
  Recalbox   336/346 (97.1%, 10 untested)
  RetroBat   341/343 (99.4%, 2 untested)
  EmuDeck    160/164 (97.6%, 4 missing rare dumps)
2026-03-17 14:15:33 +01:00
Abdessamad Derraz
0ffb8cbd0d feat: complete retrobat coverage, fix large file resolution 2026-03-17 13:03:57 +01:00
Abdessamad Derraz
4d98f17308 feat: add dolphin dsp, mame hash xmls, scummvm fonts from retrobat-bios 2026-03-17 12:42:55 +01:00
Abdessamad Derraz
3453f89d9d refactor: consolidate CI pipeline, remove third-party deps 2026-03-17 12:33:10 +01:00
Abdessamad Derraz
e6ea0484a8 expand bios collection, retrobat at 93% coverage 2026-03-17 11:53:42 +01:00
Abdessamad Derraz
13c561888d v2: automated BIOS platform with full pipeline
Reorganized 6 branches into bios/Manufacturer/Console/.
Scrapers for RetroArch, Batocera, Recalbox, and libretro core-info.
Platform-aware verification replicating native logic per platform.
Pack generation with dedup, alias resolution, variant support.
CI/CD: weekly auto-scrape, auto-release, PR validation.
Large files (>50MB) stored as GitHub Release assets, auto-fetched at build time.
2026-03-17 10:54:39 +01:00