Commit Graph

21 Commits

Author SHA1 Message Date
Abdessamad Derraz
257ec1a527 fix: round 2 audit fixes, updated emulator profiles
Scripts:
- fix generate_site nav regex destroying mkdocs.yml content
- fix auto_fetch comma-separated MD5 in find_missing
- fix verify print_platform_result conflating untested/missing
- fix validate_pr path traversal and symlink check
- fix batocera_scraper brace counting and escaped quotes in strings
- fix emudeck_scraper hash search crossing function boundaries
- fix pipeline.py cwd to repo root via Path(__file__)
- normalize SHA1 comparison to lowercase in generate_pack

Emulator profiles:
- emux_gb/nes/sms: reclassify from alias to standalone profiles
- ep128emu: remove .info-only files not referenced in source
- fbalpha2012 variants: full source-verified profiles
- fbneo_cps12: add new profile
2026-03-19 15:00:18 +01:00
Abdessamad Derraz
86dbdf28e5 feat: core profiles, data_dirs buildbot, cross_ref fix
profiles: amiberry (new), amiarcadia, atari800, azahar, b2,
bk, blastem, bluemsx, freeintv updated with source refs,
upstream field, mode field, data_directories.

_data_dirs.yml: buildbot source for retroarch platforms,
strip_components for nested ZIPs, freeintv-overlays fixed.

cross_reference.py: data_directories-aware gap analysis,
suppresses false gaps when emulator+platform share refs.

refresh_data_dirs.py: ZIP strip_components support,
for_platforms filter, ETag freshness for buildbot.

scraper: bluemsx single ref, freeintv overlays injection.
generate_pack.py: warning on missing data directory cache.
2026-03-18 21:20:02 +01:00
Abdessamad Derraz
9b537492c0 feat: scraper injects data_directories refs into retroarch.yml 2026-03-18 16:06:56 +01:00
Abdessamad Derraz
bb307aa250 feat: archive full dolphin-emu/Sys, add DSP/font/IPL paths to pack
dolphin-emu/Sys/ folder (2562 files) from libretro/dolphin Data/Sys.
retroarch.yml: DSP firmware (dsp_coef.bin, dsp_rom.bin), fonts
(font_western.bin, font_japanese.bin) at dolphin-emu/Sys/GC/ paths.
ref: DolphinLibretro/Boot.cpp:72-73, HW/DSPLLE/DSPHost.cpp,
HW/EXI/EXI_DeviceIPL.cpp. pack: 452 files, 0 missing.
2026-03-18 15:16:20 +01:00
Abdessamad Derraz
e5681c4ae8 feat: dolphin IPL.bin paths, scraper dedup by (name, destination)
dolphin: gc-ntsc-12.bin mapped to dolphin-emu/Sys/GC/<region>/IPL.bin
ref: DolphinLibretro/Boot.cpp:72-73, CommonPaths.h:139
scraper EXTRA_SYSTEM_FILES dedup now by (name, destination) to allow
same source file at multiple destinations.
retroarch pack: 448 files, 0 missing.
2026-03-18 15:08:26 +01:00
Abdessamad Derraz
4bffc23ab5 feat: 0 HIGH issues, xrick system, np2kai FONT.ROM, coleco.rom alias
verified against source: fuse flat (not fuse/), ep128emu/roms/ (not rom/).
added xrick system, np2kai FONT.ROM uppercase variant, coleco.rom alias.
quasi88 alt naming verified in quasi88-libretro/src/LIBRETRO/libretro.c:108-117.
61 systems, 445 files, 0 missing on all platforms.
2026-03-18 15:01:52 +01:00
Abdessamad Derraz
71e506e708 fix: ep128emu roms/ path (not rom/), fuse flat verified in source
ep128emu: corrected to ep128emu/roms/ per core.cpp:56,59.
fuse: verified in src/compat/paths.c — core searches flat, not fuse/.
docs are wrong on fuse/ prefix. removed from retroarch shared groups.
all refs updated to exact source lines.
added quasi88 alt names, vircon32, MacII.ROM casing.
2026-03-18 14:55:13 +01:00
Abdessamad Derraz
c09abe0179 feat: complete medium audit fixes, add vircon32, quasi88 alt names
scraper: add MacII.ROM casing fix, Vircon32 system, QUASI88 alt
naming convention (n88_0/1/2/3.rom alongside N88EXT*.ROM).
retroarch pack: 445 -> 450 files, 60 systems, 0 missing.
all choices traced to original emulator source code.
2026-03-18 14:50:07 +01:00
Abdessamad Derraz
1d6a0b9ebc feat: complete retroarch.yml conformance with libretro docs
scraper adds FBNeo subsystem BIOS (channelf, coleco, neocdz, ngp,
spectrum, spec128, spec1282a, fdsbios, aes), DSi files, SGB alt
names, JollyCV BIOS, Saturn ST-V, PSX alt BIOS. all traced to
original emulator source code. 0 missing across all platforms.
retroarch pack: 398 -> 445 files.
2026-03-18 14:46:32 +01:00
Abdessamad Derraz
3802237209 feat: add fuse shared group, scraper injects fuse/ prefix for ZX Spectrum 2026-03-18 14:29:25 +01:00
Abdessamad Derraz
513fd04bfd feat: scraper adds missing arcade BIOS, ep128emu includes, new profiles
libretro_scraper: EXTRA_ARCADE_FILES adds namcoc69/70/75, msx, qsound
to arcade system. segasp.zip added to dreamcast-arcade. ep128emu
includes injected for enterprise system.

new profiles: vba_m (GB/GBC/GBA with doc vs source notes),
beetle_gba (Mednafen GBA).
2026-03-18 14:18:56 +01:00
Abdessamad Derraz
c71378a71c feat: shared groups, scraper subdir prefixes, arcade + emulator profiles
shared groups in _shared.yml: np2kai, keropi, quasi88, kronos, ep128emu
with source references for each subdirectory requirement.

libretro_scraper: CORE_SUBDIR_MAP applies subdirectory prefixes at
generation time (np2kai/, keropi/, dc/). EXTRA_SYSTEMS adds QUASI88.
SYSTEM_SHARED_GROUPS injects includes for kronos/np2kai/keropi.

new BIOS: CPS3 (23 ZIPs), Cannonball OutRun (40 ROMs), PCem PC BIOS
(73 files), VICE Commodore ROMs, Spectrum ZIPs, dc_bios.bin, X1 fonts.

new emulator profiles: redream, melonds_ds, lrps2 with doc vs source
notes. platforms/README.md documents shared groups architecture.
2026-03-18 13:49:59 +01:00
Abdessamad Derraz
08f68e792d refactor: centralize hash logic, fix circular imports and perf bottlenecks 2026-03-18 11:51:12 +01:00
Abdessamad Derraz
3de4bf8190 refactor: extract _fetch_raw to BaseScraper (DRY)
Identical _fetch_raw() implementation (URL fetch + cache + error handling)
was duplicated in 4 scrapers. Moved to BaseScraper.__init__ with url param.

Each scraper now passes url to super().__init__() and inherits _fetch_raw().
Eliminates ~48 lines of duplicated code.

DRY audit now clean: resolve logic in common.py, scraper CLI in base_scraper,
_fetch_raw in BaseScraper. Remaining duplications are justified (different
list_platforms semantics, context-specific hash computation).
2026-03-18 08:22:21 +01:00
Abdessamad Derraz
2466fc4a97 refactor: extract scraper_cli() to base_scraper.py (DRY)
Shared CLI boilerplate for all scrapers: argparse, dry-run, json, yaml output.
4 scrapers (libretro, batocera, retrobat, emudeck) reduced from ~58 lines
main() each to 3 lines calling scraper_cli().

~220 lines of duplicated boilerplate eliminated.
recalbox + coreinfo keep custom main() (extra flags: --full, --compare-db).
2026-03-18 08:17:14 +01:00
Abdessamad Derraz
8d81aee235 refactor: quality audit fixes, honest verification reporting
- batocera_scraper: fix OrderedDict parsing for ast.literal_eval
- auto_fetch: fix TypeError when sha1/md5 is None
- verify: filter non-ZIP files for zipped_file entries (F2)
- verify: distinguish ZIP read errors from hash mismatches (F5)
- generate_pack: track seen_destinations with source hash (F7)

Batocera ep64/ep128.zip now correctly reported as MISSING
instead of false UNTESTED (resolved to .rom instead of .zip)
2026-03-17 15:35:30 +01:00
Abdessamad Derraz
af74fffa14 refactor: fix code review findings across all scripts
Critical: stream large file downloads (OOM fix), fix basename match
in auto_fetch, include hashes in pack grouping fingerprint, handle
not_in_zip status in verify, fix escaped quotes in batocera parser.

Important: deduplicate shared group includes, catch coreinfo network
errors, fix NODEDUP path component match, fix CI word splitting on
spaces, replace bare except Exception in 3 files.

Minor: argparse in list_platforms, specific exceptions in download.py.
2026-03-17 15:16:51 +01:00
Abdessamad Derraz
29d475b8b7 feat: add emudeck platform support, 126/164 verified 2026-03-17 13:33:07 +01:00
Abdessamad Derraz
3453f89d9d refactor: consolidate CI pipeline, remove third-party deps 2026-03-17 12:33:10 +01:00
Abdessamad Derraz
851a14e49a add retrobat platform support (scraper, config, verify) 2026-03-17 11:38:52 +01:00
Abdessamad Derraz
13c561888d v2: automated BIOS platform with full pipeline
Reorganized 6 branches into bios/Manufacturer/Console/.
Scrapers for RetroArch, Batocera, Recalbox, and libretro core-info.
Platform-aware verification replicating native logic per platform.
Pack generation with dedup, alias resolution, variant support.
CI/CD: weekly auto-scrape, auto-release, PR validation.
Large files (>50MB) stored as GitHub Release assets, auto-fetched at build time.
2026-03-17 10:54:39 +01:00