Commit Graph

123 Commits

Author SHA1 Message Date
Abdessamad Derraz
71b127efb5 feat: 4-source verified emulator profiles (A-E cores)
33 emulator profiles created or updated with systematic
verification against libretro docs, .info, core source,
and original emulator source code.

New profiles: bsnes-jg, bsnes2014, bsnes_cplusplus98,
bsnes_hd_beta, bsnes_mercury, citra2018, citra_canary.

Key fixes:
- dosbox_svn/svn_ce: not aliases of dosbox_core (no MUNT)
- beetle_psx: alt_names renamed to aliases (field bug)
- dolphin: added 15 Realtek BT firmware for Wiimote passthrough
- dosbox_core: added CM-32LN, 13 split ROM pairs for standalone
- duckstation: type standalone+libretro, 106 BIOS verified
- snes9x/np2kai: formal aliases for alternate filenames
2026-03-19 08:09:34 +01:00
Abdessamad Derraz
86dbdf28e5 feat: core profiles, data_dirs buildbot, cross_ref fix
profiles: amiberry (new), amiarcadia, atari800, azahar, b2,
bk, blastem, bluemsx, freeintv updated with source refs,
upstream field, mode field, data_directories.

_data_dirs.yml: buildbot source for retroarch platforms,
strip_components for nested ZIPs, freeintv-overlays fixed.

cross_reference.py: data_directories-aware gap analysis,
suppresses false gaps when emulator+platform share refs.

refresh_data_dirs.py: ZIP strip_components support,
for_platforms filter, ETag freshness for buildbot.

scraper: bluemsx single ref, freeintv overlays injection.
generate_pack.py: warning on missing data directory cache.
2026-03-18 21:20:02 +01:00
Abdessamad Derraz
fbb2079f9b feat: complete 81 profile, archive 27 standalone ROMs 2026-03-18 17:52:12 +01:00
Abdessamad Derraz
846640dd7c feat: emulator mode field, archive ZX81 standalone ROMs
emulator profiles support mode: standalone | libretro | both.
cross_reference.py skips standalone-only files for libretro platforms.
81.yml: type standalone + libretro, upstream ref added, files listed
with mode: standalone and source_refs to both codebases.
bios/Sinclair/ZX 81/: zx81.rom (8K) and dkchr.rom (4K) archived.
2026-03-18 17:37:01 +01:00
Abdessamad Derraz
b3b279ccf3 docs: document data directories in README 2026-03-18 16:14:25 +01:00
Abdessamad Derraz
0451ff6b67 fix: restore dolphin DSP/font firmware to bios/, integration test pass
dsp_coef.bin, dsp_rom.bin, font_western.bin, font_japanese.bin copied
back to bios/Nintendo/GameCube/ (needed by verify.py, were moved to
data/ with the Sys folder). Pack includes 2537 files with Dolphin Sys.
2026-03-18 16:13:00 +01:00
Abdessamad Derraz
58b2398053 chore: CI refreshes data directories, data/ in gitignore 2026-03-18 16:07:49 +01:00
Abdessamad Derraz
9b537492c0 feat: scraper injects data_directories refs into retroarch.yml 2026-03-18 16:06:56 +01:00
Abdessamad Derraz
c9e2bf8d33 feat: generate_pack.py integrates data directory refresh and packing 2026-03-18 16:04:36 +01:00
Abdessamad Derraz
976e5fbd41 feat: add refresh_data_dirs.py for upstream data sync 2026-03-18 16:01:28 +01:00
Abdessamad Derraz
e07d85fc63 refactor: move dolphin Sys to data/, reduce DB by 2500 entries 2026-03-18 15:59:02 +01:00
Abdessamad Derraz
68b1c575ae feat: add data directory registry 2026-03-18 15:56:50 +01:00
github-actions[bot]
6c38a2aa6a regenerate database and docs 2026-03-18 14:19:45 +00:00
Abdessamad Derraz
bb307aa250 feat: archive full dolphin-emu/Sys, add DSP/font/IPL paths to pack
dolphin-emu/Sys/ folder (2562 files) from libretro/dolphin Data/Sys.
retroarch.yml: DSP firmware (dsp_coef.bin, dsp_rom.bin), fonts
(font_western.bin, font_japanese.bin) at dolphin-emu/Sys/GC/ paths.
ref: DolphinLibretro/Boot.cpp:72-73, HW/DSPLLE/DSPHost.cpp,
HW/EXI/EXI_DeviceIPL.cpp. pack: 452 files, 0 missing.
2026-03-18 15:16:20 +01:00
Abdessamad Derraz
e5681c4ae8 feat: dolphin IPL.bin paths, scraper dedup by (name, destination)
dolphin: gc-ntsc-12.bin mapped to dolphin-emu/Sys/GC/<region>/IPL.bin
ref: DolphinLibretro/Boot.cpp:72-73, CommonPaths.h:139
scraper EXTRA_SYSTEM_FILES dedup now by (name, destination) to allow
same source file at multiple destinations.
retroarch pack: 448 files, 0 missing.
2026-03-18 15:08:26 +01:00
Abdessamad Derraz
4bffc23ab5 feat: 0 HIGH issues, xrick system, np2kai FONT.ROM, coleco.rom alias
verified against source: fuse flat (not fuse/), ep128emu/roms/ (not rom/).
added xrick system, np2kai FONT.ROM uppercase variant, coleco.rom alias.
quasi88 alt naming verified in quasi88-libretro/src/LIBRETRO/libretro.c:108-117.
61 systems, 445 files, 0 missing on all platforms.
2026-03-18 15:01:52 +01:00
Abdessamad Derraz
71e506e708 fix: ep128emu roms/ path (not rom/), fuse flat verified in source
ep128emu: corrected to ep128emu/roms/ per core.cpp:56,59.
fuse: verified in src/compat/paths.c — core searches flat, not fuse/.
docs are wrong on fuse/ prefix. removed from retroarch shared groups.
all refs updated to exact source lines.
added quasi88 alt names, vircon32, MacII.ROM casing.
2026-03-18 14:55:13 +01:00
Abdessamad Derraz
c09abe0179 feat: complete medium audit fixes, add vircon32, quasi88 alt names
scraper: add MacII.ROM casing fix, Vircon32 system, QUASI88 alt
naming convention (n88_0/1/2/3.rom alongside N88EXT*.ROM).
retroarch pack: 445 -> 450 files, 60 systems, 0 missing.
all choices traced to original emulator source code.
2026-03-18 14:50:07 +01:00
Abdessamad Derraz
1d6a0b9ebc feat: complete retroarch.yml conformance with libretro docs
scraper adds FBNeo subsystem BIOS (channelf, coleco, neocdz, ngp,
spectrum, spec128, spec1282a, fdsbios, aes), DSi files, SGB alt
names, JollyCV BIOS, Saturn ST-V, PSX alt BIOS. all traced to
original emulator source code. 0 missing across all platforms.
retroarch pack: 398 -> 445 files.
2026-03-18 14:46:32 +01:00
Abdessamad Derraz
2afc31e40a feat: scraper-generated retroarch.yml with shared group conformance
retroarch.yml regenerated by libretro_scraper with CORE_SUBDIR_MAP
(dc/, np2kai/, keropi/) and shared groups (fuse, kronos, ep128emu,
quasi88, np2kai, keropi). common.py dedup by (name, destination)
to allow same file at flat + subdirectory paths.

ep128emu shared group added for Enterprise system.
RetroArch pack grows from 398 to 428 files.

ref: each subdirectory traced to original emulator source code —
see platforms/README.md and _shared.yml comments.
2026-03-18 14:41:00 +01:00
Abdessamad Derraz
3802237209 feat: add fuse shared group, scraper injects fuse/ prefix for ZX Spectrum 2026-03-18 14:29:25 +01:00
Abdessamad Derraz
5dda62861e fix: redream.yml files format, cross_reference compat 2026-03-18 14:23:42 +01:00
Abdessamad Derraz
d920231c50 feat: add bioscv.rom, GameIndex.yaml, enrich mu.yml with MD5s 2026-03-18 14:22:19 +01:00
Abdessamad Derraz
513fd04bfd feat: scraper adds missing arcade BIOS, ep128emu includes, new profiles
libretro_scraper: EXTRA_ARCADE_FILES adds namcoc69/70/75, msx, qsound
to arcade system. segasp.zip added to dreamcast-arcade. ep128emu
includes injected for enterprise system.

new profiles: vba_m (GB/GBC/GBA with doc vs source notes),
beetle_gba (Mednafen GBA).
2026-03-18 14:18:56 +01:00
Abdessamad Derraz
c71378a71c feat: shared groups, scraper subdir prefixes, arcade + emulator profiles
shared groups in _shared.yml: np2kai, keropi, quasi88, kronos, ep128emu
with source references for each subdirectory requirement.

libretro_scraper: CORE_SUBDIR_MAP applies subdirectory prefixes at
generation time (np2kai/, keropi/, dc/). EXTRA_SYSTEMS adds QUASI88.
SYSTEM_SHARED_GROUPS injects includes for kronos/np2kai/keropi.

new BIOS: CPS3 (23 ZIPs), Cannonball OutRun (40 ROMs), PCem PC BIOS
(73 files), VICE Commodore ROMs, Spectrum ZIPs, dc_bios.bin, X1 fonts.

new emulator profiles: redream, melonds_ds, lrps2 with doc vs source
notes. platforms/README.md documents shared groups architecture.
2026-03-18 13:49:59 +01:00
Abdessamad Derraz
7653d5d108 feat: add 19 BIOS files, fix cross_reference resolution
New files: OpenTyrian data (11), Cave Story (2), SeaBIOS,
VGA BIOS, OpenSBI, Cromwell, xbox_hdd, Sega CD Model 2 (3),
NGP Color BIOS, Pentagon 128p-1.rom, X1 font, BK TERAK.
cross_reference.py: basename + case-insensitive lookup.
2026-03-18 12:50:55 +01:00
Abdessamad Derraz
76064605c0 fix: move zip_contents resolution after name-based lookup 2026-03-18 12:12:42 +01:00
Abdessamad Derraz
08f68e792d refactor: centralize hash logic, fix circular imports and perf bottlenecks 2026-03-18 11:51:12 +01:00
Abdessamad Derraz
becd0efb33 fix: relative links in readme, commit pending changes 2026-03-18 11:28:58 +01:00
Abdessamad Derraz
09ebaa9316 fix: retropie logo from github avatar, remove *.png gitignore 2026-03-18 11:26:46 +01:00
Abdessamad Derraz
81278bd2e4 fix: system icons (systematic theme), retropie logo 2026-03-18 11:25:14 +01:00
Abdessamad Derraz
a52ab19cf8 fix: full hashes, list format for system files 2026-03-18 11:15:11 +01:00
Abdessamad Derraz
300e5d7439 fix: redesign home page UX, fix broken retropie logo 2026-03-18 11:09:36 +01:00
Abdessamad Derraz
54c0f1d27e refactor: review fixes, DRY coverage, filter test nav
- Extract compute_coverage to common.py (was duplicated)
- Filter test cores from nav and emulator index
- Use absolute URL for README download links
- Consistent page titles with site name suffix
- Safer mkdocs.yml nav rewrite with regex
- Build all_platform_names once in gap analysis
2026-03-18 11:05:13 +01:00
Abdessamad Derraz
e218763500 feat: add emulator logos to profiles and site 2026-03-18 10:57:00 +01:00
Abdessamad Derraz
6885681c65 feat: add platform logos to registry and site 2026-03-18 10:55:47 +01:00
Abdessamad Derraz
21a50c992f feat: slim readme + ci site deployment
README: 11141 -> 43 lines. Details on the MkDocs site.
generate_readme.py: 444 -> 164 lines. Slim coverage table only.
build.yml: adds mkdocs-material install, generate_site.py, gh-deploy.
Adds pages: write permission for GitHub Pages deployment.
2026-03-18 10:44:13 +01:00
Abdessamad Derraz
32e4f6e580 fix: review fixes for generate_site.py 2026-03-18 10:39:23 +01:00
Abdessamad Derraz
0b1ed3cb1a feat: add gap analysis page + platform tracking 2026-03-18 10:31:02 +01:00
Abdessamad Derraz
883e153a62 fix: clean platform/emulator page layout 2026-03-18 10:27:08 +01:00
Abdessamad Derraz
b15b062782 feat: add mkdocs site generator, 332 pages
generate_site.py reads database.json + platforms/*.yml + emulators/*.yml
and produces a complete MkDocs Material documentation site:
- Home: stats, downloads, coverage dashboard
- 7 platform pages with per-file verification status
- 60 system pages grouped by manufacturer with cross-references
- 260 emulator pages with source code analysis
- Contributing guide

mkdocs.yml with Material theme, system fonts, auto dark mode.
Generated docs/ in .gitignore (built in CI only).
2026-03-18 10:22:00 +01:00
Abdessamad Derraz
dd9e59c8e3 chore: remove nested bios/Philips/CD-i/bios/ duplicate + empty TOS ghosts
Nested bios/ directory inside CD-i was an agent artifact.
tos100uk.img and tos206us.img were 0-byte ghost files from failed git show.
2026-03-18 08:25:56 +01:00
Abdessamad Derraz
3de4bf8190 refactor: extract _fetch_raw to BaseScraper (DRY)
Identical _fetch_raw() implementation (URL fetch + cache + error handling)
was duplicated in 4 scrapers. Moved to BaseScraper.__init__ with url param.

Each scraper now passes url to super().__init__() and inherits _fetch_raw().
Eliminates ~48 lines of duplicated code.

DRY audit now clean: resolve logic in common.py, scraper CLI in base_scraper,
_fetch_raw in BaseScraper. Remaining duplications are justified (different
list_platforms semantics, context-specific hash computation).
2026-03-18 08:22:21 +01:00
Abdessamad Derraz
2466fc4a97 refactor: extract scraper_cli() to base_scraper.py (DRY)
Shared CLI boilerplate for all scrapers: argparse, dry-run, json, yaml output.
4 scrapers (libretro, batocera, retrobat, emudeck) reduced from ~58 lines
main() each to 3 lines calling scraper_cli().

~220 lines of duplicated boilerplate eliminated.
recalbox + coreinfo keep custom main() (extra flags: --full, --compare-db).
2026-03-18 08:17:14 +01:00
Abdessamad Derraz
00700609d8 refactor: extract resolve_local_file to common.py (DRY)
Single source of truth for file resolution logic:
- common.py:resolve_local_file() = 80 lines (core resolution)
- verify.py:resolve_to_local_path() = 3 lines (thin wrapper)
- generate_pack.py:resolve_file() = 20 lines (adds storage tiers + release assets)

Before: 103 + 73 = 176 lines of duplicated logic with subtle divergences
After: 80 lines shared + 23 lines wrappers = 103 lines total (-41%)

Resolution chain: SHA1 -> MD5 multi-hash -> truncated MD5 ->
zipped_file index -> name existence -> name composite -> name fallback
-> (pack only) release assets
2026-03-18 08:11:10 +01:00
Abdessamad Derraz
0c367ca7c6 feat: restore deleted TOS UK images, regenerate database
tos102uk.img, tos104uk.img, tos106uk.img were accidentally deleted
by a background agent. Restored from git history.
2026-03-18 07:47:25 +01:00
Abdessamad Derraz
8c18638cd2 regenerate database after merge 2026-03-18 07:31:46 +01:00
Abdessamad Derraz
7b1c6a723e refactor: review fixes - resolve coherence + cleanup
1. fetch_large_file moved to last resort (avoids HTTP before name lookup)
2. fetch_large_file receives first MD5 only (not comma-separated string)
3. verify.py MD5 lookup now splits comma-separated + lowercases (matches generate_pack)
4. seen_destinations simplified to set (stored hash was dead data)
5. Variable suffix shadowing renamed to file_ext
2026-03-18 07:18:40 +01:00
Abdessamad Derraz
7ae995fb32 fix: resolve_file multi-MD5 + md5_composite for Recalbox packs
Three fixes in resolve_file():
- Split comma-separated MD5 lists (Recalbox uses multi-hash)
- Add md5_composite check in name fallback (matches verify.py logic)
- Use ".zip" in basename instead of endswith for variant files

Recalbox pack: 346/346 verified (was 332/346 with 14 wrong hash)
Batocera pack: 359/359 verified (was 304/359 with 55 inner missing)
All 5 platforms now produce 0 untested, 0 missing packs.
2026-03-18 07:18:40 +01:00
Abdessamad Derraz
a1dc6fa4ef fix: resolve_file prefers primary over variants for name fallback
When resolving by name with no MD5 (existence check), prefer files
NOT in .variants/ directory. Fixes naomi2.zip resolving to the
Recalbox variant (15 files) instead of the primary (21 files).

Also applies to hash_mismatch fallback path.
2026-03-18 07:18:40 +01:00