96 Commits

Author SHA1 Message Date
Abdessamad Derraz
73ccb216f5 feat: align gap analysis coherence, add 7 BIOS files, unsourceable field
cross_reference.py: add bios_mode/agnostic/load_from filters, archive
grouping, unsourceable field support. verify.py: case-insensitive
by_name lookup, storage:release in_repo, unsourceable skip, trailing
slash fix. generate_site.py: enriched all_declared, platform-relevant
profile filtering, proper in_repo resolution on emulator pages,
acknowledged gaps section.

New BIOS: delta2.rom (XRoar), tilekey.dat + sprites.sif (NXEngine),
Gram Kracker.ctg + cf7+.ctg + ti-pcard.ctg (ti99sim), desc.dat
(SDLPAL). Profiles: hle_fallback on tilekey.dat/key.txt, unsourceable
on 7 files with source-verified reasons.
2026-04-02 15:35:24 +02:00
Abdessamad Derraz
9e184f76fc fix: swanstation ps1_rom.bin accepts multiple sizes
SwanStation accepts PS1 (512KB), PS2 (4MB), and PS3 (0x3E66F0)
BIOS sizes but only uses the first 512KB. MD5 validates the
extracted content, not the full file. List all accepted sizes
to eliminate the false size mismatch discrepancy.

validation.py: support size as list in emulator profiles.
generate_site.py: handle list sizes in emulator page display.

All 18 original hash mismatches are now resolved: 0 discrepancies.
2026-04-02 07:39:30 +02:00
Abdessamad Derraz
0401d058a1 feat: add by_sha256 index, fix reporting attribution
generate_db: add by_sha256 index for O(1) variant lookup.
verify: _find_best_variant uses indexed sha256 instead of O(n) scan.
validation: check_file_validation returns (reason, emulators) tuple,
attributing mismatch only to emulators whose check actually failed.
beetle_psx: remove incorrect size field for ps1_rom.bin (code does
not validate size, swanstation is sole size authority).
2026-04-02 00:59:01 +02:00
Abdessamad Derraz
2326306f2b fix: adler32 byteswap for dolphin dsp validation
Dolphin computes adler32 on byte-swapped (16-bit) data, not raw
file bytes. Add adler32_byteswap flag to dolphin/primehack/ishiiruka
profiles and support it in validation.py.

Reduces hash mismatch discrepancies from 18 to 2.
2026-04-01 22:51:39 +02:00
Abdessamad Derraz
b56f8dd05f feat: add archive_prefix for core-specific BIOS subdirectories
Closes #43

FBNeo and Kronos expect BIOS archives in core-specific subdirectories
(system/fbneo/, system/kronos/). RetroArch firmware check uses .info
paths which include these prefixes, so files at root show as Missing.

Add archive_prefix field to emulator profiles. The pack code now places
archive copies in the prefixed subdirectory while keeping root copies
for cores that expect them there (e.g. Geolith for neogeo.zip).
2026-03-31 09:17:54 +02:00
Abdessamad Derraz
038c3d3b40 feat: enrich MAME/FBNeo profiles with upstream BIOS contents
auto-fetched from mamedev/mame 0.287 and finalburnneo/FBNeo v1.0.0.2.
mame: +20 new BIOS root sets, 96 entries enriched with contents.
mamearcade: 47 entries enriched with contents.
mamemess: 20 entries enriched with contents.
fbneo: +13 new ROM entries from upstream BIOS sets.
2026-03-30 21:39:02 +02:00
Abdessamad Derraz
d415777f2c feat: add PS3UPDAT.PUP to rpcs3 profile
storage: large_file, validated via SCEUF magic + HMAC-SHA1
per entry (PUP.cpp:23-77). also adds missing standard fields
(cores, core_classification, upstream), removes non-standard
firmware_file/validation/firmware_version fields.
2026-03-30 15:06:51 +02:00
Abdessamad Derraz
17777f315b feat: agnostic bios mode for filename-agnostic emulators
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
2026-03-30 14:18:54 +02:00
Abdessamad Derraz
692484d32d refactor: remove false aliases from pcsx2 and melonds profiles
aliases must be same-SHA1 alternative names, not distinct files.
pcsx2: 164 different BIOS dumps are separate DB entries, not aliases.
melonds: 6 regional NAND dumps are separate DB entries, not aliases.
also cleans pcsx2 non-standard fields, fixes display_name.
2026-03-30 12:37:32 +02:00
Abdessamad Derraz
a8430940f9 feat: add regional nand aliases to melonds profile
dsi_nand.bin aliases: DSi_Nand_USA/EUR/JPN/AUS/CHN/KOR.bin
for repo resolution. Code loads a single configurable path
(libretro.cpp:836, EmuInstance.cpp:1036-1050), validates
nocash footer (DSi_NAND.cpp:42-111). size + storage added.
2026-03-30 12:20:26 +02:00
Abdessamad Derraz
2712307420 feat: add tandy vis bios root set to mame profiles
vis.zip (p513bk0b.bin + p513bk1b.bin) from src/mame/trs/vis.cpp.
Driver added in MAME 0.180, not present in older cores.
2026-03-30 09:09:58 +02:00
Abdessamad Derraz
d3bd68f6ad feat: add vsmile, tutor, vg5k, vgmplay profiles and system aliases 2026-03-30 06:14:58 +02:00
Abdessamad Derraz
8f29e0e85c Update DB timestamp; add ti99sim emulator
Bump database.json generated_at timestamp and add a new emulator metadata file emulators/ti99sim.yml. The new YAML registers the TI-99/Sim standalone emulator (ti99sim) with system info, core version, notes on ROM handling, and multiple required/optional ROM entries (with SHA1s, sizes and validation notes) for TI-99/4A support.
2026-03-29 23:33:12 +02:00
Abdessamad Derraz
83ccf17b11 feat: add ti99 and msx BIOS entries, add fs-5500 disk rom 2026-03-29 23:20:44 +02:00
Abdessamad Derraz
56bff1d013 feat: add ti99 system to mame profile 2026-03-29 23:16:33 +02:00
Abdessamad Derraz
3824193a11 feat: add socrates and vc4000 systems to mame profiles 2026-03-29 23:15:25 +02:00
Abdessamad Derraz
0c5cde83e1 Add TRS-80, RX-78, Sega AI entries; refactor tools
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
2026-03-29 23:04:30 +02:00
Abdessamad Derraz
f3de3ead20 Add PV-2000 & Supracan BIOS; simplify Vita3K
Add Casio PV-2000 BIOS entry (pv2000.zip) to MAME and MESS profiles and update system lists/counts. Add Funtech Super A'Can BIOS entries (supracan.zip and umc6650.zip) with ROM contents to mamemess. Simplify and condense Vita3K emulator profile (rename fields, update profiled_date, add PSVUPDAT.PUP and optional PSP2UPDAT.PUP file entries, and clarify install/partition behavior). Bump database generated_at timestamp and add a system alias mapping "psvita" -> "sony-playstation-vita" in scripts/common.py.
2026-03-29 22:51:30 +02:00
Abdessamad Derraz
463fca7e7d Regenerate database and update emulator profiles
Regenerate database.json and update README counts/timestamps; add and normalize numerous BIOS entries and hashes. Key changes: update generated_at timestamp and system count (355→357) in README; add OpenBIOS / HLE fallback and additional aliases to beetle_psx, include beetle_psx core name and profiled_date update; add laseractive to ares systems; adjust atari800 systems and source_ref line numbers; mark dinothawr as a system and expand its note; update gsplus upstream/profile date, add apple-iie system and tweak source_refs; add pcsx2 core to lrps2; refresh mame profiled_date and add multiple systems and BIOS root sets. Miscellaneous script changes and other JSON normalization to reflect newly discovered/merged BIOS files.
2026-03-29 22:41:01 +02:00
Abdessamad Derraz
95e16c9e7a Add Adam/Advision/APF systems and update metadata
Add support for Coleco Adam, Entex Adventure Vision and APF M-1000 BIOS/ROM sets in MAME and MESS metadata (multiple Adam device MCU ROMs and optional FDC/SPI variants, Advision and APF BIOS entries). Update generated metadata across the repo: README coverage numbers and per-platform coverage rows, database generated timestamp and totals (total_files 7245), and various install manifests (notably batocera.json) with new timestamps, adjusted file counts/sizes, SHA1s, repo_path fixes and an added adam_fdc_320kb.zip entry. Also update notes to reflect the new system ROM sets in the emulators entries.
2026-03-29 16:07:15 +02:00
Abdessamad Derraz
97eb4835be feat: add load_from field for non-system_dir files
Replaces mode: standalone hack with load_from: save_dir on Panda3DS
files. The load_from field documents which libretro directory callback
provides the base path (system_dir default, save_dir, content_dir).
Pack generator and cross-reference skip files not targeting system_dir.
2026-03-29 13:07:30 +02:00
Abdessamad Derraz
d38f2b7180 fix: exclude save_dir paths from pack multi-dest 2026-03-29 12:02:30 +02:00
Abdessamad Derraz
e13e8734b9 chore: standardize notes: to note: in file entries
240 file-level entries used notes: instead of the canonical note:
field. verify.py and cross_reference.py only read note:, so these
were silently ignored.
2026-03-29 10:02:20 +02:00
Abdessamad Derraz
b7528a71e7 fix: correct type and path fields across 58 emulator profiles
49 libretro cores had type: game/utility/test instead of type: libretro,
breaking the all_libretro filtering in resolve_platform_cores and
excluding them from platform packs (e.g. cannonball missing from
RetroArch). core_classification already carries the descriptive role.

9 profiles with subdirectory-loading cores (cannonball/, nxengine/,
Citra/sysdata/, mame2003/, mame2003-plus/, mame2010/) now have path:
fields so cross-reference places files at the correct destination.

resolve_local_file now tries basename when name contains a path
separator (e.g. res/tilemap.bin -> tilemap.bin), fixing resolution
of files with subdirectory names.
2026-03-29 09:53:46 +02:00
Abdessamad Derraz
c241a44c33 fix: fbneo data dir paths (hiscore, samples) 2026-03-29 06:50:36 +02:00
Abdessamad Derraz
1381a17f9c feat: complete fbneo samples (49 small + 3 large) 2026-03-29 01:24:25 +01:00
Abdessamad Derraz
a37bf11524 fix: add path fields for correct pack subdirectory placement
Cores that load files from system_dir subdirectories (same_cdi/bios/,
neocd/, cannonball/, Citra/sysdata/, mame2003/, etc.) need path: on
each file entry so cross-reference and pack generation place files at
the correct destination. Also fixes neocd.yml using non-standard dest:
field instead of path:.

Closes #41
2026-03-29 00:59:57 +01:00
Abdessamad Derraz
b01a138f1c feat: add fbneo samples data directory 2026-03-29 00:53:27 +01:00
Abdessamad Derraz
c835336999 fix: correct OpenTyrian shapes profiling error
Remove 30 phantom shapes[2-v].dat entries. shapeFile[34] in
lvlmast.c maps to newsh?.shp (enemy sprites), NOT shapes?.dat
(level tilesets). Only 5 shapes exist: ), W, X, Y, Z — the
characters actually referenced by level data. Verified against
trapexit/libretro-opentyrian source. Cross-ref: 54 -> 24.
2026-03-29 00:30:06 +01:00
Abdessamad Derraz
329deb82fd feat: add fbneo cheats data directory 2026-03-29 00:21:53 +01:00
Abdessamad Derraz
23dd7657aa fix: restore all emulator profiles to ground truth
Previous commit failed to fully revert fuse.yml (agent-added
MD5/SHA1 still present). Also revert qemu.yml and higan_sfc.yml.
All profiles now match their pre-collection state (2666ebd).
Emulator profiles are source-verified ground truth — agents
must never modify them without code verification.
2026-03-28 17:39:31 +01:00
Abdessamad Derraz
fea0338062 fix: revert agent modifications to fuse.yml and qemu.yml
Agents added MD5/SHA1 hashes to fuse.yml computed from downloaded
files, not from source code (ground truth violation: validation
is [size] only). Agents added firmware entries to qemu.yml without
source code verification. Profiles must reflect code, not files.
2026-03-28 17:36:39 +01:00
Abdessamad Derraz
3d5d5b601c feat: mass BIOS collection (+1020 files, 7520 total)
Add ~1500 BIOS, firmware and game data files sourced from
24 parallel agents. Migrate game engine data (ScummVM,
SDLPAL, OpenTyrian, Cave Story, Syobon Action) from bios/
to data/ via _data_dirs.yml. Add Roland MT-32/CM-32L ROMs
to bios/Roland/. Upload large files (EDK2, SDLPAL, ScummVM,
arcade ZIPs, EasyRPG soundfont) to large-files release.

New coverage: QEMU firmware (67), PCem (137), VICE (53),
Cemu Wii U (58), shadPS4 (14), Amiga (24), MAME ZIPs (724),
O2EM voice (270), Azahar 3DS (4), EightyOne ZX81 (35),
Stella KidVid (14), XRoar (12), and 77 emulators total.
2026-03-28 16:51:43 +01:00
Abdessamad Derraz
c8cba9d519 fix: add beetle_supergrafx, caprice32, mess2016 aliases 2026-03-27 12:00:27 +01:00
Abdessamad Derraz
6b87c75e11 fix: add upstream buildbot aliases to 18 emulator profiles 2026-03-27 11:57:10 +01:00
Abdessamad Derraz
21fa98c82f fix: add missing buildbot aliases to avoid false target skips 2026-03-27 11:43:29 +01:00
Abdessamad Derraz
1bde69f173 fix: add mednafen_wswan alias to beetle_wswan profile 2026-03-27 11:26:47 +01:00
Abdessamad Derraz
74f48b8881 Add emulator profiles and update docs
Add new emulator profiles (ares, BigPEmu, Eden, Model2, PrimeHack, Supermodel, Suyu, Xenia Canary, Yuzu) under emulators/*.yml with metadata, notes, file lists and exclusion/hle details. Update README counts (emulators/systems/files summary and auto-generated timestamp) and bump generated_at in database.json. Adjust mkdocs.yml navigation counts and add Yuzu entry to the Emulators nav.
2026-03-26 13:22:11 +01:00
Abdessamad Derraz
bbe4936cf2 feat: add 12 batocera standalone profiles, update 3 existing
New profiles: citron (Switch), clk (multi-system), demul (Hikaru),
eka2l1 (Symbian/N-Gage), gsplus (Apple IIGS), lexaloffle (PICO-8),
openmsx (MSX), ryujinx (Switch), shadps4 (PS4), tsugaru (FM Towns),
x16emu (Commander X16), xroar (Dragon/CoCo), zc210 (Zelda Classic).
Updated: amiberry, fsuae (standalone Amiga), ymir (Saturn standalone).
2026-03-26 07:01:16 +01:00
Abdessamad Derraz
3f676b75e8 feat: standalone emulator support for batocera and multi-platform name mapping
resolve_platform_cores() builds reverse index from profile cores: field,
fixing 17 name mismatches across Batocera, RetroBat, and Recalbox
(genesisplusgx, pce_fast, pcfx, vb, mame078plus, vice cores, etc.).

standalone_path field on file entries + standalone_cores on platform
YAMLs enable mode-aware pack generation. find_undeclared_files() uses
standalone_path for cores the platform runs standalone, filters by
mode: libretro/standalone per file.

batocera.yml gains standalone_cores (92 entries from configgen-defaults).
generate_readme.py dynamically lists platforms from registry.
3 profiles updated for standalone type/path (mame, hatari, mupen64plus_next).
78 E2E tests pass, pipeline verified.
2026-03-26 00:44:21 +01:00
Abdessamad Derraz
dfd8f8657c feat: re-profile 4 emulators, add ymir 2026-03-25 16:49:49 +01:00
Abdessamad Derraz
8a8765c0ee feat: re-profile 9 emulators, add 9 vice machine profiles 2026-03-25 15:41:36 +01:00
Abdessamad Derraz
a7bcd9b252 feat: re-profile 20 emulators, add trident and vice_x128 2026-03-25 12:23:32 +01:00
Abdessamad Derraz
ebb55a445b feat: re-profile 40 emulators, harden CI workflows
profile emulators pd777 through tic80, add frozen snapshots
(puae2021, snes9x2002/2005/2010, stella2014/2023).

CI: replace github-script with gh CLI, add test execution,
job-level permissions, propagate changed output, pin jsonschema.
2026-03-25 07:00:17 +01:00
Abdessamad Derraz
0543165ed2 feat: re-profile 22 emulators, refactor validation to common.py
batch re-profiled nekop2 through pokemini. mupen64plus renamed to
mupen64plus_next. new profiles: nes, mupen64plus_next.
validation functions (_build_validation_index, check_file_validation)
consolidated in common.py — single source of truth for verify.py
and generate_pack.py. pipeline 100% consistent on all 6 platforms.
2026-03-24 22:31:22 +01:00
Abdessamad Derraz
69987cc0bb feat: re-profile 30 emulators, beetle to parallel_n64
source-verified: mesen FdsBios.bin→alias, MesenDB.txt phantom (embedded).
mupen64plus 64DD_IPL.bin→IPL.n64 (code path fix). neocd 000-lo.lo/ng-lo.rom
phantoms (algorithmically generated). new profiles: mesen-s, mupen64plus
next develop/gles2/gles3. regenerate database.
2026-03-24 14:06:20 +01:00
Abdessamad Derraz
cda012b10a fix: source-verified corrections from review
dosbox/dosbox_pure: added validation: [sha1, size] to MT-32 ROM
entries — MUNT rejects unrecognized SHA1 (ROMInfo.cpp:171-180).
duckstation: fixed BIOS count to 105 (24 PS1 + 80 PS2 + 1 OpenBIOS).
added exclusion_note to 6 profiles with files: [] (dosbox_svn,
dosbox_svn_ce, doukutsu_rs, emux_chip8, emux_nes, fake08).
2026-03-24 12:21:32 +01:00
Abdessamad Derraz
491ea2f9ba fix: source-verified corrections on 7 profiles
beetle_pce: +core_classification, +upstream, cores=[mednafen_pce],
alt_names→aliases. 6 files confirmed loaded by code.
mame: removed 19 phantom XaviX entries (MACHINE_IS_BIOS_ROOT commented
out in source). 80 real BIOS sets verified.
mame2010: fixed galgbios.zip source_ref, documented cheat.zip as
non-functional in libretro port (cheatpath never initialized).
ishiiruka: trimmed verbose advisory notes to factual descriptions.
mamearcade: added system: field to 29 bios_zip entries, 24 new systems.
mednafen_ngp: removed redundant cores: on alias, added note:.
mednafen_gba: normalized validation format to flow sequence.
2026-03-24 12:17:45 +01:00
Abdessamad Derraz
ef25f8cebf feat: re-profile 40 emulators from fbalpha2012 to holani
batch re-profiled beetle, holani, kronos, mame variants, mednafen
cores with upstream verification. beetle_pce, mame, mame2010 excluded
(regressions detected, need manual review).
2026-03-24 11:36:58 +01:00
Abdessamad Derraz
ef365c6680 fix: restore 4 neo-geo BIOS removed by batch re-profile
trackball BIOS (236-bios.sp1, 236-bios_japan_hack.sp1) for irrmaze,
PCB BIOS (sp-4x.sp1, spj.sp1) for ms5pcb/svcpcb/kf2k3pcb. all carry
BRF_ESS flag in d_neogeo.cpp, verified against source.
2026-03-24 10:43:35 +01:00