Commit Graph

503 Commits

Author SHA1 Message Date
Abdessamad Derraz
d415777f2c feat: add PS3UPDAT.PUP to rpcs3 profile
storage: large_file, validated via SCEUF magic + HMAC-SHA1
per entry (PUP.cpp:23-77). also adds missing standard fields
(cores, core_classification, upstream), removes non-standard
firmware_file/validation/firmware_version fields.
2026-03-30 15:06:51 +02:00
Abdessamad Derraz
eafabd20f3 refactor: skip writing generated files when content unchanged
write_if_changed in common.py compares content after stripping
timestamps (generated_at, Auto-generated on, Generated on).
applied to generate_db, generate_readme, generate_site.
eliminates timestamp-only diffs in database.json, README.md,
mkdocs.yml, and 423 docs pages.
2026-03-30 14:33:44 +02:00
Abdessamad Derraz
2aca4927c0 chore: regenerate database, readme, manifests, site 2026-03-30 14:19:00 +02:00
Abdessamad Derraz
17777f315b feat: agnostic bios mode for filename-agnostic emulators
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
2026-03-30 14:18:54 +02:00
Abdessamad Derraz
692484d32d refactor: remove false aliases from pcsx2 and melonds profiles
aliases must be same-SHA1 alternative names, not distinct files.
pcsx2: 164 different BIOS dumps are separate DB entries, not aliases.
melonds: 6 regional NAND dumps are separate DB entries, not aliases.
also cleans pcsx2 non-standard fields, fixes display_name.
2026-03-30 12:37:32 +02:00
Abdessamad Derraz
a8430940f9 feat: add regional nand aliases to melonds profile
dsi_nand.bin aliases: DSi_Nand_USA/EUR/JPN/AUS/CHN/KOR.bin
for repo resolution. Code loads a single configurable path
(libretro.cpp:836, EmuInstance.cpp:1036-1050), validates
nocash footer (DSi_NAND.cpp:42-111). size + storage added.
2026-03-30 12:20:26 +02:00
Abdessamad Derraz
1f073f521d fix: preserve batocera version when github fetch fails 2026-03-30 11:55:57 +02:00
Abdessamad Derraz
903c49edcf feat: add tests for registry merge, all_libretro expansion, hash fallback, system normalization 2026-03-30 11:33:59 +02:00
Abdessamad Derraz
d3a2224dd2 chore: regenerate database, readme, manifests, site 2026-03-30 09:42:39 +02:00
Abdessamad Derraz
f898f26847 chore: regenerate retrobat.yml with corrected system slugs 2026-03-30 09:11:00 +02:00
Abdessamad Derraz
2712307420 feat: add tandy vis bios root set to mame profiles
vis.zip (p513bk0b.bin + p513bk1b.bin) from src/mame/trs/vis.cpp.
Driver added in MAME 0.180, not present in older cores.
2026-03-30 09:09:58 +02:00
Abdessamad Derraz
54022e9db1 feat: hash-based matching for cross-reference
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
2026-03-30 08:25:54 +02:00
Abdessamad Derraz
4db9e4350c fix: add missing system slugs to batocera and retrobat scrapers 2026-03-30 07:58:46 +02:00
Abdessamad Derraz
6864ce6584 feat: diff hash fallback detects platform renames 2026-03-30 07:53:59 +02:00
Abdessamad Derraz
12196b6445 feat: add 55 missing cores across 6 platform registries 2026-03-30 07:17:57 +02:00
Abdessamad Derraz
7551e41a7b feat: load_platform_config merges cores from registry 2026-03-30 07:13:15 +02:00
Abdessamad Derraz
7b484605d4 feat: add 47 missing cores to batocera platform config 2026-03-30 07:09:52 +02:00
Abdessamad Derraz
b587381f05 feat: resolve_platform_cores expands all_libretro in list 2026-03-30 07:06:32 +02:00
Abdessamad Derraz
261f3c1671 chore: regenerate database 2026-03-30 06:15:02 +02:00
Abdessamad Derraz
d3bd68f6ad feat: add vsmile, tutor, vg5k, vgmplay profiles and system aliases 2026-03-30 06:14:58 +02:00
Abdessamad Derraz
8f29e0e85c Update DB timestamp; add ti99sim emulator
Bump database.json generated_at timestamp and add a new emulator metadata file emulators/ti99sim.yml. The new YAML registers the TI-99/Sim standalone emulator (ti99sim) with system info, core version, notes on ROM handling, and multiple required/optional ROM entries (with SHA1s, sizes and validation notes) for TI-99/4A support.
2026-03-29 23:33:12 +02:00
Abdessamad Derraz
f39b11955f feat: add ROM_OS_B_NTSC and fs-5500_disk.rom, fix retrodeck
Add Atari 800 OS Rev B NTSC (CRC32 0e86d61d, canonical sysrom.c
match) and National FS-5500 disk controller ROM for openMSX.
Remove ROM_400/800_CUSTOM from retrodeck.yml (config slot key with
forward slash in name, not a real file).
2026-03-29 23:23:52 +02:00
Abdessamad Derraz
83ccf17b11 feat: add ti99 and msx BIOS entries, add fs-5500 disk rom 2026-03-29 23:20:44 +02:00
Abdessamad Derraz
56bff1d013 feat: add ti99 system to mame profile 2026-03-29 23:16:33 +02:00
Abdessamad Derraz
3bd9c0ebef chore: regenerate database, add atari 400/800 BIOS 2026-03-29 23:16:19 +02:00
Abdessamad Derraz
3824193a11 feat: add socrates and vc4000 systems to mame profiles 2026-03-29 23:15:25 +02:00
Abdessamad Derraz
2e21d64a08 refactor: harden codebase and remove unicode artifacts
- fix urllib.parse.quote import (was urllib.request.quote)
- add operator precedence parens in generate_pack dedup check
- narrow bare except to specific types in batocera target scraper
- cache load_platform_config and build_zip_contents_index results
- add selective algorithm support to compute_hashes
- atomic write for fetch_large_file (tmp + rename)
- add response size limit to base scraper fetch
- extract build_target_cores_cache to common.py (dedup verify/pack)
- hoist _build_supplemental_index out of per-platform loop
- migrate function-attribute caches to module-level dicts
- add @abstractmethod to BaseTargetScraper.fetch_targets
- remove backward-compat re-exports from common.py
- replace em-dashes and unicode arrows with ASCII equivalents
- remove decorative section dividers and obvious comments
2026-03-29 23:15:20 +02:00
Abdessamad Derraz
0c5cde83e1 Add TRS-80, RX-78, Sega AI entries; refactor tools
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
2026-03-29 23:04:30 +02:00
Abdessamad Derraz
a08c730805 fix: case-insensitive data dir basename resolution 2026-03-29 23:01:32 +02:00
Abdessamad Derraz
84decad08d chore: remove zero-byte placeholder files from bios/ 2026-03-29 22:57:07 +02:00
Abdessamad Derraz
f3de3ead20 Add PV-2000 & Supracan BIOS; simplify Vita3K
Add Casio PV-2000 BIOS entry (pv2000.zip) to MAME and MESS profiles and update system lists/counts. Add Funtech Super A'Can BIOS entries (supracan.zip and umc6650.zip) with ROM contents to mamemess. Simplify and condense Vita3K emulator profile (rename fields, update profiled_date, add PSVUPDAT.PUP and optional PSP2UPDAT.PUP file entries, and clarify install/partition behavior). Bump database generated_at timestamp and add a system alias mapping "psvita" -> "sony-playstation-vita" in scripts/common.py.
2026-03-29 22:51:30 +02:00
Abdessamad Derraz
463fca7e7d Regenerate database and update emulator profiles
Regenerate database.json and update README counts/timestamps; add and normalize numerous BIOS entries and hashes. Key changes: update generated_at timestamp and system count (355→357) in README; add OpenBIOS / HLE fallback and additional aliases to beetle_psx, include beetle_psx core name and profiled_date update; add laseractive to ares systems; adjust atari800 systems and source_ref line numbers; mark dinothawr as a system and expand its note; update gsplus upstream/profile date, add apple-iie system and tweak source_refs; add pcsx2 core to lrps2; refresh mame profiled_date and add multiple systems and BIOS root sets. Miscellaneous script changes and other JSON normalization to reflect newly discovered/merged BIOS files.
2026-03-29 22:41:01 +02:00
Abdessamad Derraz
daa396618d refactor: deduplicate hash comparison loop in check_file_validation 2026-03-29 17:15:24 +02:00
Abdessamad Derraz
2c2b761e60 refactor: extract helpers from print_platform_result in verify.py 2026-03-29 17:11:33 +02:00
Abdessamad Derraz
be68ed19a3 refactor: extract dispatch functions from generate_pack main() 2026-03-29 17:10:01 +02:00
Abdessamad Derraz
b4c5d77e4b refactor: deduplicate yaml import pattern via require_yaml() 2026-03-29 17:07:27 +02:00
Abdessamad Derraz
3c7fc26354 refactor: extract validation and truth modules from common.py 2026-03-29 16:41:24 +02:00
Abdessamad Derraz
95e16c9e7a Add Adam/Advision/APF systems and update metadata
Add support for Coleco Adam, Entex Adventure Vision and APF M-1000 BIOS/ROM sets in MAME and MESS metadata (multiple Adam device MCU ROMs and optional FDC/SPI variants, Advision and APF BIOS entries). Update generated metadata across the repo: README coverage numbers and per-platform coverage rows, database generated timestamp and totals (total_files 7245), and various install manifests (notably batocera.json) with new timestamps, adjusted file counts/sizes, SHA1s, repo_path fixes and an added adam_fdc_320kb.zip entry. Also update notes to reflect the new system ROM sets in the emulators entries.
2026-03-29 16:07:15 +02:00
Abdessamad Derraz
f621f3db84 feat: truth includes systems with 0 files from resolved cores 2026-03-29 14:28:52 +02:00
Abdessamad Derraz
cd2a3bdfa2 chore: add acorn, gce, texas-instruments to manufacturer prefixes 2026-03-29 14:25:40 +02:00
Abdessamad Derraz
d274d8a65c fix: guard MD5 lookup against cross-name contamination
resolve_local_file step 2 (pure MD5 lookup) now verifies that the
found file's name matches the requested name or is a .variants/
derivative. Prevents serving wrong files when an unrelated file
shares the same MD5 in the index (e.g. spi.zip returned for
a7ports.zip because RetroDECK expected an MD5 we don't have).
2026-03-29 14:15:06 +02:00
Abdessamad Derraz
13b20c2742 fix: normalize system IDs in diff matching 2026-03-29 14:07:14 +02:00
Abdessamad Derraz
ae71a41b32 feat: verify_pack checks data dirs and rebuilt ZIPs
verify_pack now verifies files against data directory caches and
validates rebuilt MAME ZIPs by comparing inner ROM CRC32s against
source. Reduces false untracked count from 6242 to 0 for RetroArch.
2026-03-29 14:04:46 +02:00
Abdessamad Derraz
dee37c2530 fix: truth generation uses platform config not registry 2026-03-29 13:41:49 +02:00
Abdessamad Derraz
3bf6e5c961 feat: add truth and diff integration test 2026-03-29 13:27:32 +02:00
Abdessamad Derraz
a800ded477 chore: remove 4 remaining bios/data duplicates 2026-03-29 13:25:33 +02:00
Abdessamad Derraz
ac2502d767 feat: add --with-truth and --with-export to pipeline 2026-03-29 13:22:20 +02:00
Abdessamad Derraz
bb257f52ce feat: add export_native.py CLI 2026-03-29 13:21:08 +02:00
Abdessamad Derraz
e86d8d68af feat: add exporter plugin architecture 2026-03-29 13:19:38 +02:00
Abdessamad Derraz
2ce8db1754 feat: add native_id to scraper output 2026-03-29 13:15:57 +02:00