Commit Graph

436 Commits

Author SHA1 Message Date
Abdessamad Derraz
b587381f05 feat: resolve_platform_cores expands all_libretro in list 2026-03-30 07:06:32 +02:00
Abdessamad Derraz
261f3c1671 chore: regenerate database 2026-03-30 06:15:02 +02:00
Abdessamad Derraz
d3bd68f6ad feat: add vsmile, tutor, vg5k, vgmplay profiles and system aliases 2026-03-30 06:14:58 +02:00
Abdessamad Derraz
8f29e0e85c Update DB timestamp; add ti99sim emulator
Bump database.json generated_at timestamp and add a new emulator metadata file emulators/ti99sim.yml. The new YAML registers the TI-99/Sim standalone emulator (ti99sim) with system info, core version, notes on ROM handling, and multiple required/optional ROM entries (with SHA1s, sizes and validation notes) for TI-99/4A support.
2026-03-29 23:33:12 +02:00
Abdessamad Derraz
f39b11955f feat: add ROM_OS_B_NTSC and fs-5500_disk.rom, fix retrodeck
Add Atari 800 OS Rev B NTSC (CRC32 0e86d61d, canonical sysrom.c
match) and National FS-5500 disk controller ROM for openMSX.
Remove ROM_400/800_CUSTOM from retrodeck.yml (config slot key with
forward slash in name, not a real file).
2026-03-29 23:23:52 +02:00
Abdessamad Derraz
83ccf17b11 feat: add ti99 and msx BIOS entries, add fs-5500 disk rom 2026-03-29 23:20:44 +02:00
Abdessamad Derraz
56bff1d013 feat: add ti99 system to mame profile 2026-03-29 23:16:33 +02:00
Abdessamad Derraz
3bd9c0ebef chore: regenerate database, add atari 400/800 BIOS 2026-03-29 23:16:19 +02:00
Abdessamad Derraz
3824193a11 feat: add socrates and vc4000 systems to mame profiles 2026-03-29 23:15:25 +02:00
Abdessamad Derraz
2e21d64a08 refactor: harden codebase and remove unicode artifacts
- fix urllib.parse.quote import (was urllib.request.quote)
- add operator precedence parens in generate_pack dedup check
- narrow bare except to specific types in batocera target scraper
- cache load_platform_config and build_zip_contents_index results
- add selective algorithm support to compute_hashes
- atomic write for fetch_large_file (tmp + rename)
- add response size limit to base scraper fetch
- extract build_target_cores_cache to common.py (dedup verify/pack)
- hoist _build_supplemental_index out of per-platform loop
- migrate function-attribute caches to module-level dicts
- add @abstractmethod to BaseTargetScraper.fetch_targets
- remove backward-compat re-exports from common.py
- replace em-dashes and unicode arrows with ASCII equivalents
- remove decorative section dividers and obvious comments
2026-03-29 23:15:20 +02:00
Abdessamad Derraz
0c5cde83e1 Add TRS-80, RX-78, Sega AI entries; refactor tools
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
2026-03-29 23:04:30 +02:00
Abdessamad Derraz
a08c730805 fix: case-insensitive data dir basename resolution 2026-03-29 23:01:32 +02:00
Abdessamad Derraz
84decad08d chore: remove zero-byte placeholder files from bios/ 2026-03-29 22:57:07 +02:00
Abdessamad Derraz
f3de3ead20 Add PV-2000 & Supracan BIOS; simplify Vita3K
Add Casio PV-2000 BIOS entry (pv2000.zip) to MAME and MESS profiles and update system lists/counts. Add Funtech Super A'Can BIOS entries (supracan.zip and umc6650.zip) with ROM contents to mamemess. Simplify and condense Vita3K emulator profile (rename fields, update profiled_date, add PSVUPDAT.PUP and optional PSP2UPDAT.PUP file entries, and clarify install/partition behavior). Bump database generated_at timestamp and add a system alias mapping "psvita" -> "sony-playstation-vita" in scripts/common.py.
2026-03-29 22:51:30 +02:00
Abdessamad Derraz
463fca7e7d Regenerate database and update emulator profiles
Regenerate database.json and update README counts/timestamps; add and normalize numerous BIOS entries and hashes. Key changes: update generated_at timestamp and system count (355→357) in README; add OpenBIOS / HLE fallback and additional aliases to beetle_psx, include beetle_psx core name and profiled_date update; add laseractive to ares systems; adjust atari800 systems and source_ref line numbers; mark dinothawr as a system and expand its note; update gsplus upstream/profile date, add apple-iie system and tweak source_refs; add pcsx2 core to lrps2; refresh mame profiled_date and add multiple systems and BIOS root sets. Miscellaneous script changes and other JSON normalization to reflect newly discovered/merged BIOS files.
2026-03-29 22:41:01 +02:00
Abdessamad Derraz
daa396618d refactor: deduplicate hash comparison loop in check_file_validation 2026-03-29 17:15:24 +02:00
Abdessamad Derraz
2c2b761e60 refactor: extract helpers from print_platform_result in verify.py 2026-03-29 17:11:33 +02:00
Abdessamad Derraz
be68ed19a3 refactor: extract dispatch functions from generate_pack main() 2026-03-29 17:10:01 +02:00
Abdessamad Derraz
b4c5d77e4b refactor: deduplicate yaml import pattern via require_yaml() 2026-03-29 17:07:27 +02:00
Abdessamad Derraz
3c7fc26354 refactor: extract validation and truth modules from common.py 2026-03-29 16:41:24 +02:00
Abdessamad Derraz
95e16c9e7a Add Adam/Advision/APF systems and update metadata
Add support for Coleco Adam, Entex Adventure Vision and APF M-1000 BIOS/ROM sets in MAME and MESS metadata (multiple Adam device MCU ROMs and optional FDC/SPI variants, Advision and APF BIOS entries). Update generated metadata across the repo: README coverage numbers and per-platform coverage rows, database generated timestamp and totals (total_files 7245), and various install manifests (notably batocera.json) with new timestamps, adjusted file counts/sizes, SHA1s, repo_path fixes and an added adam_fdc_320kb.zip entry. Also update notes to reflect the new system ROM sets in the emulators entries.
2026-03-29 16:07:15 +02:00
Abdessamad Derraz
f621f3db84 feat: truth includes systems with 0 files from resolved cores 2026-03-29 14:28:52 +02:00
Abdessamad Derraz
cd2a3bdfa2 chore: add acorn, gce, texas-instruments to manufacturer prefixes 2026-03-29 14:25:40 +02:00
Abdessamad Derraz
d274d8a65c fix: guard MD5 lookup against cross-name contamination
resolve_local_file step 2 (pure MD5 lookup) now verifies that the
found file's name matches the requested name or is a .variants/
derivative. Prevents serving wrong files when an unrelated file
shares the same MD5 in the index (e.g. spi.zip returned for
a7ports.zip because RetroDECK expected an MD5 we don't have).
2026-03-29 14:15:06 +02:00
Abdessamad Derraz
13b20c2742 fix: normalize system IDs in diff matching 2026-03-29 14:07:14 +02:00
Abdessamad Derraz
ae71a41b32 feat: verify_pack checks data dirs and rebuilt ZIPs
verify_pack now verifies files against data directory caches and
validates rebuilt MAME ZIPs by comparing inner ROM CRC32s against
source. Reduces false untracked count from 6242 to 0 for RetroArch.
2026-03-29 14:04:46 +02:00
Abdessamad Derraz
dee37c2530 fix: truth generation uses platform config not registry 2026-03-29 13:41:49 +02:00
Abdessamad Derraz
3bf6e5c961 feat: add truth and diff integration test 2026-03-29 13:27:32 +02:00
Abdessamad Derraz
a800ded477 chore: remove 4 remaining bios/data duplicates 2026-03-29 13:25:33 +02:00
Abdessamad Derraz
ac2502d767 feat: add --with-truth and --with-export to pipeline 2026-03-29 13:22:20 +02:00
Abdessamad Derraz
bb257f52ce feat: add export_native.py CLI 2026-03-29 13:21:08 +02:00
Abdessamad Derraz
e86d8d68af feat: add exporter plugin architecture 2026-03-29 13:19:38 +02:00
Abdessamad Derraz
2ce8db1754 feat: add native_id to scraper output 2026-03-29 13:15:57 +02:00
Abdessamad Derraz
5247512bea feat: add diff_truth.py CLI 2026-03-29 13:12:29 +02:00
Abdessamad Derraz
2aab7420d7 feat: add diff_platform_truth function and tests 2026-03-29 13:09:08 +02:00
Abdessamad Derraz
97eb4835be feat: add load_from field for non-system_dir files
Replaces mode: standalone hack with load_from: save_dir on Panda3DS
files. The load_from field documents which libretro directory callback
provides the base path (system_dir default, save_dir, content_dir).
Pack generator and cross-reference skip files not targeting system_dir.
2026-03-29 13:07:30 +02:00
Abdessamad Derraz
2dd2b724c1 feat: add generate_truth.py CLI 2026-03-29 13:05:51 +02:00
Abdessamad Derraz
525f51d5fb feat: add truth mode filtering and dedup tests 2026-03-29 13:01:37 +02:00
Abdessamad Derraz
d38f2b7180 fix: exclude save_dir paths from pack multi-dest 2026-03-29 12:02:30 +02:00
Abdessamad Derraz
abca70cb5b feat: add generate_platform_truth to common.py 2026-03-29 11:22:35 +02:00
Abdessamad Derraz
5ac48ef703 fix: recursive basename search in data dir fallback 2026-03-29 11:18:03 +02:00
Abdessamad Derraz
44145d41cd chore: remove 399 bios/ files now sourced from buildbot
Files verified by MD5 to be identical to their buildbot-fetched
copies in data/. resolve_local_file data directory fallback ensures
they remain resolvable for verify and pack generation.
2026-03-29 11:13:08 +02:00
Abdessamad Derraz
c513d6c0ad feat: resolve_local_file data directory fallback 2026-03-29 11:08:31 +02:00
Abdessamad Derraz
500e33d1ff feat: add --check-buildbot pipeline step 2026-03-29 11:04:04 +02:00
Abdessamad Derraz
b9be8b9783 feat: add buildbot system change detection 2026-03-29 11:03:15 +02:00
Abdessamad Derraz
44cb444a7b chore: regenerate install manifests after path fixes 2026-03-29 10:13:02 +02:00
Abdessamad Derraz
e13e8734b9 chore: standardize notes: to note: in file entries
240 file-level entries used notes: instead of the canonical note:
field. verify.py and cross_reference.py only read note:, so these
were silently ignored.
2026-03-29 10:02:20 +02:00
Abdessamad Derraz
b7528a71e7 fix: correct type and path fields across 58 emulator profiles
49 libretro cores had type: game/utility/test instead of type: libretro,
breaking the all_libretro filtering in resolve_platform_cores and
excluding them from platform packs (e.g. cannonball missing from
RetroArch). core_classification already carries the descriptive role.

9 profiles with subdirectory-loading cores (cannonball/, nxengine/,
Citra/sysdata/, mame2003/, mame2003-plus/, mame2010/) now have path:
fields so cross-reference places files at the correct destination.

resolve_local_file now tries basename when name contains a path
separator (e.g. res/tilemap.bin -> tilemap.bin), fixing resolution
of files with subdirectory names.
2026-03-29 09:53:46 +02:00
Abdessamad Derraz
36f6ad1379 fix: correct stale scummvm.zip and cpc464.rom hashes
RetroDECK had stale buildbot hash a17e0e01 for scummvm.zip
(from old 9.5MB build, current is 79MB) copied to cpc464.rom.
RomM had same stale hash. Updated to current verified values.
All 8 platforms now 100% OK, 0 untested.
2026-03-29 09:52:32 +02:00
Abdessamad Derraz
718ea2192d feat: add FDS original BIOS and GameCube DSP hardware dumps
disksys.rom non-Rev1 variant (SHA256 fdc1a76e, ares-compatible)
from Myrient No-Intro. GameCube dsp_rom.bin + dsp_coef.bin real
hardware dumps (adler32 match Dolphin) from Redump collection.
All placed as .variants/ — primaries unchanged.
2026-03-29 09:35:48 +02:00