Commit Graph

441 Commits

Author SHA1 Message Date
Abdessamad Derraz
4db9e4350c fix: add missing system slugs to batocera and retrobat scrapers 2026-03-30 07:58:46 +02:00
Abdessamad Derraz
6864ce6584 feat: diff hash fallback detects platform renames 2026-03-30 07:53:59 +02:00
Abdessamad Derraz
12196b6445 feat: add 55 missing cores across 6 platform registries 2026-03-30 07:17:57 +02:00
Abdessamad Derraz
7551e41a7b feat: load_platform_config merges cores from registry 2026-03-30 07:13:15 +02:00
Abdessamad Derraz
7b484605d4 feat: add 47 missing cores to batocera platform config 2026-03-30 07:09:52 +02:00
Abdessamad Derraz
b587381f05 feat: resolve_platform_cores expands all_libretro in list 2026-03-30 07:06:32 +02:00
Abdessamad Derraz
261f3c1671 chore: regenerate database 2026-03-30 06:15:02 +02:00
Abdessamad Derraz
d3bd68f6ad feat: add vsmile, tutor, vg5k, vgmplay profiles and system aliases 2026-03-30 06:14:58 +02:00
Abdessamad Derraz
8f29e0e85c Update DB timestamp; add ti99sim emulator
Bump database.json generated_at timestamp and add a new emulator metadata file emulators/ti99sim.yml. The new YAML registers the TI-99/Sim standalone emulator (ti99sim) with system info, core version, notes on ROM handling, and multiple required/optional ROM entries (with SHA1s, sizes and validation notes) for TI-99/4A support.
2026-03-29 23:33:12 +02:00
Abdessamad Derraz
f39b11955f feat: add ROM_OS_B_NTSC and fs-5500_disk.rom, fix retrodeck
Add Atari 800 OS Rev B NTSC (CRC32 0e86d61d, canonical sysrom.c
match) and National FS-5500 disk controller ROM for openMSX.
Remove ROM_400/800_CUSTOM from retrodeck.yml (config slot key with
forward slash in name, not a real file).
2026-03-29 23:23:52 +02:00
Abdessamad Derraz
83ccf17b11 feat: add ti99 and msx BIOS entries, add fs-5500 disk rom 2026-03-29 23:20:44 +02:00
Abdessamad Derraz
56bff1d013 feat: add ti99 system to mame profile 2026-03-29 23:16:33 +02:00
Abdessamad Derraz
3bd9c0ebef chore: regenerate database, add atari 400/800 BIOS 2026-03-29 23:16:19 +02:00
Abdessamad Derraz
3824193a11 feat: add socrates and vc4000 systems to mame profiles 2026-03-29 23:15:25 +02:00
Abdessamad Derraz
2e21d64a08 refactor: harden codebase and remove unicode artifacts
- fix urllib.parse.quote import (was urllib.request.quote)
- add operator precedence parens in generate_pack dedup check
- narrow bare except to specific types in batocera target scraper
- cache load_platform_config and build_zip_contents_index results
- add selective algorithm support to compute_hashes
- atomic write for fetch_large_file (tmp + rename)
- add response size limit to base scraper fetch
- extract build_target_cores_cache to common.py (dedup verify/pack)
- hoist _build_supplemental_index out of per-platform loop
- migrate function-attribute caches to module-level dicts
- add @abstractmethod to BaseTargetScraper.fetch_targets
- remove backward-compat re-exports from common.py
- replace em-dashes and unicode arrows with ASCII equivalents
- remove decorative section dividers and obvious comments
2026-03-29 23:15:20 +02:00
Abdessamad Derraz
0c5cde83e1 Add TRS-80, RX-78, Sega AI entries; refactor tools
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
2026-03-29 23:04:30 +02:00
Abdessamad Derraz
a08c730805 fix: case-insensitive data dir basename resolution 2026-03-29 23:01:32 +02:00
Abdessamad Derraz
84decad08d chore: remove zero-byte placeholder files from bios/ 2026-03-29 22:57:07 +02:00
Abdessamad Derraz
f3de3ead20 Add PV-2000 & Supracan BIOS; simplify Vita3K
Add Casio PV-2000 BIOS entry (pv2000.zip) to MAME and MESS profiles and update system lists/counts. Add Funtech Super A'Can BIOS entries (supracan.zip and umc6650.zip) with ROM contents to mamemess. Simplify and condense Vita3K emulator profile (rename fields, update profiled_date, add PSVUPDAT.PUP and optional PSP2UPDAT.PUP file entries, and clarify install/partition behavior). Bump database generated_at timestamp and add a system alias mapping "psvita" -> "sony-playstation-vita" in scripts/common.py.
2026-03-29 22:51:30 +02:00
Abdessamad Derraz
463fca7e7d Regenerate database and update emulator profiles
Regenerate database.json and update README counts/timestamps; add and normalize numerous BIOS entries and hashes. Key changes: update generated_at timestamp and system count (355→357) in README; add OpenBIOS / HLE fallback and additional aliases to beetle_psx, include beetle_psx core name and profiled_date update; add laseractive to ares systems; adjust atari800 systems and source_ref line numbers; mark dinothawr as a system and expand its note; update gsplus upstream/profile date, add apple-iie system and tweak source_refs; add pcsx2 core to lrps2; refresh mame profiled_date and add multiple systems and BIOS root sets. Miscellaneous script changes and other JSON normalization to reflect newly discovered/merged BIOS files.
2026-03-29 22:41:01 +02:00
Abdessamad Derraz
daa396618d refactor: deduplicate hash comparison loop in check_file_validation 2026-03-29 17:15:24 +02:00
Abdessamad Derraz
2c2b761e60 refactor: extract helpers from print_platform_result in verify.py 2026-03-29 17:11:33 +02:00
Abdessamad Derraz
be68ed19a3 refactor: extract dispatch functions from generate_pack main() 2026-03-29 17:10:01 +02:00
Abdessamad Derraz
b4c5d77e4b refactor: deduplicate yaml import pattern via require_yaml() 2026-03-29 17:07:27 +02:00
Abdessamad Derraz
3c7fc26354 refactor: extract validation and truth modules from common.py 2026-03-29 16:41:24 +02:00
Abdessamad Derraz
95e16c9e7a Add Adam/Advision/APF systems and update metadata
Add support for Coleco Adam, Entex Adventure Vision and APF M-1000 BIOS/ROM sets in MAME and MESS metadata (multiple Adam device MCU ROMs and optional FDC/SPI variants, Advision and APF BIOS entries). Update generated metadata across the repo: README coverage numbers and per-platform coverage rows, database generated timestamp and totals (total_files 7245), and various install manifests (notably batocera.json) with new timestamps, adjusted file counts/sizes, SHA1s, repo_path fixes and an added adam_fdc_320kb.zip entry. Also update notes to reflect the new system ROM sets in the emulators entries.
2026-03-29 16:07:15 +02:00
Abdessamad Derraz
f621f3db84 feat: truth includes systems with 0 files from resolved cores 2026-03-29 14:28:52 +02:00
Abdessamad Derraz
cd2a3bdfa2 chore: add acorn, gce, texas-instruments to manufacturer prefixes 2026-03-29 14:25:40 +02:00
Abdessamad Derraz
d274d8a65c fix: guard MD5 lookup against cross-name contamination
resolve_local_file step 2 (pure MD5 lookup) now verifies that the
found file's name matches the requested name or is a .variants/
derivative. Prevents serving wrong files when an unrelated file
shares the same MD5 in the index (e.g. spi.zip returned for
a7ports.zip because RetroDECK expected an MD5 we don't have).
2026-03-29 14:15:06 +02:00
Abdessamad Derraz
13b20c2742 fix: normalize system IDs in diff matching 2026-03-29 14:07:14 +02:00
Abdessamad Derraz
ae71a41b32 feat: verify_pack checks data dirs and rebuilt ZIPs
verify_pack now verifies files against data directory caches and
validates rebuilt MAME ZIPs by comparing inner ROM CRC32s against
source. Reduces false untracked count from 6242 to 0 for RetroArch.
2026-03-29 14:04:46 +02:00
Abdessamad Derraz
dee37c2530 fix: truth generation uses platform config not registry 2026-03-29 13:41:49 +02:00
Abdessamad Derraz
3bf6e5c961 feat: add truth and diff integration test 2026-03-29 13:27:32 +02:00
Abdessamad Derraz
a800ded477 chore: remove 4 remaining bios/data duplicates 2026-03-29 13:25:33 +02:00
Abdessamad Derraz
ac2502d767 feat: add --with-truth and --with-export to pipeline 2026-03-29 13:22:20 +02:00
Abdessamad Derraz
bb257f52ce feat: add export_native.py CLI 2026-03-29 13:21:08 +02:00
Abdessamad Derraz
e86d8d68af feat: add exporter plugin architecture 2026-03-29 13:19:38 +02:00
Abdessamad Derraz
2ce8db1754 feat: add native_id to scraper output 2026-03-29 13:15:57 +02:00
Abdessamad Derraz
5247512bea feat: add diff_truth.py CLI 2026-03-29 13:12:29 +02:00
Abdessamad Derraz
2aab7420d7 feat: add diff_platform_truth function and tests 2026-03-29 13:09:08 +02:00
Abdessamad Derraz
97eb4835be feat: add load_from field for non-system_dir files
Replaces mode: standalone hack with load_from: save_dir on Panda3DS
files. The load_from field documents which libretro directory callback
provides the base path (system_dir default, save_dir, content_dir).
Pack generator and cross-reference skip files not targeting system_dir.
2026-03-29 13:07:30 +02:00
Abdessamad Derraz
2dd2b724c1 feat: add generate_truth.py CLI 2026-03-29 13:05:51 +02:00
Abdessamad Derraz
525f51d5fb feat: add truth mode filtering and dedup tests 2026-03-29 13:01:37 +02:00
Abdessamad Derraz
d38f2b7180 fix: exclude save_dir paths from pack multi-dest 2026-03-29 12:02:30 +02:00
Abdessamad Derraz
abca70cb5b feat: add generate_platform_truth to common.py 2026-03-29 11:22:35 +02:00
Abdessamad Derraz
5ac48ef703 fix: recursive basename search in data dir fallback 2026-03-29 11:18:03 +02:00
Abdessamad Derraz
44145d41cd chore: remove 399 bios/ files now sourced from buildbot
Files verified by MD5 to be identical to their buildbot-fetched
copies in data/. resolve_local_file data directory fallback ensures
they remain resolvable for verify and pack generation.
2026-03-29 11:13:08 +02:00
Abdessamad Derraz
c513d6c0ad feat: resolve_local_file data directory fallback 2026-03-29 11:08:31 +02:00
Abdessamad Derraz
500e33d1ff feat: add --check-buildbot pipeline step 2026-03-29 11:04:04 +02:00
Abdessamad Derraz
b9be8b9783 feat: add buildbot system change detection 2026-03-29 11:03:15 +02:00