Commit Graph

65 Commits

Author SHA1 Message Date
Abdessamad Derraz
6a21a99c22 feat: platform-core registry for exact pack generation
resolve_platform_cores() links platforms to their cores via
three strategies: all_libretro, explicit list, system ID
fallback. Pack generation always includes core requirements
beyond platform baseline. Case-insensitive dedup prevents
conflicts on Windows/macOS. Data dir strip_components fixes
doubled paths for Dolphin and PPSSPP caches.
2026-03-19 16:10:43 +01:00
Abdessamad Derraz
38d605c7d5 fix: audit fixes across verify, pack, security, and performance
- fix KeyError in compute_coverage (generate_readme, generate_site)
- fix comma-separated MD5 handling in generate_pack check_inside_zip
- fix _verify_file_hash to handle multi-MD5 for large files
- fix external downloads not tracked in seen_destinations/file_status
- fix tar path traversal in _is_safe_tar_member (refresh_data_dirs)
- fix predictable tmp path in download.py
- fix _sanitize_path to filter "." components
- remove blanket data_dir suppression in find_undeclared_files
- remove blanket data_dir suppression in cross_reference
- add status_counts to verify_platform return value
- add md5_composite cache for repeated ZIP hashing
2026-03-19 14:04:34 +01:00
Abdessamad Derraz
6d9edc5110 fix: review findings — hoist constants, cache emu profiles, renumber steps
- Hoist sev_order/sev_prio dicts to module-level constants (was rebuilt
  every loop iteration)
- Cache emulator profiles across platforms in verify main() (was loading
  260 YAMLs per platform, now loaded once)
- Renumber resolve_local_file steps 1-5 (was 1,2,3,5,6 after removal)
- Pass emu_profiles through verify_platform → find_undeclared_files
2026-03-19 11:22:58 +01:00
Abdessamad Derraz
b9cdda07ee refactor: DRY consolidation + 83 unit tests
Moved shared functions to common.py (single source of truth):
- check_inside_zip (was in verify.py, imported by generate_pack)
- build_zip_contents_index (was duplicated in verify + generate_pack)
- load_emulator_profiles (was in verify, cross_reference, generate_site)
- group_identical_platforms (was in verify + generate_pack)

Added tests/ with 83 unit tests covering:
- resolve_local_file: SHA1, MD5, name, alias, truncated, zip_contents
- verify: existence, md5, zipped_file, multi-hash, severity mapping
- aliases: field parsing, by_name indexing, beetle_psx field rename
- pack: dedup, file_status, zipped_file inner check, EmuDeck entries
- severity: all 12 combinations, platform-native behavior

0 regressions: pipeline.py --all produces identical results.
2026-03-19 11:19:50 +01:00
Abdessamad Derraz
f3db61162c feat: aliases support in resolve and db generation
generate_db.py now reads aliases from emulator YAMLs and indexes
them in database.json by_name. resolve_local_file in common.py
tries all alias names when the primary name fails to match.

beetle_psx alt_names renamed to aliases (was not indexed before).
snes9x BS-X.bios, np2kai FONT.ROM/ide.rom/pci.rom fallbacks,
all now formally declared as aliases and indexed.

verify --all and generate_pack --all pass with 0 regressions.
2026-03-19 08:15:13 +01:00
Abdessamad Derraz
c9e2bf8d33 feat: generate_pack.py integrates data directory refresh and packing 2026-03-18 16:04:36 +01:00
Abdessamad Derraz
2afc31e40a feat: scraper-generated retroarch.yml with shared group conformance
retroarch.yml regenerated by libretro_scraper with CORE_SUBDIR_MAP
(dc/, np2kai/, keropi/) and shared groups (fuse, kronos, ep128emu,
quasi88, np2kai, keropi). common.py dedup by (name, destination)
to allow same file at flat + subdirectory paths.

ep128emu shared group added for Enterprise system.
RetroArch pack grows from 398 to 428 files.

ref: each subdirectory traced to original emulator source code —
see platforms/README.md and _shared.yml comments.
2026-03-18 14:41:00 +01:00
Abdessamad Derraz
76064605c0 fix: move zip_contents resolution after name-based lookup 2026-03-18 12:12:42 +01:00
Abdessamad Derraz
08f68e792d refactor: centralize hash logic, fix circular imports and perf bottlenecks 2026-03-18 11:51:12 +01:00
Abdessamad Derraz
54c0f1d27e refactor: review fixes, DRY coverage, filter test nav
- Extract compute_coverage to common.py (was duplicated)
- Filter test cores from nav and emulator index
- Use absolute URL for README download links
- Consistent page titles with site name suffix
- Safer mkdocs.yml nav rewrite with regex
- Build all_platform_names once in gap analysis
2026-03-18 11:05:13 +01:00
Abdessamad Derraz
00700609d8 refactor: extract resolve_local_file to common.py (DRY)
Single source of truth for file resolution logic:
- common.py:resolve_local_file() = 80 lines (core resolution)
- verify.py:resolve_to_local_path() = 3 lines (thin wrapper)
- generate_pack.py:resolve_file() = 20 lines (adds storage tiers + release assets)

Before: 103 + 73 = 176 lines of duplicated logic with subtle divergences
After: 80 lines shared + 23 lines wrappers = 103 lines total (-41%)

Resolution chain: SHA1 -> MD5 multi-hash -> truncated MD5 ->
zipped_file index -> name existence -> name composite -> name fallback
-> (pack only) release assets
2026-03-18 08:11:10 +01:00
Abdessamad Derraz
bb1855d3f7 feat: recalbox 346/346 via md5_composite, add mame variants
Add md5_composite() to verify.py replicating Recalbox Zip::Md5Composite
(sorted filenames, sequential content hash). Independent of ZIP
compression level, resolves all 9 MAME arcade untested entries.

Add Recalbox-specific MAME ZIP variants from Recalbox 10 pack.
Batocera 671/680 (9 untested MAME-specific), all others 100%.
2026-03-17 16:08:39 +01:00
Abdessamad Derraz
af74fffa14 refactor: fix code review findings across all scripts
Critical: stream large file downloads (OOM fix), fix basename match
in auto_fetch, include hashes in pack grouping fingerprint, handle
not_in_zip status in verify, fix escaped quotes in batocera parser.

Important: deduplicate shared group includes, catch coreinfo network
errors, fix NODEDUP path component match, fix CI word splitting on
spaces, replace bare except Exception in 3 files.

Minor: argparse in list_platforms, specific exceptions in download.py.
2026-03-17 15:16:51 +01:00
Abdessamad Derraz
3453f89d9d refactor: consolidate CI pipeline, remove third-party deps 2026-03-17 12:33:10 +01:00
Abdessamad Derraz
13c561888d v2: automated BIOS platform with full pipeline
Reorganized 6 branches into bios/Manufacturer/Console/.
Scrapers for RetroArch, Batocera, Recalbox, and libretro core-info.
Platform-aware verification replicating native logic per platform.
Pack generation with dedup, alias resolution, variant support.
CI/CD: weekly auto-scrape, auto-release, PR validation.
Large files (>50MB) stored as GitHub Release assets, auto-fetched at build time.
2026-03-17 10:54:39 +01:00