Commit Graph

38 Commits

Author SHA1 Message Date
Abdessamad Derraz
abca70cb5b feat: add generate_platform_truth to common.py 2026-03-29 11:22:35 +02:00
Abdessamad Derraz
c513d6c0ad feat: resolve_local_file data directory fallback 2026-03-29 11:08:31 +02:00
Abdessamad Derraz
536300984d feat: add howard.o Lynx development bootloader
513-byte BS93-format bootloader reconstructed from Handy
emulator source code analysis. Header verified: BRA +8,
load_addr 0x0212, magic BS93, SEI at code entry. CRam
reset simulation passed.
2026-03-29 09:18:30 +02:00
Abdessamad Derraz
080cc8f19d fix: manifest dest paths relative to bios dir 2026-03-28 18:42:19 +01:00
Abdessamad Derraz
46426aa1e4 feat: add install.py universal installer engine 2026-03-28 18:07:28 +01:00
Abdessamad Derraz
5763ce731d feat: add --manifest mode to generate_pack 2026-03-28 18:02:54 +01:00
Abdessamad Derraz
c6a24446ba feat: add install metadata to platform registry 2026-03-28 17:58:05 +01:00
Abdessamad Derraz
7dc8428ac1 refactor: fix cross-reference archive grouping and path resolution
Group archived files by archive unit in find_undeclared_files instead
of reporting individual ROMs. Add path-based fallback for descriptive
names (e.g. "SeaBIOS (128 KB)" resolves via path: bios.bin). Update
_collect_extras to use archive name for pack resolution. Regenerate
database with new bios files. 6 new E2E tests covering archive
in_repo, missing archives, descriptive names, and pack extras.
2026-03-28 14:00:08 +01:00
Abdessamad Derraz
543cb504bb fix: skip file/directory path conflicts in pack generation 2026-03-28 12:57:30 +01:00
Abdessamad Derraz
2738a4d326 feat: add bizhawk scraper for FirmwareDatabase.cs 2026-03-28 09:41:52 +01:00
Abdessamad Derraz
b75f2b2a43 feat: add sha1 verification mode for bizhawk 2026-03-28 09:35:13 +01:00
Abdessamad Derraz
0624e9d87e test: add validation tests for pack arg combos 2026-03-28 00:59:02 +01:00
Abdessamad Derraz
3ded72f72b feat: add --group-by manufacturer for split packs 2026-03-28 00:45:12 +01:00
Abdessamad Derraz
94a28f5459 feat: add --split flag for per-system packs 2026-03-28 00:43:20 +01:00
Abdessamad Derraz
43cb7a9884 feat: allow --platform + --system combination 2026-03-28 00:36:51 +01:00
Abdessamad Derraz
020ff148c2 feat: add --required-only flag to generate_pack 2026-03-28 00:32:58 +01:00
Abdessamad Derraz
6cbdd4c40c test: add integration tests for ground truth full chain 2026-03-27 23:41:06 +01:00
Abdessamad Derraz
37acc8d0fc feat: add --verbose flag and ground truth rendering 2026-03-27 23:38:43 +01:00
Abdessamad Derraz
2cf1398786 feat: attach ground truth to emulator verification results 2026-03-27 23:33:53 +01:00
Abdessamad Derraz
6b14b5e2b1 feat: attach ground truth to platform verification results 2026-03-27 23:30:49 +01:00
Abdessamad Derraz
6d959ff2b0 feat: add per-emulator ground truth to validation index 2026-03-27 23:25:42 +01:00
Abdessamad Derraz
1c34790737 feat: propagate target_cores through find_undeclared_files, find_exclusion_notes, verify_platform, _collect_emulator_extras 2026-03-26 08:44:44 +01:00
Abdessamad Derraz
e17d771710 feat: add target_cores filter to resolve_platform_cores
Optional target_cores parameter intersects the resolved core set,
enabling per-target filtering without changing existing call sites.
Includes 2 E2E tests covering intersection and None pass-through.
2026-03-26 08:42:08 +01:00
Abdessamad Derraz
0549b8945e feat: add load_target_config and list_available_targets to common.py
Loads per-platform target YAML files from platforms/targets/,
resolves aliases and applies add_cores/remove_cores from _overrides.yml.
Includes 7 E2E tests covering alias resolution, overrides, and error paths.
2026-03-26 08:41:37 +01:00
Abdessamad Derraz
3f676b75e8 feat: standalone emulator support for batocera and multi-platform name mapping
resolve_platform_cores() builds reverse index from profile cores: field,
fixing 17 name mismatches across Batocera, RetroBat, and Recalbox
(genesisplusgx, pce_fast, pcfx, vb, mame078plus, vice cores, etc.).

standalone_path field on file entries + standalone_cores on platform
YAMLs enable mode-aware pack generation. find_undeclared_files() uses
standalone_path for cores the platform runs standalone, filters by
mode: libretro/standalone per file.

batocera.yml gains standalone_cores (92 entries from configgen-defaults).
generate_readme.py dynamically lists platforms from registry.
3 profiles updated for standalone type/path (mame, hatari, mupen64plus_next).
78 E2E tests pass, pipeline verified.
2026-03-26 00:44:21 +01:00
Abdessamad Derraz
69131f4ad1 fix: emulator validation is informational, not a platform failure 2026-03-25 17:34:56 +01:00
Abdessamad Derraz
0543165ed2 feat: re-profile 22 emulators, refactor validation to common.py
batch re-profiled nekop2 through pokemini. mupen64plus renamed to
mupen64plus_next. new profiles: nes, mupen64plus_next.
validation functions (_build_validation_index, check_file_validation)
consolidated in common.py — single source of truth for verify.py
and generate_pack.py. pipeline 100% consistent on all 6 platforms.
2026-03-24 22:31:22 +01:00
Abdessamad Derraz
34e4c36f1c feat: pack integrity verification, manifests, SHA256SUMS
post-generation verification: reopen each ZIP, hash every file,
check against database.json. inject manifest.json inside each pack
(self-documenting: path, sha1, md5, size, status per file).
generate SHA256SUMS.txt alongside packs for download verification.

validation index now uses sets for hashes and sizes to support
multiple valid ROM versions (MT-32 v1.04-v2.07, CM-32L variants).
69 tests pass, pipeline complete.
2026-03-24 14:56:02 +01:00
Abdessamad Derraz
8141a34faa feat: full ground truth validation in verify.py
adler32 hash via zlib.adler32(), min_size/max_size range checks,
signature/crypto tracked as non-reproducible (console-specific keys).
compute_hashes now returns adler32. 69 tests pass including 3 new
tests for adler32, size ranges, and crypto tracking.
2026-03-24 11:11:38 +01:00
Abdessamad Derraz
1d350f0578 feat: add emulator/system pack generation, validation checks, path resolution
add --emulator, --system, --standalone, --list-emulators, --list-systems
to verify.py and generate_pack.py. packs are RTU with data directories,
regional BIOS variants, and archive support.

validation: field per file (size, crc32, md5, sha1) with conflict
detection. by_path_suffix index in database.json for regional variant
resolution via dest_hint. restructure GameCube IPL to regional subdirs.

66 E2E tests, full pipeline verified.
2026-03-22 14:02:20 +01:00
Abdessamad Derraz
6a21a99c22 feat: platform-core registry for exact pack generation
resolve_platform_cores() links platforms to their cores via
three strategies: all_libretro, explicit list, system ID
fallback. Pack generation always includes core requirements
beyond platform baseline. Case-insensitive dedup prevents
conflicts on Windows/macOS. Data dir strip_components fixes
doubled paths for Dolphin and PPSSPP caches.
2026-03-19 16:10:43 +01:00
Abdessamad Derraz
38d605c7d5 fix: audit fixes across verify, pack, security, and performance
- fix KeyError in compute_coverage (generate_readme, generate_site)
- fix comma-separated MD5 handling in generate_pack check_inside_zip
- fix _verify_file_hash to handle multi-MD5 for large files
- fix external downloads not tracked in seen_destinations/file_status
- fix tar path traversal in _is_safe_tar_member (refresh_data_dirs)
- fix predictable tmp path in download.py
- fix _sanitize_path to filter "." components
- remove blanket data_dir suppression in find_undeclared_files
- remove blanket data_dir suppression in cross_reference
- add status_counts to verify_platform return value
- add md5_composite cache for repeated ZIP hashing
2026-03-19 14:04:34 +01:00
Abdessamad Derraz
d5daf98e5e feat: hle_fallback field + launcher filtering in verify
Added hle_fallback: true/false per file in emulator profiles.
When a core has HLE and the file is missing, severity downgrades
to INFO instead of CRITICAL — core works without it.

verify.py builds an HLE index from emulator profiles and applies
it during severity computation. Cross-reference now skips launcher
profiles (type: launcher) and includes hle_fallback in undeclared
file reports.

33 E2E tests (4 new: HLE severity, HLE index, launcher skip,
cross-ref HLE). 0 regressions.

Based on source code analysis:
- RetroArch core_info.c:2233 — existence check only, no blocking
- PCSX ReARMed psxbios.c:28 — full HLE BIOS replacement
- Dolphin CommonPaths.h — all files optional with HLE
- snes9x — DSP HLE built-in, coprocessor files optional
2026-03-19 12:51:52 +01:00
Abdessamad Derraz
13e5dd37f4 chore: remove redundant tests, keep E2E only (29 tests)
Removed 7 test files (135 tests) fully covered by test_e2e.py.
The E2E creates all fixtures in setUp, exercises every code path
with real files and real functions. Single source of regression.
2026-03-19 12:22:56 +01:00
Abdessamad Derraz
eb270a030f feat: E2E regression test with unified YAML fixtures (164 total)
Single test class with shared setUp creating all synthetic fixtures:
- bios files with known hashes (present/missing/wrong/alias/variants)
- ZIPs with inner ROMs (correct/wrong/missing inner)
- platform YAMLs (existence, md5, inherited, shared groups)
- emulator profiles (aliases, standalone mode, data_directories)

29 E2E tests covering: resolution (9), verification (5), severity (2),
platform config (2), cross-reference (4), grouping (2), storage (2),
md5_composite (1), check_inside_zip (4).

One fixture set, all scenarios, easy to maintain.
2026-03-19 11:37:11 +01:00
Abdessamad Derraz
c8861192e6 feat: advanced tests — md5_composite, storage tiers, data dirs (135 total)
17 new tests covering remaining scenarios:
- md5_composite: manual calc, dirs excluded, compression-independent,
  used in resolve_local_file
- storage tiers: external, user_provided, embedded, fetch_large_file
  with cache hit and hash rejection
- data_directories: suppress cross-ref gaps when matched, show gaps
  when unmatched
- shared groups: _shared.yml includes injection, empty group safe
- YAML inheritance: systems inherited, verification_mode override
- platform grouping: identical merged, different base_dest separated
2026-03-19 11:34:30 +01:00
Abdessamad Derraz
06d39cedad feat: YAML integration tests with fixtures (35 new, 118 total)
tests/fixtures/ with synthetic platforms, emulators, bios files:
- test_existence.yml: required/optional present/missing
- test_md5.yml: correct/wrong hash, zipped_file (3 cases),
  multi-hash Recalbox, truncated MD5 Batocera, dedup
- test_inherit.yml: YAML inheritance
- _shared.yml: shared groups
- emulator profiles with aliases, standalone mode, alias type

tests/test_integration.py covers:
- existence/md5 mode severity counts
- zipped_file inner ROM verification (3 cases)
- Recalbox multi-hash + Batocera truncated MD5
- worst-status aggregation per destination
- YAML inheritance
- cross-reference: undeclared, standalone skipped, alias skipped
- alias resolution
- pack consistency (verify == pack counts)
- shared groups parsing

All tests use real files, real hashes, real functions, no mocking.
2026-03-19 11:29:00 +01:00
Abdessamad Derraz
b9cdda07ee refactor: DRY consolidation + 83 unit tests
Moved shared functions to common.py (single source of truth):
- check_inside_zip (was in verify.py, imported by generate_pack)
- build_zip_contents_index (was duplicated in verify + generate_pack)
- load_emulator_profiles (was in verify, cross_reference, generate_site)
- group_identical_platforms (was in verify + generate_pack)

Added tests/ with 83 unit tests covering:
- resolve_local_file: SHA1, MD5, name, alias, truncated, zip_contents
- verify: existence, md5, zipped_file, multi-hash, severity mapping
- aliases: field parsing, by_name indexing, beetle_psx field rename
- pack: dedup, file_status, zipped_file inner check, EmuDeck entries
- severity: all 12 combinations, platform-native behavior

0 regressions: pipeline.py --all produces identical results.
2026-03-19 11:19:50 +01:00