Commit Graph

61 Commits

Author SHA1 Message Date
Abdessamad Derraz
34e4c36f1c feat: pack integrity verification, manifests, SHA256SUMS
post-generation verification: reopen each ZIP, hash every file,
check against database.json. inject manifest.json inside each pack
(self-documenting: path, sha1, md5, size, status per file).
generate SHA256SUMS.txt alongside packs for download verification.

validation index now uses sets for hashes and sizes to support
multiple valid ROM versions (MT-32 v1.04-v2.07, CM-32L variants).
69 tests pass, pipeline complete.
2026-03-24 14:56:02 +01:00
Abdessamad Derraz
8141a34faa feat: full ground truth validation in verify.py
adler32 hash via zlib.adler32(), min_size/max_size range checks,
signature/crypto tracked as non-reproducible (console-specific keys).
compute_hashes now returns adler32. 69 tests pass including 3 new
tests for adler32, size ranges, and crypto tracking.
2026-03-24 11:11:38 +01:00
Abdessamad Derraz
1d350f0578 feat: add emulator/system pack generation, validation checks, path resolution
add --emulator, --system, --standalone, --list-emulators, --list-systems
to verify.py and generate_pack.py. packs are RTU with data directories,
regional BIOS variants, and archive support.

validation: field per file (size, crc32, md5, sha1) with conflict
detection. by_path_suffix index in database.json for regional variant
resolution via dest_hint. restructure GameCube IPL to regional subdirs.

66 E2E tests, full pipeline verified.
2026-03-22 14:02:20 +01:00
Abdessamad Derraz
6a21a99c22 feat: platform-core registry for exact pack generation
resolve_platform_cores() links platforms to their cores via
three strategies: all_libretro, explicit list, system ID
fallback. Pack generation always includes core requirements
beyond platform baseline. Case-insensitive dedup prevents
conflicts on Windows/macOS. Data dir strip_components fixes
doubled paths for Dolphin and PPSSPP caches.
2026-03-19 16:10:43 +01:00
Abdessamad Derraz
38d605c7d5 fix: audit fixes across verify, pack, security, and performance
- fix KeyError in compute_coverage (generate_readme, generate_site)
- fix comma-separated MD5 handling in generate_pack check_inside_zip
- fix _verify_file_hash to handle multi-MD5 for large files
- fix external downloads not tracked in seen_destinations/file_status
- fix tar path traversal in _is_safe_tar_member (refresh_data_dirs)
- fix predictable tmp path in download.py
- fix _sanitize_path to filter "." components
- remove blanket data_dir suppression in find_undeclared_files
- remove blanket data_dir suppression in cross_reference
- add status_counts to verify_platform return value
- add md5_composite cache for repeated ZIP hashing
2026-03-19 14:04:34 +01:00
Abdessamad Derraz
d5daf98e5e feat: hle_fallback field + launcher filtering in verify
Added hle_fallback: true/false per file in emulator profiles.
When a core has HLE and the file is missing, severity downgrades
to INFO instead of CRITICAL — core works without it.

verify.py builds an HLE index from emulator profiles and applies
it during severity computation. Cross-reference now skips launcher
profiles (type: launcher) and includes hle_fallback in undeclared
file reports.

33 E2E tests (4 new: HLE severity, HLE index, launcher skip,
cross-ref HLE). 0 regressions.

Based on source code analysis:
- RetroArch core_info.c:2233 — existence check only, no blocking
- PCSX ReARMed psxbios.c:28 — full HLE BIOS replacement
- Dolphin CommonPaths.h — all files optional with HLE
- snes9x — DSP HLE built-in, coprocessor files optional
2026-03-19 12:51:52 +01:00
Abdessamad Derraz
13e5dd37f4 chore: remove redundant tests, keep E2E only (29 tests)
Removed 7 test files (135 tests) fully covered by test_e2e.py.
The E2E creates all fixtures in setUp, exercises every code path
with real files and real functions. Single source of regression.
2026-03-19 12:22:56 +01:00
Abdessamad Derraz
eb270a030f feat: E2E regression test with unified YAML fixtures (164 total)
Single test class with shared setUp creating all synthetic fixtures:
- bios files with known hashes (present/missing/wrong/alias/variants)
- ZIPs with inner ROMs (correct/wrong/missing inner)
- platform YAMLs (existence, md5, inherited, shared groups)
- emulator profiles (aliases, standalone mode, data_directories)

29 E2E tests covering: resolution (9), verification (5), severity (2),
platform config (2), cross-reference (4), grouping (2), storage (2),
md5_composite (1), check_inside_zip (4).

One fixture set, all scenarios, easy to maintain.
2026-03-19 11:37:11 +01:00
Abdessamad Derraz
c8861192e6 feat: advanced tests — md5_composite, storage tiers, data dirs (135 total)
17 new tests covering remaining scenarios:
- md5_composite: manual calc, dirs excluded, compression-independent,
  used in resolve_local_file
- storage tiers: external, user_provided, embedded, fetch_large_file
  with cache hit and hash rejection
- data_directories: suppress cross-ref gaps when matched, show gaps
  when unmatched
- shared groups: _shared.yml includes injection, empty group safe
- YAML inheritance: systems inherited, verification_mode override
- platform grouping: identical merged, different base_dest separated
2026-03-19 11:34:30 +01:00
Abdessamad Derraz
06d39cedad feat: YAML integration tests with fixtures (35 new, 118 total)
tests/fixtures/ with synthetic platforms, emulators, bios files:
- test_existence.yml: required/optional present/missing
- test_md5.yml: correct/wrong hash, zipped_file (3 cases),
  multi-hash Recalbox, truncated MD5 Batocera, dedup
- test_inherit.yml: YAML inheritance
- _shared.yml: shared groups
- emulator profiles with aliases, standalone mode, alias type

tests/test_integration.py covers:
- existence/md5 mode severity counts
- zipped_file inner ROM verification (3 cases)
- Recalbox multi-hash + Batocera truncated MD5
- worst-status aggregation per destination
- YAML inheritance
- cross-reference: undeclared, standalone skipped, alias skipped
- alias resolution
- pack consistency (verify == pack counts)
- shared groups parsing

All tests use real files, real hashes, real functions, no mocking.
2026-03-19 11:29:00 +01:00
Abdessamad Derraz
b9cdda07ee refactor: DRY consolidation + 83 unit tests
Moved shared functions to common.py (single source of truth):
- check_inside_zip (was in verify.py, imported by generate_pack)
- build_zip_contents_index (was duplicated in verify + generate_pack)
- load_emulator_profiles (was in verify, cross_reference, generate_site)
- group_identical_platforms (was in verify + generate_pack)

Added tests/ with 83 unit tests covering:
- resolve_local_file: SHA1, MD5, name, alias, truncated, zip_contents
- verify: existence, md5, zipped_file, multi-hash, severity mapping
- aliases: field parsing, by_name indexing, beetle_psx field rename
- pack: dedup, file_status, zipped_file inner check, EmuDeck entries
- severity: all 12 combinations, platform-native behavior

0 regressions: pipeline.py --all produces identical results.
2026-03-19 11:19:50 +01:00