Commit Graph

64 Commits

Author SHA1 Message Date
Abdessamad Derraz
2c1c2a7bfe feat: add combined source+required variant tests 2026-04-01 14:42:52 +02:00
Abdessamad Derraz
423a1b201e feat: add source param to generate_pack 2026-04-01 14:39:04 +02:00
Abdessamad Derraz
9c6b3dfe96 feat: add include_all to _collect_emulator_extras 2026-04-01 14:33:54 +02:00
Abdessamad Derraz
b070fa41de feat: add include_all param to find_undeclared_files 2026-04-01 14:29:31 +02:00
Abdessamad Derraz
0a272dc4e9 chore: lint and format entire codebase
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).

Run isort --profile black: normalize import ordering across all files.

Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.

3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
2026-04-01 13:17:55 +02:00
Abdessamad Derraz
e5859eb761 refactor: dry pack integrity into cli and update docs
Move verification logic to generate_pack.py --verify-packs (single
source of truth). test_pack_integrity.py is now a thin wrapper that
calls the CLI. Pipeline step 6/8 uses the same CLI entry point.

Renumber all pipeline steps 1-8 (was skipping from 5 to 8/9).

Update generate_site.py with pack integrity test documentation.
2026-04-01 12:31:10 +02:00
Abdessamad Derraz
754e829b35 feat: add pack integrity test and integrate into pipeline
Extract each platform ZIP to tmp/ (real filesystem, not /tmp tmpfs)
and verify every declared file exists at the correct path with the
correct hash per the platform's native verification mode.

Handles ZIP inner content verification (checkInsideZip, md5_composite,
inner ROM MD5) and path collision deduplication.

Integrated as pipeline step 6/8. Renumber all pipeline steps to be
sequential (was skipping from 5 to 8).
2026-04-01 12:22:50 +02:00
Abdessamad Derraz
75e34898ee feat: add MAME/FBNeo hash auto-fetch scrapers
sparse clone upstream repos, parse BIOS root sets from C source,
cache as JSON, merge into emulator profiles with backup.
covers macro expansion, version detection, subset profile protection.
2026-03-30 19:11:26 +02:00
Abdessamad Derraz
94c3ac9834 feat: add hash merge for MAME and FBNeo profiles 2026-03-30 18:33:09 +02:00
Abdessamad Derraz
319a1d2041 feat: add MAME source code parser for BIOS root sets 2026-03-30 18:29:31 +02:00
Abdessamad Derraz
00d7b57884 feat: add FBNeo source parser for BIOS sets 2026-03-30 18:29:06 +02:00
Abdessamad Derraz
74269bab84 fix: rewrite exporters to match exact native formats 2026-03-30 15:49:33 +02:00
Abdessamad Derraz
1e6b499602 feat: add batocera, recalbox, retrobat native exporters 2026-03-30 15:31:44 +02:00
Abdessamad Derraz
17777f315b feat: agnostic bios mode for filename-agnostic emulators
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
2026-03-30 14:18:54 +02:00
Abdessamad Derraz
903c49edcf feat: add tests for registry merge, all_libretro expansion, hash fallback, system normalization 2026-03-30 11:33:59 +02:00
Abdessamad Derraz
54022e9db1 feat: hash-based matching for cross-reference
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
2026-03-30 08:25:54 +02:00
Abdessamad Derraz
2e21d64a08 refactor: harden codebase and remove unicode artifacts
- fix urllib.parse.quote import (was urllib.request.quote)
- add operator precedence parens in generate_pack dedup check
- narrow bare except to specific types in batocera target scraper
- cache load_platform_config and build_zip_contents_index results
- add selective algorithm support to compute_hashes
- atomic write for fetch_large_file (tmp + rename)
- add response size limit to base scraper fetch
- extract build_target_cores_cache to common.py (dedup verify/pack)
- hoist _build_supplemental_index out of per-platform loop
- migrate function-attribute caches to module-level dicts
- add @abstractmethod to BaseTargetScraper.fetch_targets
- remove backward-compat re-exports from common.py
- replace em-dashes and unicode arrows with ASCII equivalents
- remove decorative section dividers and obvious comments
2026-03-29 23:15:20 +02:00
Abdessamad Derraz
0c5cde83e1 Add TRS-80, RX-78, Sega AI entries; refactor tools
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
2026-03-29 23:04:30 +02:00
Abdessamad Derraz
3c7fc26354 refactor: extract validation and truth modules from common.py 2026-03-29 16:41:24 +02:00
Abdessamad Derraz
13b20c2742 fix: normalize system IDs in diff matching 2026-03-29 14:07:14 +02:00
Abdessamad Derraz
dee37c2530 fix: truth generation uses platform config not registry 2026-03-29 13:41:49 +02:00
Abdessamad Derraz
3bf6e5c961 feat: add truth and diff integration test 2026-03-29 13:27:32 +02:00
Abdessamad Derraz
e86d8d68af feat: add exporter plugin architecture 2026-03-29 13:19:38 +02:00
Abdessamad Derraz
2ce8db1754 feat: add native_id to scraper output 2026-03-29 13:15:57 +02:00
Abdessamad Derraz
2aab7420d7 feat: add diff_platform_truth function and tests 2026-03-29 13:09:08 +02:00
Abdessamad Derraz
525f51d5fb feat: add truth mode filtering and dedup tests 2026-03-29 13:01:37 +02:00
Abdessamad Derraz
abca70cb5b feat: add generate_platform_truth to common.py 2026-03-29 11:22:35 +02:00
Abdessamad Derraz
c513d6c0ad feat: resolve_local_file data directory fallback 2026-03-29 11:08:31 +02:00
Abdessamad Derraz
536300984d feat: add howard.o Lynx development bootloader
513-byte BS93-format bootloader reconstructed from Handy
emulator source code analysis. Header verified: BRA +8,
load_addr 0x0212, magic BS93, SEI at code entry. CRam
reset simulation passed.
2026-03-29 09:18:30 +02:00
Abdessamad Derraz
080cc8f19d fix: manifest dest paths relative to bios dir 2026-03-28 18:42:19 +01:00
Abdessamad Derraz
46426aa1e4 feat: add install.py universal installer engine 2026-03-28 18:07:28 +01:00
Abdessamad Derraz
5763ce731d feat: add --manifest mode to generate_pack 2026-03-28 18:02:54 +01:00
Abdessamad Derraz
c6a24446ba feat: add install metadata to platform registry 2026-03-28 17:58:05 +01:00
Abdessamad Derraz
7dc8428ac1 refactor: fix cross-reference archive grouping and path resolution
Group archived files by archive unit in find_undeclared_files instead
of reporting individual ROMs. Add path-based fallback for descriptive
names (e.g. "SeaBIOS (128 KB)" resolves via path: bios.bin). Update
_collect_extras to use archive name for pack resolution. Regenerate
database with new bios files. 6 new E2E tests covering archive
in_repo, missing archives, descriptive names, and pack extras.
2026-03-28 14:00:08 +01:00
Abdessamad Derraz
543cb504bb fix: skip file/directory path conflicts in pack generation 2026-03-28 12:57:30 +01:00
Abdessamad Derraz
2738a4d326 feat: add bizhawk scraper for FirmwareDatabase.cs 2026-03-28 09:41:52 +01:00
Abdessamad Derraz
b75f2b2a43 feat: add sha1 verification mode for bizhawk 2026-03-28 09:35:13 +01:00
Abdessamad Derraz
0624e9d87e test: add validation tests for pack arg combos 2026-03-28 00:59:02 +01:00
Abdessamad Derraz
3ded72f72b feat: add --group-by manufacturer for split packs 2026-03-28 00:45:12 +01:00
Abdessamad Derraz
94a28f5459 feat: add --split flag for per-system packs 2026-03-28 00:43:20 +01:00
Abdessamad Derraz
43cb7a9884 feat: allow --platform + --system combination 2026-03-28 00:36:51 +01:00
Abdessamad Derraz
020ff148c2 feat: add --required-only flag to generate_pack 2026-03-28 00:32:58 +01:00
Abdessamad Derraz
6cbdd4c40c test: add integration tests for ground truth full chain 2026-03-27 23:41:06 +01:00
Abdessamad Derraz
37acc8d0fc feat: add --verbose flag and ground truth rendering 2026-03-27 23:38:43 +01:00
Abdessamad Derraz
2cf1398786 feat: attach ground truth to emulator verification results 2026-03-27 23:33:53 +01:00
Abdessamad Derraz
6b14b5e2b1 feat: attach ground truth to platform verification results 2026-03-27 23:30:49 +01:00
Abdessamad Derraz
6d959ff2b0 feat: add per-emulator ground truth to validation index 2026-03-27 23:25:42 +01:00
Abdessamad Derraz
1c34790737 feat: propagate target_cores through find_undeclared_files, find_exclusion_notes, verify_platform, _collect_emulator_extras 2026-03-26 08:44:44 +01:00
Abdessamad Derraz
e17d771710 feat: add target_cores filter to resolve_platform_cores
Optional target_cores parameter intersects the resolved core set,
enabling per-target filtering without changing existing call sites.
Includes 2 E2E tests covering intersection and None pass-through.
2026-03-26 08:42:08 +01:00
Abdessamad Derraz
0549b8945e feat: add load_target_config and list_available_targets to common.py
Loads per-platform target YAML files from platforms/targets/,
resolves aliases and applies add_cores/remove_cores from _overrides.yml.
Includes 7 E2E tests covering alias resolution, overrides, and error paths.
2026-03-26 08:41:37 +01:00