Commit Graph
75 Commits
Author SHA1 Message Date
Abdessamad Derraz ecbe69760d refactor: check packs through a single path 2026-08-11 11:31:21 +02:00
Abdessamad Derraz d8e4325af2 fix: skip conformance on required-only packs
A required-only build is narrower than the platform declares by
design, like the source-restricted variants already handled, so the
full expectation must not be applied to it. --region is also refused
alongside --manifest-targets, which carries no region dimension.
2026-08-11 05:34:21 +02:00
Abdessamad Derraz 367d4efaac feat: tag filtered packs by region and target
A region- or target-filtered build wrote to the same filename as the
full one. Both filters now appear in the pack and manifest names, and
verify.py accepts --region for emulator and system reports so the
coverage figures come from the same selection the builder used.
2026-08-11 00:56:14 +02:00
Abdessamad Derraz ba84f5701f chore: cover region filter with split and target 2026-08-10 16:54:39 +02:00
Abdessamad Derraz e8ee8b0954 fix: decide hash mismatch by native mode
A declared hash that the local dump contradicts is not one situation. An
existence platform never reads the bytes, so withholding the file lets an
upstream list error remove something the frontend would have loaded; a
hash platform would reject it, so shipping it is pointless.

The mode now decides, at every point that had an opinion: pack building,
core complement, emulator packs, manifests, conformance and
_intentional_hash_exclusion. verify.find_undeclared_files follows, since
verify and generate_pack must agree file for file.

Also here: resolution reports which evidence matched rather than a flat
"exact", a path or filename can no longer override a declared hash, and
safe_extract_zip treats a Windows backslash as the separator it is
instead of refusing the archive.
2026-08-10 13:36:03 +02:00
Abdessamad Derraz 45f89cc6c0 feat: add region filters and profiles 2026-08-09 14:15:56 +02:00
Abdessamad Derraz 25f8537a4a fix: point manifest entries at a fetchable file 2026-08-08 06:19:38 +02:00
Abdessamad Derraz 8bd2083e80 feat: report hash proof in existence packs 2026-08-07 16:23:15 +02:00
Abdessamad Derraz 3ecb72d275 fix: deflate manifest on zip append path 2026-08-07 15:45:48 +02:00
Abdessamad Derraz 587aadde08 fix: coerce numeric hashes to strings 2026-08-07 15:30:37 +02:00
Abdessamad Derraz 18c7b31734 fix: honor platform filter in cross-reference 2026-08-07 13:28:15 +02:00
Abdessamad Derraz 8663ec17b7 feat: add misterfpga platform support 2026-08-07 13:28:15 +02:00
Abdessamad Derraz 3a5cbc4c6c fix: resolve large file assets renamed by github 2026-08-07 11:53:29 +02:00
Abdessamad Derraz 792bad3fd5 fix: align pack checks with builder, purge stale packs 2026-08-06 16:41:47 +02:00
Abdessamad Derraz 69b8ab1434 feat: resolve files by sha256 index 2026-08-06 10:39:58 +02:00
Abdessamad Derraz 36b9b59e2e fix: match pack zips to declared platform hashes 2026-08-06 05:00:50 +02:00
Abdessamad Derraz 617f5cc364 refactor: hash-anchor truth enrichment 2026-04-18 09:00:31 +02:00
Abdessamad Derraz 06c48e071a test: update grouping test for flat zip behavior 2026-04-03 12:06:38 +02:00
Abdessamad Derraz 9ba8b02ff1 fix: verify functions handle flat zip extraction 2026-04-03 11:50:26 +02:00
Abdessamad Derraz 0401d058a1 feat: add by_sha256 index, fix reporting attribution
generate_db: add by_sha256 index for O(1) variant lookup.
verify: _find_best_variant uses indexed sha256 instead of O(n) scan.
validation: check_file_validation returns (reason, emulators) tuple,
attributing mismatch only to emulators whose check actually failed.
beetle_psx: remove incorrect size field for ps1_rom.bin (code does
not validate size, swanstation is sole size authority).
2026-04-02 00:59:01 +02:00
Abdessamad Derraz 2c1c2a7bfe feat: add combined source+required variant tests 2026-04-01 14:42:52 +02:00
Abdessamad Derraz 423a1b201e feat: add source param to generate_pack 2026-04-01 14:39:04 +02:00
Abdessamad Derraz 9c6b3dfe96 feat: add include_all to _collect_emulator_extras 2026-04-01 14:33:54 +02:00
Abdessamad Derraz b070fa41de feat: add include_all param to find_undeclared_files 2026-04-01 14:29:31 +02:00
Abdessamad Derraz 0a272dc4e9 chore: lint and format entire codebase
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).

Run isort --profile black: normalize import ordering across all files.

Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.

3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
2026-04-01 13:17:55 +02:00
Abdessamad Derraz 75e34898ee feat: add MAME/FBNeo hash auto-fetch scrapers
sparse clone upstream repos, parse BIOS root sets from C source,
cache as JSON, merge into emulator profiles with backup.
covers macro expansion, version detection, subset profile protection.
2026-03-30 19:11:26 +02:00
Abdessamad Derraz 74269bab84 fix: rewrite exporters to match exact native formats 2026-03-30 15:49:33 +02:00
Abdessamad Derraz 1e6b499602 feat: add batocera, recalbox, retrobat native exporters 2026-03-30 15:31:44 +02:00
Abdessamad Derraz 17777f315b feat: agnostic bios mode for filename-agnostic emulators
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
2026-03-30 14:18:54 +02:00
Abdessamad Derraz 903c49edcf feat: add tests for registry merge, all_libretro expansion, hash fallback, system normalization 2026-03-30 11:33:59 +02:00
Abdessamad Derraz 54022e9db1 feat: hash-based matching for cross-reference
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
2026-03-30 08:25:54 +02:00
Abdessamad Derraz 2e21d64a08 refactor: harden codebase and remove unicode artifacts
- fix urllib.parse.quote import (was urllib.request.quote)
- add operator precedence parens in generate_pack dedup check
- narrow bare except to specific types in batocera target scraper
- cache load_platform_config and build_zip_contents_index results
- add selective algorithm support to compute_hashes
- atomic write for fetch_large_file (tmp + rename)
- add response size limit to base scraper fetch
- extract build_target_cores_cache to common.py (dedup verify/pack)
- hoist _build_supplemental_index out of per-platform loop
- migrate function-attribute caches to module-level dicts
- add @abstractmethod to BaseTargetScraper.fetch_targets
- remove backward-compat re-exports from common.py
- replace em-dashes and unicode arrows with ASCII equivalents
- remove decorative section dividers and obvious comments
2026-03-29 23:15:20 +02:00
Abdessamad Derraz 0c5cde83e1 Add TRS-80, RX-78, Sega AI entries; refactor tools
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
2026-03-29 23:04:30 +02:00
Abdessamad Derraz 3c7fc26354 refactor: extract validation and truth modules from common.py 2026-03-29 16:41:24 +02:00
Abdessamad Derraz 13b20c2742 fix: normalize system IDs in diff matching 2026-03-29 14:07:14 +02:00
Abdessamad Derraz dee37c2530 fix: truth generation uses platform config not registry 2026-03-29 13:41:49 +02:00
Abdessamad Derraz 3bf6e5c961 feat: add truth and diff integration test 2026-03-29 13:27:32 +02:00
Abdessamad Derraz e86d8d68af feat: add exporter plugin architecture 2026-03-29 13:19:38 +02:00
Abdessamad Derraz 2ce8db1754 feat: add native_id to scraper output 2026-03-29 13:15:57 +02:00
Abdessamad Derraz 2aab7420d7 feat: add diff_platform_truth function and tests 2026-03-29 13:09:08 +02:00
Abdessamad Derraz 525f51d5fb feat: add truth mode filtering and dedup tests 2026-03-29 13:01:37 +02:00
Abdessamad Derraz abca70cb5b feat: add generate_platform_truth to common.py 2026-03-29 11:22:35 +02:00
Abdessamad Derraz c513d6c0ad feat: resolve_local_file data directory fallback 2026-03-29 11:08:31 +02:00
Abdessamad Derraz 536300984d feat: add howard.o Lynx development bootloader
513-byte BS93-format bootloader reconstructed from Handy
emulator source code analysis. Header verified: BRA +8,
load_addr 0x0212, magic BS93, SEI at code entry. CRam
reset simulation passed.
2026-03-29 09:18:30 +02:00
Abdessamad Derraz 080cc8f19d fix: manifest dest paths relative to bios dir 2026-03-28 18:42:19 +01:00
Abdessamad Derraz 46426aa1e4 feat: add install.py universal installer engine 2026-03-28 18:07:28 +01:00
Abdessamad Derraz 5763ce731d feat: add --manifest mode to generate_pack 2026-03-28 18:02:54 +01:00
Abdessamad Derraz c6a24446ba feat: add install metadata to platform registry 2026-03-28 17:58:05 +01:00
Abdessamad Derraz 7dc8428ac1 refactor: fix cross-reference archive grouping and path resolution
Group archived files by archive unit in find_undeclared_files instead
of reporting individual ROMs. Add path-based fallback for descriptive
names (e.g. "SeaBIOS (128 KB)" resolves via path: bios.bin). Update
_collect_extras to use archive name for pack resolution. Regenerate
database with new bios files. 6 new E2E tests covering archive
in_repo, missing archives, descriptive names, and pack extras.
2026-03-28 14:00:08 +01:00
Abdessamad Derraz 543cb504bb fix: skip file/directory path conflicts in pack generation 2026-03-28 12:57:30 +01:00