Commit Graph

88 Commits

Author SHA1 Message Date
Abdessamad Derraz
0a272dc4e9 chore: lint and format entire codebase
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).

Run isort --profile black: normalize import ordering across all files.

Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.

3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
2026-04-01 13:17:55 +02:00
Abdessamad Derraz
e5859eb761 refactor: dry pack integrity into cli and update docs
Move verification logic to generate_pack.py --verify-packs (single
source of truth). test_pack_integrity.py is now a thin wrapper that
calls the CLI. Pipeline step 6/8 uses the same CLI entry point.

Renumber all pipeline steps 1-8 (was skipping from 5 to 8/9).

Update generate_site.py with pack integrity test documentation.
2026-04-01 12:31:10 +02:00
Abdessamad Derraz
7beb651049 fix: correct core extras placement for retrodeck and romm packs
RetroDECK: core extras with subdirectory paths (e.g. vice/C64/,
fbneo/, dc/) were placed outside bios/ because the prefix was only
inferred for bare filenames. Add _detect_extras_prefix() to infer
the dominant BIOS prefix from YAML destinations.

RomM: core extras landed flat at bios/{file} instead of the required
bios/{platform_slug}/{file}. Add _detect_slug_structure() to detect
per-system slug layouts and _map_emulator_to_slug() to route each
extra to the correct slug subfolder.

Also skip manifest writes when only the generated timestamp changed,
preventing unnecessary diffs in install/*.json.
2026-04-01 11:08:01 +02:00
Abdessamad Derraz
6b5c3d8bf2 fix: pack conformance matches builder logic for perfect stats 2026-03-31 12:25:07 +02:00
Abdessamad Derraz
b56f8dd05f feat: add archive_prefix for core-specific BIOS subdirectories
Closes #43

FBNeo and Kronos expect BIOS archives in core-specific subdirectories
(system/fbneo/, system/kronos/). RetroArch firmware check uses .info
paths which include these prefixes, so files at root show as Missing.

Add archive_prefix field to emulator profiles. The pack code now places
archive copies in the prefixed subdirectory while keeping root copies
for cores that expect them there (e.g. Geolith for neogeo.zip).
2026-03-31 09:17:54 +02:00
Abdessamad Derraz
17777f315b feat: agnostic bios mode for filename-agnostic emulators
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
2026-03-30 14:18:54 +02:00
Abdessamad Derraz
54022e9db1 feat: hash-based matching for cross-reference
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
2026-03-30 08:25:54 +02:00
Abdessamad Derraz
2e21d64a08 refactor: harden codebase and remove unicode artifacts
- fix urllib.parse.quote import (was urllib.request.quote)
- add operator precedence parens in generate_pack dedup check
- narrow bare except to specific types in batocera target scraper
- cache load_platform_config and build_zip_contents_index results
- add selective algorithm support to compute_hashes
- atomic write for fetch_large_file (tmp + rename)
- add response size limit to base scraper fetch
- extract build_target_cores_cache to common.py (dedup verify/pack)
- hoist _build_supplemental_index out of per-platform loop
- migrate function-attribute caches to module-level dicts
- add @abstractmethod to BaseTargetScraper.fetch_targets
- remove backward-compat re-exports from common.py
- replace em-dashes and unicode arrows with ASCII equivalents
- remove decorative section dividers and obvious comments
2026-03-29 23:15:20 +02:00
Abdessamad Derraz
0c5cde83e1 Add TRS-80, RX-78, Sega AI entries; refactor tools
Add many MAME/MESS BIOS entries (TRS-80 family, Bandai RX-78, Sega AI) and update docs/navigation counts (README, mkdocs). Remove empty supplemental file references from database.json and update generated timestamps and totals. Harden and refactor tooling: add MAX_RESPONSE_SIZE limited reader in base_scraper, make target scrapers an abstract base, narrow exception handling in the Batocera targets parser, and switch generate_pack.py and verify.py to use build_target_cores_cache (simplifies target config loading and error handling). verify.py also loads supplemental cross-reference names and accepts them through verify_platform. Update tests to import from updated modules (validation/truth). Misc: small bugfix for case-insensitive path conflict check.
2026-03-29 23:04:30 +02:00
Abdessamad Derraz
463fca7e7d Regenerate database and update emulator profiles
Regenerate database.json and update README counts/timestamps; add and normalize numerous BIOS entries and hashes. Key changes: update generated_at timestamp and system count (355→357) in README; add OpenBIOS / HLE fallback and additional aliases to beetle_psx, include beetle_psx core name and profiled_date update; add laseractive to ares systems; adjust atari800 systems and source_ref line numbers; mark dinothawr as a system and expand its note; update gsplus upstream/profile date, add apple-iie system and tweak source_refs; add pcsx2 core to lrps2; refresh mame profiled_date and add multiple systems and BIOS root sets. Miscellaneous script changes and other JSON normalization to reflect newly discovered/merged BIOS files.
2026-03-29 22:41:01 +02:00
Abdessamad Derraz
be68ed19a3 refactor: extract dispatch functions from generate_pack main() 2026-03-29 17:10:01 +02:00
Abdessamad Derraz
b4c5d77e4b refactor: deduplicate yaml import pattern via require_yaml() 2026-03-29 17:07:27 +02:00
Abdessamad Derraz
3c7fc26354 refactor: extract validation and truth modules from common.py 2026-03-29 16:41:24 +02:00
Abdessamad Derraz
ae71a41b32 feat: verify_pack checks data dirs and rebuilt ZIPs
verify_pack now verifies files against data directory caches and
validates rebuilt MAME ZIPs by comparing inner ROM CRC32s against
source. Reduces false untracked count from 6242 to 0 for RetroArch.
2026-03-29 14:04:46 +02:00
Abdessamad Derraz
97eb4835be feat: add load_from field for non-system_dir files
Replaces mode: standalone hack with load_from: save_dir on Panda3DS
files. The load_from field documents which libretro directory callback
provides the base path (system_dir default, save_dir, content_dir).
Pack generator and cross-reference skip files not targeting system_dir.
2026-03-29 13:07:30 +02:00
Abdessamad Derraz
d38f2b7180 fix: exclude save_dir paths from pack multi-dest 2026-03-29 12:02:30 +02:00
Abdessamad Derraz
c513d6c0ad feat: resolve_local_file data directory fallback 2026-03-29 11:08:31 +02:00
Abdessamad Derraz
536300984d feat: add howard.o Lynx development bootloader
513-byte BS93-format bootloader reconstructed from Handy
emulator source code analysis. Header verified: BRA +8,
load_addr 0x0212, magic BS93, SEI at code entry. CRam
reset simulation passed.
2026-03-29 09:18:30 +02:00
Abdessamad Derraz
9f229753b2 feat: platform-specific notes in zip readme 2026-03-28 19:27:49 +01:00
Abdessamad Derraz
0419c6a780 fix: generate manifest aliases for grouped platforms 2026-03-28 18:58:24 +01:00
Abdessamad Derraz
080cc8f19d fix: manifest dest paths relative to bios dir 2026-03-28 18:42:19 +01:00
Abdessamad Derraz
f453a7c750 feat: include readme.txt in pack zips 2026-03-28 18:11:37 +01:00
Abdessamad Derraz
7f46996b4d feat: add --manifest-targets for installer 2026-03-28 18:04:27 +01:00
Abdessamad Derraz
5763ce731d feat: add --manifest mode to generate_pack 2026-03-28 18:02:54 +01:00
Abdessamad Derraz
7dc8428ac1 refactor: fix cross-reference archive grouping and path resolution
Group archived files by archive unit in find_undeclared_files instead
of reporting individual ROMs. Add path-based fallback for descriptive
names (e.g. "SeaBIOS (128 KB)" resolves via path: bios.bin). Update
_collect_extras to use archive name for pack resolution. Regenerate
database with new bios files. 6 new E2E tests covering archive
in_repo, missing archives, descriptive names, and pack extras.
2026-03-28 14:00:08 +01:00
Abdessamad Derraz
543cb504bb fix: skip file/directory path conflicts in pack generation 2026-03-28 12:57:30 +01:00
Abdessamad Derraz
67186448a2 fix: verify and add manifests to split packs 2026-03-28 08:43:41 +01:00
Abdessamad Derraz
70891314d3 fix: include core extras in split packs 2026-03-28 08:29:37 +01:00
Abdessamad Derraz
a1aa97a70e fix: include core extras in split packs 2026-03-28 08:06:22 +01:00
Abdessamad Derraz
97b1c2c08a feat: add --from-md5 lookup and pack builder 2026-03-28 01:02:25 +01:00
Abdessamad Derraz
3ded72f72b feat: add --group-by manufacturer for split packs 2026-03-28 00:45:12 +01:00
Abdessamad Derraz
94a28f5459 feat: add --split flag for per-system packs 2026-03-28 00:43:20 +01:00
Abdessamad Derraz
837ac80cca refactor: deduplicate manufacturer prefix list 2026-03-28 00:39:28 +01:00
Abdessamad Derraz
43cb7a9884 feat: allow --platform + --system combination 2026-03-28 00:36:51 +01:00
Abdessamad Derraz
020ff148c2 feat: add --required-only flag to generate_pack 2026-03-28 00:32:58 +01:00
Abdessamad Derraz
7f265b3cb2 refactor: split pack check into baseline and cores with clear counts 2026-03-27 20:37:48 +01:00
Abdessamad Derraz
3ea1e09cb0 feat: verify core extras presence in pack alongside baseline 2026-03-27 19:41:47 +01:00
Abdessamad Derraz
89d6dd2eee feat: add platform conformance check to pack verification 2026-03-27 19:21:29 +01:00
Abdessamad Derraz
181248b6db fix: case-sensitive packs for linux platforms, remove empty bios placeholder 2026-03-27 12:58:08 +01:00
Abdessamad Derraz
a117b13b49 fix: data directory path construction avoids double slash duplicates 2026-03-27 12:42:21 +01:00
Abdessamad Derraz
0a1880f606 fix: filter baseline by platform-scoped cores, include retroarch cores in emudeck targets 2026-03-26 10:20:43 +01:00
Abdessamad Derraz
6402b77374 fix: filter baseline systems by target-available cores 2026-03-26 09:54:28 +01:00
Abdessamad Derraz
5ac14079d6 feat: add --target and --list-targets to generate_pack.py 2026-03-26 08:48:31 +01:00
Abdessamad Derraz
1c34790737 feat: propagate target_cores through find_undeclared_files, find_exclusion_notes, verify_platform, _collect_emulator_extras 2026-03-26 08:44:44 +01:00
Abdessamad Derraz
3f676b75e8 feat: standalone emulator support for batocera and multi-platform name mapping
resolve_platform_cores() builds reverse index from profile cores: field,
fixing 17 name mismatches across Batocera, RetroBat, and Recalbox
(genesisplusgx, pce_fast, pcfx, vb, mame078plus, vice cores, etc.).

standalone_path field on file entries + standalone_cores on platform
YAMLs enable mode-aware pack generation. find_undeclared_files() uses
standalone_path for cores the platform runs standalone, filters by
mode: libretro/standalone per file.

batocera.yml gains standalone_cores (92 entries from configgen-defaults).
generate_readme.py dynamically lists platforms from registry.
3 profiles updated for standalone type/path (mame, hatari, mupen64plus_next).
78 E2E tests pass, pipeline verified.
2026-03-26 00:44:21 +01:00
Abdessamad Derraz
d2cc9b8f29 feat: add doom engine wad files, emulatorjs base config 2026-03-25 23:12:53 +01:00
Abdessamad Derraz
1ad10eddb7 feat: include platform version in pack filenames 2026-03-25 18:55:35 +01:00
Abdessamad Derraz
dbc26b11c1 refactor: move fetch_large_file to common, auto-download on db rebuild 2026-03-25 13:19:12 +01:00
Abdessamad Derraz
47e6174ed4 fix: pack naming, large file preservation, discrepancy reporting 2026-03-25 12:23:40 +01:00
Abdessamad Derraz
0543165ed2 feat: re-profile 22 emulators, refactor validation to common.py
batch re-profiled nekop2 through pokemini. mupen64plus renamed to
mupen64plus_next. new profiles: nes, mupen64plus_next.
validation functions (_build_validation_index, check_file_validation)
consolidated in common.py — single source of truth for verify.py
and generate_pack.py. pipeline 100% consistent on all 6 platforms.
2026-03-24 22:31:22 +01:00