A clone group is only visible while both copies are on disk, and the
run then deletes the clone, so writing only what this run saw erased
every mapping an earlier run had recorded. One real run took the map
from 69 entries to 1, and the canonical zips silently stopped answering
to the names they stand in for.
The map is merged now, the 68 lost entries are restored, and the one
whose canonical file is no longer in the collection is dropped.
deduplicate reached complexity 50 by mixing the question of which
copies may go with the work of removing them. The decision is now its
own function: same name in two directories is a true duplicate,
different names only collapse as MAME device clones, and anywhere else
each name may be the one some emulator looks for.
Verified against the real collection: both versions plan the same 99
removals.
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).
Run isort --profile black: normalize import ordering across all files.
Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.
3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
moved _mame_clones.json out of bios/ (was indexed by generate_db.py
as BIOS file). clone resolution now in common.py resolve_local_file
so all tools (verify, pack, cross_reference) resolve clones
transparently. removed duplicate clone code from generate_pack.py.
added error handling on os.remove in dedup.py. consistency check
now passes for Batocera/EmuDeck/Lakka/RetroArch (4/6 platforms).
non-ZIP files with different names but same content (64DD_IPL_US.n64
vs IPL_USA.n64) are now preserved — each name may be needed by a
different emulator. only same-name duplicates and MAME ZIP clones
are removed. added empty directory cleanup post-dedup.
Critical: stream large file downloads (OOM fix), fix basename match
in auto_fetch, include hashes in pack grouping fingerprint, handle
not_in_zip status in verify, fix escaped quotes in batocera parser.
Important: deduplicate shared group includes, catch coreinfo network
errors, fix NODEDUP path component match, fix CI word splitting on
spaces, replace bare except Exception in 3 files.
Minor: argparse in list_platforms, specific exceptions in download.py.
Abdessamad Derraz