Commit Graph
1507 Commits
Author SHA1 Message Date
Abdessamad Derraz d904116a79 fix: verify each 3DS file once per content
check_crypto_validation keeps its answer keyed by the file and keys file as compute_hashes keys them (path, inode, size, mtime, ctime). verify --all ran the verifiers 35 times for 5 files (0.88 s), now 5 times (0.13 s). verify --all --json output is byte-identical.
2026-10-21 10:03:27 +02:00
Abdessamad Derraz e65ece743d fix: name where a held profile entry sits
164 entries wore the in repo badge with no copy named: they are held as
a ZIP member or in a data directory, which the database does not index.
held_copy() finds that copy and the card names it. storage: large_file
no longer counts as held by itself: kenji-nx's user-picked firmware
package read as held although nothing collects it.
2026-10-21 10:02:50 +02:00
Abdessamad Derraz 6c076884ed fix: let auto_fetch prove sha256 and crc32 entries
verify_content read sha1 then md5 only, so an entry pinned by sha256 or
crc32 could be searched and never accepted, and one without any hash was
announced as searched. The strongest declared hash now decides, crc32
counts with its size, the database lookup reads sha256 and crc32, and an
entry with nothing to prove a download by is reported, not searched.
2026-10-21 10:02:05 +02:00
Abdessamad Derraz 3b60facd34 fix: count core copies under their own category
The second pass, the archive-prefix pass and the agnostic scan built
extras without source_name or source_path, so verify found no gap report
entry for them and counted game data such as hiscore.dat as core files.
Every extra now carries its entry's category and key, and verify reads
the category from the extra the builder places.
2026-10-21 10:01:08 +02:00
Abdessamad Derraz 11c836cdee fix: hash only same-size files in dedup scan
Equal files have equal sizes, and the scan only uses the sha1. Files alone at their size are no longer read and the rest get one digest instead of five. Same 278 groups (645 paths), same dry-run output, 36.7 s to 1.9 s.
2026-10-21 10:00:53 +02:00
Abdessamad Derraz 7b9c8ae663 fix: refuse a mistyped --target in cross_reference
load_target_config raised out of cross_reference.py as a traceback. The hint now names the full verify.py command, since cross_reference and slots have no --list-targets.
2026-10-21 10:00:47 +02:00
Abdessamad Derraz 932410e00d fix: send GITHUB_TOKEN from download.sh
download.py authenticated its release lookup and download.sh did not, so
past the anonymous quota the shell twin failed with 403. The header goes
to api.github.com only and is read from stdin, out of the process list.
2026-10-21 10:00:44 +02:00
Abdessamad Derraz f3586001ab fix: fail freshness on unknown, name the refresher
check_freshness exited 0 when no upstream answered. An unknown row now
holds the exit code, and a profile skipped by nature is reported skipped
so it does not hold it forever. --only names the valid areas, a stale
native row names its platform, and the summary names refresh_stale.py.
2026-10-21 10:00:36 +02:00
Abdessamad Derraz 0c65d93248 fix: state only what lands in the export summary
The summary in exports/<platform>.json read '82 hashes corrected' and '854 additions left out'. It now counts what the written file carries, in the exports page's terms (hashes updated to the emulator source, required flags aligned with the code). Files the format cannot carry, withheld updates and content-found labels are printed on a line of their own and stay in the report's counts.
2026-10-21 09:59:49 +02:00
Abdessamad Derraz 5e4d9199c2 feat: accept platform system names in --system
The manifest now carries system_names, each system's native_system names from the platform's own file, and --system, its suggestions and --list-systems read them: --platform batocera --system psx installs sony-playstation.
2026-10-21 09:59:25 +02:00
Abdessamad Derraz cc57d44de5 fix: order inherited rescrapes, end on verify
refresh_stale ran a parent's rescrape and its children's truth and
export jobs side by side, so a child read the parent mid-rewrite. A job
now starts once the rescrapes of its ancestors have ended, and a child
no row names gets its export rewritten after them. A rescrape chain ends
on verify.py --platform, whose summary closes the report.
2026-10-21 09:58:58 +02:00
Abdessamad Derraz 8ed1a737a8 fix: keep external downloads under the run policy
download_external stayed in generate_pack, outside the offline switch: it fetched under --offline, wrote to the system /tmp and held the body in memory. It now lives in packresolve, honours the switch, streams into the repository tmp/ and hashes on the way.
2026-10-21 09:58:21 +02:00
Abdessamad Derraz 084dff84f4 fix: list what the emulator pack writes
verify --emulator listed only OK destinations, while the emulator pack
also writes a copy whose hash the profile contradicts, loose or archive.
pack_destinations now records the paths the replay places, and both
branches resolve through packresolve.resolve_file as the pack does.
2026-10-21 09:58:09 +02:00
Abdessamad Derraz 287a16f41c fix: name --fetch when an original is not cached
The export failed with 'absent and fetching is off'. It now names the cache directory it looked in and the flag that downloads the file.
2026-10-21 09:57:44 +02:00
Abdessamad Derraz 83a4b21d30 fix: keep platform hashes cores disagree on
When two cores declare different values for one file, the truth keeps the first and marks the entry hash_conflict. Its hashes pair an entry and never replace the platform's: armsx2's GameIndex.yaml no longer overwrites System.dat's line for LRPS2.
2026-10-21 09:57:20 +02:00
Abdessamad Derraz f997bdc81a fix: check a recalbox zip by its composite only
Bios::Scan takes Zip::Md5Composite for every .zip (Bios.cpp:64-68). The resolver already honoured zip_digest: composite, but digest_verdict accepted the container's own md5 first, so the pack check passed bytes Recalbox marks red. The page described that order.
2026-10-21 09:57:06 +02:00
Abdessamad Derraz 585be38ee3 fix: transform each held content once
Step 1b read 18 924 held files and members for 7 407 distinct contents and ran every transform on each read. A content already transformed is now skipped (the first path still keeps the match), the power-of-two crc shift operators are squared once for the pass instead of once per length, and a group pairs members by the sizes it holds. derivable.json and the step output are byte-identical, 33.4 s to 25.1 s.
2026-10-21 09:56:53 +02:00
Abdessamad Derraz 9bb1a73f91 fix: hold every data tree, present or not
export_native and cross_reference read data/ without the shared hold, so
a refresh could swap a tree mid-walk. hold_data_dirs skipped a cache not
yet on disk, so a refresh could install it between verify and the packs
of one pipeline run. Both now hold every registered tree.
2026-10-21 09:56:49 +02:00
Abdessamad Derraz 4fec09a014 fix: name the selection mode in --from-md5 packs
BizHawk is both a platform and a profile display name: --from-md5 with --platform bizhawk and with --emulator bizhawk wrote one file in two layouts. The emulator mode now carries _Emulator in the name.
2026-10-21 09:56:45 +02:00
Abdessamad Derraz 4ceca6acc5 fix: credit platforms with the file the pack ships
FileConsumers resolved platform declarations with their destination as
a hint, which the builder, the manifest and verify never pass. RetroBat's
63 mame/hash lists and the curated gc-ntsc-12.bin credited another copy
than the one the manifests install.
2026-10-21 09:56:08 +02:00
Abdessamad Derraz 2af9509ef4 fix: resolve declarations through one function
verify resolved platform entries with common.resolve_local_file and put
the slot override first, and the region pass added a destination hint.
packresolve.resolve_declaration now serves the builder, the manifest,
verify and the region pass: release assets cached for a file absent from
bios/ count as shipped in all four, and a release asset keeps its answer
over a slot override.
2026-10-21 09:56:07 +02:00
Abdessamad Derraz a3e625cd6a fix: refuse a default value typed out as a flag
The refusal tables compared each flag with its default value, so --source full and --group-by system passed beside the modes that refuse every other value. Both now default to None and take their effective value after the checks, in generate_pack and pipeline.
2026-10-21 09:55:51 +02:00
Abdessamad Derraz 563ce1ebf2 fix: read fbneo dats at one checked commit
The FBNeo import recorded the listing's blob sha for bytes fetched from a
branch URL a CDN caches, and --fetch <revision> still read the tip. The
revision is now resolved to a commit, listed and downloaded there, each
DAT is held against its blob sha, and a DAT read at a named revision is
labelled with it without replacing the tip's recorded blobs.
2026-10-21 09:55:50 +02:00
Abdessamad Derraz d8e9ad23d3 docs: describe short md5 handling as the code does
No platform YAML holds an md5 shorter than 32 characters: the Batocera scraper completes one when a single collection md5 starts with it. The pages said some entries carry 29 characters and that scrapers store them as-is.
2026-10-21 09:54:57 +02:00
Abdessamad Derraz 45db77cfd0 fix: link a system to its maker's folder
The database keeps one path per content, so a system placed by the
copy deduplication kept linked to another maker: nintendo-nes and
nintendo-fds to Arcade > FBNeo, sega-mastersystem and
sinclair-zx-spectrum to Other > zesarux. Every folder holding a copy now
votes, an id that spells one console folder links to it, and a maker's
folder outranks the folders that file copies by emulator.
2026-10-21 09:54:56 +02:00
Abdessamad Derraz 8721dafd2a fix: load every script under the 3.9 floor
A module-level alias written with a PEP 604 union compiled under 3.9 and
raised TypeError when check_release_assets.py, and check_release.py
through it, were loaded. The floor test now imports every module under
scripts/ with a real 3.9 interpreter, which CI gives pyyaml and
jsonschema.
2026-10-21 09:53:34 +02:00
Abdessamad Derraz 12cc71b634 fix: hold the pipeline's fixed outputs for a run
The pipeline locked only --output-dir while it also writes database.json,
install/, README.md and docs/. A run on a scratch output directory could
start beside the run on dist/ and rewrite them. Every run now holds the
repository's .lock first.
2026-10-21 09:53:15 +02:00
Abdessamad Derraz a6d37e40ae docs: show the handy output verify prints 2026-10-21 09:52:57 +02:00
Abdessamad Derraz f7b758a41d fix: patch batocera entries past commented items
_patched_entry read '#{ ... }' lines as live items, found no match in the model and gave up, so atarist was written again whole (52 lines changed for 3 additions). A commented item is now left as written.
2026-10-21 09:52:34 +02:00
Abdessamad Derraz 3d2a521f8a fix: match profile_sync --system as verify does
--system compared the raw id, so the normalized spelling printed by verify.py --list-systems selected 1 Saturn profile of 11. The hint now counts the spellings that listing prints.
2026-10-21 09:52:13 +02:00
Abdessamad Derraz 43337c6719 fix: match release packs by exact name
release_record rebuilt nothing from the name builder: a substring match
took RetroArch_v1.22.2_AndroidArm64V8A_BIOS_Pack.zip for the RetroArch
pack. The record now refuses any pack that is not a registered platform's
full pack and stores the platforms each pack serves, which the readers
use instead of the pack name.
2026-10-21 09:51:55 +02:00
Abdessamad Derraz 9bb3dfbd1b fix: keep release checks importable on python 3.9
check_release_assets.py built a tuple[..., int | None] alias at import, a TypeError on the 3.9 floor tools.md states, and check_release.py imports it. A test reads every script for unions and match statements evaluated at runtime.
2026-10-21 09:51:33 +02:00
Abdessamad Derraz 5ec1208a83 fix: write romm keys with a bare file name
RomM lists the platform firmware folder without recursing and looks each file up as slug:file_name (models/firmware.py, base_handler.list_files at 5.4.0). An addition named with a directory (fuse/48.rom) is not written.
2026-10-21 09:51:32 +02:00
Abdessamad Derraz 6877512af7 feat: carry new upstream names and hashes in json
--detect-new-files and --watch-hashes were printed only by the text report and refused with --json, so check_freshness --profiles never saw a name or hash the cited code gained. Both now live on the report, and check_freshness passes them and marks such a profile stale.
2026-10-21 09:51:14 +02:00
Abdessamad Derraz 4205bc2233 fix: count one retrobat addition per path
The render writes one biosFiles item per path, as checkBios reads one md5 per item. A second addition at a held path was dropped from the file and still counted added (78 announced, 75 written).
2026-10-21 09:50:46 +02:00
Abdessamad Derraz 9409ec12b3 fix: keep the two mt32emu.h citations together
Each MT-32 entry cites the SHA-1 line and the ROMInfo line of one
mt32emu.h. dosbox_pure wrote them under two repository words, so the
ROMInfo line was self-checked alone and read MOVED onto the SHA-1 line.
Both now sit in the port that carries the file, and in the upstream for
dosbox-pure-unleashed, whose tree has no src/gui.

A recale that lands on a line a sibling part already cites goes to
review, and a part settled by its anchor keeps that anchor's repository.
2026-10-21 09:50:40 +02:00
Abdessamad Derraz 3545c1b333 fix: scope gap report entries by platform
cross_reference --platform read bundled from the profile alone and never
read unread_on. It now calls validation.installed_with_emulator and
unread_on with the platform key, as verify does, and a file installed
with its emulator is reported as bundled whether held or not.
2026-10-21 09:49:49 +02:00
Abdessamad Derraz fd9a9b14f8 fix: map rows edited in place one to one
A replace opcode of equal size is a line-by-line edit. Collapsing it onto its first line sent every ref citing a table row to the first row.
2026-10-21 09:48:33 +02:00
Abdessamad Derraz bc2233e725 fix: write truth systems in a stable order
Systems a core covers without a file were added while walking a set,
so the hash seed placed them and two runs wrote dist/truth/<platform>.yml
in two orders. The normalized system map was built from a set too.
2026-10-21 06:57:49 +02:00
Abdessamad Derraz 6c2204a7a8 chore: merge tooling follow-up
# Conflicts:
#	tests/test_audit_regressions.py
2026-10-21 06:50:20 +02:00
Abdessamad Derraz 768b42e807 fix: fold case in the site gap report too 2026-10-21 06:50:10 +02:00
Abdessamad Derraz 378a9e4da5 chore: merge audit fixes (integrity)
# Conflicts:
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/retropie.json
#	install/rocknix.json
#	install/romm.json
#	scripts/generate_pack.py
#	wiki/tools.md
2026-10-21 06:48:42 +02:00
Abdessamad Derraz 4ba935374e fix: keep pack order outside contested paths
Ranking contested core claims sorted the whole gap report, which
reordered every manifest. Only the claims on one contested path move now,
grouped where the first of them stood, and two spellings of a path are
one group where the filesystem folds case (RetroBat's 64DD_IPL_DEV.bin).

An entry without a destination (EmuDeck's system:md5 whitelist) no
longer counts as a pack path in verify, and step 5 prints one status for
counts and paths together. A full offline run compared all 12 archives
and 930 emulator packs path by path, every one equal.
2026-10-21 06:47:44 +02:00
Abdessamad Derraz 85a44d9bf7 chore: merge amiberry and hatari re-profile
# Conflicts:
#	README.md
#	emulators/amiberry.yml
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/rocknix.json
2026-10-21 06:21:30 +02:00
Abdessamad Derraz 76e44a74b4 refactor: drop comments that restate the code
Thirty one-line labels in verify.py, generate_site.py and
generate_truth.py named the statement under them ("# Load registry",
"# Copy branding assets"). Comments that give a reason stay, as do the
module section headers. A test reads the three files for new ones.
2026-10-21 06:20:50 +02:00
Abdessamad Derraz 3613c0927b docs: open the download page on its content
The which-pack page announced itself ("This page picks the right pack
for a setup") before naming a pack. The sentence goes, the pointer to
Getting started stays.
2026-10-21 06:18:59 +02:00
Abdessamad Derraz 566d36275f fix: count alias readers in a file's requirers
Ymir reads sega_101.bin or saturn_bios.bin, so the sega_101.bin entry
carries saturn_bios.bin as an alias. kronos, yabause and yabasanshiro
open saturn_bios.bin and fall back to sega_101.bin: they were in that
entry's _cores but not its _required_by, while their saturn_bios.bin
entry had them requiring it. A reader that requires one of an entry's
aliases now requires the entry. No reader is added, and same-name
entries kept apart by path are left alone. No export changes.
2026-10-21 06:18:32 +02:00
Abdessamad Derraz 5ce7ab8386 fix: self-check a ref in every repository
A prose citation of the port's own lines was read only in the upstream file that a source_ref attributes the path to, and a range past that file's end came back GONE although the port carries it.
2026-10-21 06:18:00 +02:00
Abdessamad Derraz f19a8f99d3 fix: read composite zip md5s apart in diff_truth
Recalbox hashes a .zip by its members. diff_truth set that composite
against the profile's container md5 and reported neocdz.zip and
neocd.zip as mismatches. _hash_mismatch now drops it through
comparable(), the rule pair_entries and the export already apply.
2026-10-21 06:12:08 +02:00
Abdessamad Derraz b8baf92f1d fix: give atomic writes the umask mode
mkstemp creates its scratch 0600 and os.replace kept that mode, so
database.json, the manifests, the packs, the recipe snapshots and the
large-file cache were readable by their owner only. set_plain_mode
gives the scratch 0o666 less the umask, as open() would, before each
rename. A test walks every function that renames a mkstemp file.
2026-10-21 06:09:55 +02:00