Commit Graph
1425 Commits
Author SHA1 Message Date
Abdessamad Derraz 055627f39c chore: merge round 2 fixes (tooling) 2026-10-21 10:31:04 +02:00
Abdessamad Derraz 6409757c39 chore: merge round 2 fixes (docs)
# Conflicts:
#	CONTRIBUTING.md
#	scripts/check_release_assets.py
#	scripts/generate_readme.py
2026-10-21 10:31:04 +02:00
Abdessamad Derraz ce1f165d2f chore: merge round 2 fixes (tests)
# Conflicts:
#	install.ps1
#	install.py
#	install.sh
2026-10-21 10:29:19 +02:00
Abdessamad Derraz d87e1035ed chore: drop a clause semicolon in profile_sync 2026-10-21 10:24:39 +02:00
Abdessamad Derraz c8c4f3f486 chore: merge round 2 fixes (installer) 2026-10-21 10:17:44 +02:00
Abdessamad Derraz d7afac0993 chore: merge round 2 fixes (profile names) 2026-10-21 10:14:59 +02:00
Abdessamad Derraz 3ccee1b216 fix: drop the gba_bios.bin alias of nanoboyadvance
NanoBoyAdvance opens general.bios_path, default bios.bin (src/platform/core/src/config.cpp:36 at 3bb6f478). gba_bios.bin is the path ROCKNIX's config.toml sets, recorded under launchers: in rocknix.yml.

Without the alias the entry met only PCem's and QEMU's bios.bin, 64 and 128 KB, which BIOSLoader refuses (bios.cpp:17-48). verify counted it held by name, so the launcher layer left it alone and the ROCKNIX pack gained PCem's image. A hashless entry whose every same-named copy fails its size is now not held, and the gba_bios.bin ROCKNIX declares settles it.
2026-10-21 10:14:15 +02:00
Abdessamad Derraz 36d93c1750 feat: name the folder of a profile-only file
validate_pr.py built its folder advice from platform lists alone, so a
file only an emulator profile declares got the generic
bios/Manufacturer/Console/ advice. Profiles now feed the same table
through each entry's system, and the contributing guide says the
command answers where a file goes.
2026-10-21 10:13:09 +02:00
Abdessamad Derraz 2eee7a991d fix: accept -p and -t in every tool
verify, generate_pack and cross_reference took -p and -t, slots, generate_truth, diff_truth, export_native and refresh_data_dirs refused them. A test reads every add_argument call.
2026-10-21 10:11:22 +02:00
Abdessamad Derraz 746dee530f fix: compare a fork clone with the project main
A fork's clone has origin alone. Pushed to the fork's main, the commit
left origin/main...HEAD empty and --changed reported nothing with rc 0.
When no remote names the project, its main is fetched and compared, and
a fetch that fails is an error.
2026-10-21 10:10:41 +02:00
Abdessamad Derraz de8ab51d6e fix: derive pack install guides from the registry
The pack README's one-line install was written per platform: ROCKNIX and MiSTer had none, EmuDeck's Windows edition was told to open Konsole, and RomM users were sent to library/bios/ whatever their firmware template. The automatic section now follows each platform's install.detect systems, the EmuDeck and RomM manual steps name every layout, and the download page lists EmuDeck under Windows and macOS.
2026-10-21 10:10:15 +02:00
Abdessamad Derraz fac47dc4c0 fix: say how to run the schema check step 0 skips
Without jsonschema the step printed only the missing module, and the run then read COMPLETE while PR validation runs the same check. The skip now names the install command and the PR check.
2026-10-21 10:09:28 +02:00
Abdessamad Derraz ce7655affc chore: merge round 2 fixes (perf) 2026-10-21 10:08:54 +02:00
Abdessamad Derraz f8a9f98ae8 fix: title every consistency header from the list
Three headers of step 5 still read 5/8 beside the computed 5 consistency check [20/24], and a fourth had no number. All four now come from step_title.
2026-10-21 10:08:51 +02:00
Abdessamad Derraz d890ad40e6 chore: merge round 2 fixes (ops) 2026-10-21 10:08:27 +02:00
Abdessamad Derraz 78e172fd5f fix: index archive members once per profile
archive_members scanned the whole file list on every call, and callers ask once per entry of the same profile: 38 million entries walked in one verify --all over mame.yml. Lists of 64 entries or more are now indexed by archive once (held so the id is not reused, indexed again when the length changes), members still read from the entries at each call. archive_members drops from 2.30 s to 0.14 s tottime under the profiler. verify --all (text and --json), slots --json and the retroarch manifest are byte-identical.
2026-10-21 10:08:25 +02:00
Abdessamad Derraz ec1001de95 fix: read validate_pr data from the repo root
Run from another directory, validate_pr.py read no database, platform or
profile and reported every file as referenced by no platform and outside
bios/. It now moves to the root as verify.py does, reading the files
named from where they were typed.
2026-10-21 10:08:23 +02:00
Abdessamad Derraz fb40b903ca fix: refuse an unknown --system before the refresh
The platform's systems are known once its config is loaded. The filter ran inside the build, after the output lock and a refresh of every data directory, and ended on a WARNING with exit 1. It is now a usage error before any work.
2026-10-21 10:08:17 +02:00
Abdessamad Derraz c9780c8822 docs: add one BIOS from the browser
A browser upload into bios/ lands on a fork and opens the pull request, and CI already hashes the file and names its folder. CONTRIBUTING asked for a fork, a clone, five packages and the whole suite, and promised a PR comment that a fork's read-only token cannot post. The issue chooser links the upload page.
2026-10-21 10:06:57 +02:00
Abdessamad Derraz a6eda07b8a fix: end --verify-packs on a summary line
The check ended on the last platform's line: a failing pack was named only above its own errors. It now names every failing pack on stderr at the end, as the build path does, or counts the packs found OK.
2026-10-21 10:06:52 +02:00
Abdessamad Derraz a37af8e42e docs: readme installer options and plain wording
The quick install block gains the PowerShell form that passes options and a link to the installer page. The collection size uses the GB format of the download table and the site, counts get separators, and the sentences about RetroArch's existence check and the method describe the code instead of judging documentation.
2026-10-21 10:06:49 +02:00
Abdessamad Derraz d63c881648 chore: drop label comments across scripts
CommentsSayWhy read three scripts while 22 comments in 13 other files
named the next statement. They are removed or replaced by the reason,
and the test reads every tracked script and install.py.
2026-10-21 10:06:39 +02:00
Abdessamad Derraz 648db6bcbf fix: merge mame devices when the pin moves
Step 2a advanced mame.yml's core_version and upstream_commit to a new
release and left the device ROM sets on the previous one. A plain run
that moves the pin now fetches that release's listxml into .cache/dats
and runs the device merge, which checks version and commit itself.
2026-10-21 10:06:13 +02:00
Abdessamad Derraz 8c7c705169 fix: ask the release only for registered names
asset_candidates fell back to the bare name when no .gitignore line carried it: every online build asked large-files for aes.zip and five others and printed an HTTP 404 per name, pointing at an upload that never existed. The tests that fetch now register their names.
2026-10-21 10:06:11 +02:00
Abdessamad Derraz a74a56abc4 feat: name the profile a missing dump would fill
The acquisition report never read a profile, so XM7's required DICROM.ROM
sat among 2 287 targets at the rank of dumps no core reads. A missing
entry that a profile reads and the collection lacks is now listed first
with the profile, matched by hash or, for a hashless entry with a size,
by name and size. 29 entries on the current snapshots.
2026-10-21 10:05:10 +02:00
Abdessamad Derraz d16f8fd17a fix: accept downloads by cryptographic hash only
A crc32 and a size can be met by crafted bytes, so a remote download
matched on them could land in bios/. verify_content now requires every
sha256, sha1 and md5 the entry declares to match, and an entry declaring
none is reported as not verifiable from the network and nothing is
written. crc32 and size still find a file the database already holds.
2026-10-21 10:04:20 +02:00
Abdessamad Derraz d904116a79 fix: verify each 3DS file once per content
check_crypto_validation keeps its answer keyed by the file and keys file as compute_hashes keys them (path, inode, size, mtime, ctime). verify --all ran the verifiers 35 times for 5 files (0.88 s), now 5 times (0.13 s). verify --all --json output is byte-identical.
2026-10-21 10:03:27 +02:00
Abdessamad Derraz 6c076884ed fix: let auto_fetch prove sha256 and crc32 entries
verify_content read sha1 then md5 only, so an entry pinned by sha256 or
crc32 could be searched and never accepted, and one without any hash was
announced as searched. The strongest declared hash now decides, crc32
counts with its size, the database lookup reads sha256 and crc32, and an
entry with nothing to prove a download by is reported, not searched.
2026-10-21 10:02:05 +02:00
Abdessamad Derraz 11c836cdee fix: hash only same-size files in dedup scan
Equal files have equal sizes, and the scan only uses the sha1. Files alone at their size are no longer read and the rest get one digest instead of five. Same 278 groups (645 paths), same dry-run output, 36.7 s to 1.9 s.
2026-10-21 10:00:53 +02:00
Abdessamad Derraz 7b9c8ae663 fix: refuse a mistyped --target in cross_reference
load_target_config raised out of cross_reference.py as a traceback. The hint now names the full verify.py command, since cross_reference and slots have no --list-targets.
2026-10-21 10:00:47 +02:00
Abdessamad Derraz 932410e00d fix: send GITHUB_TOKEN from download.sh
download.py authenticated its release lookup and download.sh did not, so
past the anonymous quota the shell twin failed with 403. The header goes
to api.github.com only and is read from stdin, out of the process list.
2026-10-21 10:00:44 +02:00
Abdessamad Derraz f3586001ab fix: fail freshness on unknown, name the refresher
check_freshness exited 0 when no upstream answered. An unknown row now
holds the exit code, and a profile skipped by nature is reported skipped
so it does not hold it forever. --only names the valid areas, a stale
native row names its platform, and the summary names refresh_stale.py.
2026-10-21 10:00:36 +02:00
Abdessamad Derraz 0c65d93248 fix: state only what lands in the export summary
The summary in exports/<platform>.json read '82 hashes corrected' and '854 additions left out'. It now counts what the written file carries, in the exports page's terms (hashes updated to the emulator source, required flags aligned with the code). Files the format cannot carry, withheld updates and content-found labels are printed on a line of their own and stay in the report's counts.
2026-10-21 09:59:49 +02:00
Abdessamad Derraz 5e4d9199c2 feat: accept platform system names in --system
The manifest now carries system_names, each system's native_system names from the platform's own file, and --system, its suggestions and --list-systems read them: --platform batocera --system psx installs sony-playstation.
2026-10-21 09:59:25 +02:00
Abdessamad Derraz cc57d44de5 fix: order inherited rescrapes, end on verify
refresh_stale ran a parent's rescrape and its children's truth and
export jobs side by side, so a child read the parent mid-rewrite. A job
now starts once the rescrapes of its ancestors have ended, and a child
no row names gets its export rewritten after them. A rescrape chain ends
on verify.py --platform, whose summary closes the report.
2026-10-21 09:58:58 +02:00
Abdessamad Derraz 8ed1a737a8 fix: keep external downloads under the run policy
download_external stayed in generate_pack, outside the offline switch: it fetched under --offline, wrote to the system /tmp and held the body in memory. It now lives in packresolve, honours the switch, streams into the repository tmp/ and hashes on the way.
2026-10-21 09:58:21 +02:00
Abdessamad Derraz 287a16f41c fix: name --fetch when an original is not cached
The export failed with 'absent and fetching is off'. It now names the cache directory it looked in and the flag that downloads the file.
2026-10-21 09:57:44 +02:00
Abdessamad Derraz 83a4b21d30 fix: keep platform hashes cores disagree on
When two cores declare different values for one file, the truth keeps the first and marks the entry hash_conflict. Its hashes pair an entry and never replace the platform's: armsx2's GameIndex.yaml no longer overwrites System.dat's line for LRPS2.
2026-10-21 09:57:20 +02:00
Abdessamad Derraz f997bdc81a fix: check a recalbox zip by its composite only
Bios::Scan takes Zip::Md5Composite for every .zip (Bios.cpp:64-68). The resolver already honoured zip_digest: composite, but digest_verdict accepted the container's own md5 first, so the pack check passed bytes Recalbox marks red. The page described that order.
2026-10-21 09:57:06 +02:00
Abdessamad Derraz 585be38ee3 fix: transform each held content once
Step 1b read 18 924 held files and members for 7 407 distinct contents and ran every transform on each read. A content already transformed is now skipped (the first path still keeps the match), the power-of-two crc shift operators are squared once for the pass instead of once per length, and a group pairs members by the sizes it holds. derivable.json and the step output are byte-identical, 33.4 s to 25.1 s.
2026-10-21 09:56:53 +02:00
Abdessamad Derraz 9bb1a73f91 fix: hold every data tree, present or not
export_native and cross_reference read data/ without the shared hold, so
a refresh could swap a tree mid-walk. hold_data_dirs skipped a cache not
yet on disk, so a refresh could install it between verify and the packs
of one pipeline run. Both now hold every registered tree.
2026-10-21 09:56:49 +02:00
Abdessamad Derraz 4fec09a014 fix: name the selection mode in --from-md5 packs
BizHawk is both a platform and a profile display name: --from-md5 with --platform bizhawk and with --emulator bizhawk wrote one file in two layouts. The emulator mode now carries _Emulator in the name.
2026-10-21 09:56:45 +02:00
Abdessamad Derraz a3e625cd6a fix: refuse a default value typed out as a flag
The refusal tables compared each flag with its default value, so --source full and --group-by system passed beside the modes that refuse every other value. Both now default to None and take their effective value after the checks, in generate_pack and pipeline.
2026-10-21 09:55:51 +02:00
Abdessamad Derraz 563ce1ebf2 fix: read fbneo dats at one checked commit
The FBNeo import recorded the listing's blob sha for bytes fetched from a
branch URL a CDN caches, and --fetch <revision> still read the tip. The
revision is now resolved to a commit, listed and downloaded there, each
DAT is held against its blob sha, and a DAT read at a named revision is
labelled with it without replacing the tip's recorded blobs.
2026-10-21 09:55:50 +02:00
Abdessamad Derraz d8e9ad23d3 docs: describe short md5 handling as the code does
No platform YAML holds an md5 shorter than 32 characters: the Batocera scraper completes one when a single collection md5 starts with it. The pages said some entries carry 29 characters and that scrapers store them as-is.
2026-10-21 09:54:57 +02:00
Abdessamad Derraz 8721dafd2a fix: load every script under the 3.9 floor
A module-level alias written with a PEP 604 union compiled under 3.9 and
raised TypeError when check_release_assets.py, and check_release.py
through it, were loaded. The floor test now imports every module under
scripts/ with a real 3.9 interpreter, which CI gives pyyaml and
jsonschema.
2026-10-21 09:53:34 +02:00
Abdessamad Derraz 12cc71b634 fix: hold the pipeline's fixed outputs for a run
The pipeline locked only --output-dir while it also writes database.json,
install/, README.md and docs/. A run on a scratch output directory could
start beside the run on dist/ and rewrite them. Every run now holds the
repository's .lock first.
2026-10-21 09:53:15 +02:00
Abdessamad Derraz a6d37e40ae docs: show the handy output verify prints 2026-10-21 09:52:57 +02:00
Abdessamad Derraz f7b758a41d fix: patch batocera entries past commented items
_patched_entry read '#{ ... }' lines as live items, found no match in the model and gave up, so atarist was written again whole (52 lines changed for 3 additions). A commented item is now left as written.
2026-10-21 09:52:34 +02:00
Abdessamad Derraz 3d2a521f8a fix: match profile_sync --system as verify does
--system compared the raw id, so the normalized spelling printed by verify.py --list-systems selected 1 Saturn profile of 11. The hint now counts the spellings that listing prints.
2026-10-21 09:52:13 +02:00