Commit Graph
330 Commits
Author SHA1 Message Date
Abdessamad Derraz 7ca6adca56 fix: check narrowed packs against their own list
After --all-variants, --verify-packs printed "hashes checked during the
build" for the platform-only, truth and required-only packs and compared
none of them, so one missing a file its narrowed list requires passed.
Each pack is now held to the platform list narrowed the way it was built:
source and required_only, read from its name, reach the baseline, the
region grouping and the replay of the core extras.
2026-10-21 11:17:30 +02:00
Abdessamad Derraz b4e5e42c97 fix: leave out core files the core refuses by size
A core extra resolved by name shipped on existence platforms even when
the core's own size check turns the copy away. The RetroPie pack carried
the 128 KB Apple IIGS ROM as Basilisk II's ROM (min 512 KB). The builder,
the manifest and verify now read validation.core_size_refusal, which
accepts a copy any version of the same profile slot takes.
2026-10-21 10:46:52 +02:00
Abdessamad Derraz 052f837e69 chore: merge round 2 fixes (integrity) 2026-10-21 10:31:04 +02:00
Abdessamad Derraz 055627f39c chore: merge round 2 fixes (tooling) 2026-10-21 10:31:04 +02:00
Abdessamad Derraz ce1f165d2f chore: merge round 2 fixes (tests)
# Conflicts:
#	install.ps1
#	install.py
#	install.sh
2026-10-21 10:29:19 +02:00
Abdessamad Derraz de8ab51d6e fix: derive pack install guides from the registry
The pack README's one-line install was written per platform: ROCKNIX and MiSTer had none, EmuDeck's Windows edition was told to open Konsole, and RomM users were sent to library/bios/ whatever their firmware template. The automatic section now follows each platform's install.detect systems, the EmuDeck and RomM manual steps name every layout, and the download page lists EmuDeck under Windows and macOS.
2026-10-21 10:10:15 +02:00
Abdessamad Derraz fb40b903ca fix: refuse an unknown --system before the refresh
The platform's systems are known once its config is loaded. The filter ran inside the build, after the output lock and a refresh of every data directory, and ended on a WARNING with exit 1. It is now a usage error before any work.
2026-10-21 10:08:17 +02:00
Abdessamad Derraz a6eda07b8a fix: end --verify-packs on a summary line
The check ended on the last platform's line: a failing pack was named only above its own errors. It now names every failing pack on stderr at the end, as the build path does, or counts the packs found OK.
2026-10-21 10:06:52 +02:00
Abdessamad Derraz d63c881648 chore: drop label comments across scripts
CommentsSayWhy read three scripts while 22 comments in 13 other files
named the next statement. They are removed or replaced by the reason,
and the test reads every tracked script and install.py.
2026-10-21 10:06:39 +02:00
Abdessamad Derraz 5e4d9199c2 feat: accept platform system names in --system
The manifest now carries system_names, each system's native_system names from the platform's own file, and --system, its suggestions and --list-systems read them: --platform batocera --system psx installs sony-playstation.
2026-10-21 09:59:25 +02:00
Abdessamad Derraz 8ed1a737a8 fix: keep external downloads under the run policy
download_external stayed in generate_pack, outside the offline switch: it fetched under --offline, wrote to the system /tmp and held the body in memory. It now lives in packresolve, honours the switch, streams into the repository tmp/ and hashes on the way.
2026-10-21 09:58:21 +02:00
Abdessamad Derraz 4fec09a014 fix: name the selection mode in --from-md5 packs
BizHawk is both a platform and a profile display name: --from-md5 with --platform bizhawk and with --emulator bizhawk wrote one file in two layouts. The emulator mode now carries _Emulator in the name.
2026-10-21 09:56:45 +02:00
Abdessamad Derraz 2af9509ef4 fix: resolve declarations through one function
verify resolved platform entries with common.resolve_local_file and put
the slot override first, and the region pass added a destination hint.
packresolve.resolve_declaration now serves the builder, the manifest,
verify and the region pass: release assets cached for a file absent from
bios/ count as shipped in all four, and a release asset keeps its answer
over a slot override.
2026-10-21 09:56:07 +02:00
Abdessamad Derraz a3e625cd6a fix: refuse a default value typed out as a flag
The refusal tables compared each flag with its default value, so --source full and --group-by system passed beside the modes that refuse every other value. Both now default to None and take their effective value after the checks, in generate_pack and pipeline.
2026-10-21 09:55:51 +02:00
Abdessamad Derraz 6c2204a7a8 chore: merge tooling follow-up
# Conflicts:
#	tests/test_audit_regressions.py
2026-10-21 06:50:20 +02:00
Abdessamad Derraz 378a9e4da5 chore: merge audit fixes (integrity)
# Conflicts:
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/retropie.json
#	install/rocknix.json
#	install/romm.json
#	scripts/generate_pack.py
#	wiki/tools.md
2026-10-21 06:48:42 +02:00
Abdessamad Derraz b8baf92f1d fix: give atomic writes the umask mode
mkstemp creates its scratch 0600 and os.replace kept that mode, so
database.json, the manifests, the packs, the recipe snapshots and the
large-file cache were readable by their owner only. set_plain_mode
gives the scratch 0o666 less the umask, as open() would, before each
rename. A test walks every function that renames a mkstemp file.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 9f1dbaa6c2 chore: merge audit fixes (packcli)
# Conflicts:
#	README.md
#	scripts/generate_pack.py
#	scripts/pipeline.py
2026-10-21 05:51:12 +02:00
Abdessamad Derraz 53da2a95d2 chore: merge audit fixes (docs)
# Conflicts:
#	tests/test_doc_surfaces.py
#	wiki/installer.md
#	wiki/testing-guide.md
2026-10-21 05:50:25 +02:00
Abdessamad Derraz f4a225488d chore: merge audit fixes (scrapers) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 738577b5a4 fix: run verify and generate_pack from any dir
Started outside the repository root, both ended on a FileNotFoundError
traceback: the database and the data registry name files from the root.
They now move to the root after reading the paths the caller typed, which
stay relative to where they were typed.
2026-10-21 05:25:29 +02:00
Abdessamad Derraz 76618af460 docs: shorten two script docstrings
_enrich_hashes repeated "ground truth" five times with capitalised
insistence for a three-step lookup, and the MAME ZIP rebuild listed
three guarantees of which two were the same.
2026-10-21 05:24:34 +02:00
Abdessamad Derraz 4fa598645e fix: print pack and lock errors on stderr
The lock refusal, "no pack for", "no manifest for" and "no pack to
verify" went to stdout, so a run redirected to a log exited 1 with
nothing on the terminal. A test reads both scripts and refuses an ERROR
line printed to stdout.
2026-10-21 05:22:05 +02:00
Abdessamad Derraz 0061041466 fix: claim the pack output before refreshing data
generate_pack refreshed the data directories before probing its output
lock. The refresh waits for every run holding a tree, and a pipeline holds
them all: a second build sat silent for hours instead of being refused.
The output lock is now taken first, and a refresh that has to wait for a
tree names it.
2026-10-21 05:21:54 +02:00
Abdessamad Derraz a160a5bc6b fix: name the hash formats --from-md5 accepts
A value of another length was looked up as an MD5 and reported NOT FOUND,
a SHA-256 was dropped, and an input with no hash ended on "No valid
hashes found in input". Digests are now read by length (CRC32, MD5 and
its 29-digit Batocera form, SHA-1, SHA-256), SHA-256 is looked up in the
database, and any other value is refused with the formats accepted.
2026-10-21 05:20:59 +02:00
Abdessamad Derraz bc68e517cb fix: suggest the closest system for --platform
--platform retroarch --system sony-playstaton listed the first ten systems
of the alphabet. The warning now names the closest systems and the
command that lists the platform's own, as --system alone already did.
2026-10-21 05:19:27 +02:00
Abdessamad Derraz 959934d43c refactor: compute the gap report once per build
The region grouping and phase 2 of each pack and manifest called
find_undeclared_files with the same arguments: 30 reports for the 15
manifests of --all --manifest, 2.8 s, now 15 and 1.7 s. The 15 manifests
are byte-identical before and after.
2026-10-21 05:18:53 +02:00
Abdessamad Derraz 38af19091d fix: compare pack paths, not only counts
Step 5 compared (ok, total) and the core file count. RetroArch, Lakka
and RetroPie read 526/526 on both sides while verify credited four paths
below SGB1.sfc and SGB2.sfc that the pack never writes: a declared file
blocks them as a directory. verify now places the platform files in the
builder's order with the same path rules, and a user_provided or
external declaration keeps its slot on both sides, so no core file takes
it in the report only.

verify --destinations writes the paths each pack holds by the report,
and step 5 compares them with the archive members (notes and data tree
members set aside). Every emulator profile, in each build it has, is
built in memory (PackRecord, nothing read or written) and compared with
verify --emulator path by path. Cost measured here: about 1 s per
platform archive and 11 s for the 930 emulator packs.
2026-10-21 05:18:02 +02:00
Abdessamad Derraz f7d561e42f fix: store already-compressed pack members
Every member was deflated, archives and PNGs included. On the 2026-10-20
RetroArch pack the .zip, .7z, .chd, .png, .ogg and .pk4 members are 2.1 GB
that took 28 s to deflate again for 9.6 MB saved (0.45 %), and about 16 GB
for 72 MB across all packs. Those formats are now stored. The choice reads
the member name only, so a pack stays byte-reproducible.
2026-10-21 05:10:00 +02:00
Abdessamad Derraz ae874534e2 fix: reread a contradicted hash from disk
_stable_hashes retried through the hash cache, so both retries returned
the first reading and the file was read once before the manifest run
stopped. The retries now read the file again.
2026-10-21 05:07:06 +02:00
Abdessamad Derraz 8008439033 fix: keep manifests and target lists apart
Both are written as <platform>.json. --manifest-targets aimed at install/
replaced install/retroarch.json with a target table, and --manifest aimed
at install/targets/ did the reverse, both with exit 0. Each mode now stops
when the output directory holds the other kind and names the directory to
use.
2026-10-21 05:06:36 +02:00
Abdessamad Derraz fb639647e6 fix: resolve a loose archive entry by its members
gngeopi and neo-emu declare neogeo.zip loose and, apart, the ROMs they
open inside it. The archive pass took the copy holding them, the loose
entry resolved by name to another copy, and both verify --emulator and
the emulator pack reported the archive missing while the ZIP held it.
with_archive_members() gives the loose entry the members in verify, the
pack, slots and the gap report.
2026-10-21 05:05:54 +02:00
Abdessamad Derraz 5fc2423ca5 fix: apply --required-only to data directories
The Required packs and manifests carried every data tree, the FreeIntv
overlays included. A data_directories entry now accepts required: false,
and the platform, emulator and manifest walks leave such a tree out under
--required-only. FreeIntv reads its overlays only with the multi-screen
option and draws a placeholder keypad without one (src/libretro.c:294-296,
350-369).
2026-10-21 05:05:49 +02:00
Abdessamad Derraz 83d49d93a8 fix: name every --all-variants pack in its check
--verify-packs after an --all-variants build rebuilt only the full pack's
name and opened it alone. The five narrowed packs of the same run passed
without a line. Each one is now named with the hash check the build ran
on its members.
2026-10-21 05:03:42 +02:00
Abdessamad Derraz 7095aa61e7 fix: make maintainer tool output actionable
--emulator accepts a buildbot core name from a profile's cores:, and a
mistyped target gets the closest names instead of every target. diff_truth
shows destinations and reports unpaired entries as evidence, not removals.
The verify summary counts discrepancies, a shared destination names both
systems, a truth hash conflict names both cores, and pipeline steps carry
their place in the real run instead of N/8. Every GitHub API caller takes
the token from github_token(), which falls back to gh auth token.
2026-10-21 04:42:01 +02:00
Abdessamad Derraz 6a852737a8 fix: one file under two spellings keeps both cores
BlastEm opens 32X_M_BIOS.bin and PicoDrive 32X_M_BIOS.BIN. On a case-insensitive platform the manifest kept the first and dropped the second core's claim, so an install narrowed to PicoDrive (RetroArch on Switch) lost the 32X BIOS its pack carries. The claim now joins the entry kept.
2026-10-20 11:27:08 +02:00
Abdessamad Derraz 71b59eb2a3 docs: rewrite half-dash asides in comments
Dropping the double-dash asides left 72 comments and docstrings with
half a dash glued to the next word ("shape -a size or a size range -is
all"). They now read as a colon, a comma or parentheses, and a prose
style test names any that comes back.
2026-10-20 09:17:10 +02:00
Abdessamad Derraz 6df087e3e9 fix: a platform's own data trees leave its pack
The platform layer names data trees for its systems: retroarch.yml carries dolphin-sys and ppsspp-assets, and Lakka inherits them, while Lakka's core packages install both trees into the system directory. A profile records that with installed_on on its data directory, and the pack and the install manifest now leave those trees out for that platform.
2026-10-20 06:36:32 +02:00
Abdessamad Derraz ddc7432825 feat: packs leave out what the emulator installs
A file or tree installed with the emulator is already on the user's machine, so packs, install manifests and verify's pack count leave it out. bundled is scoped by mode like any entry: Dolphin's Sys and PPSSPP's assets ship with the standalone releases, not with the libretro cores, so they now follow the libretro build only and still reach RetroArch packs. PrimeHack, standalone only, ships its own Sys.
2026-10-20 06:04:10 +02:00
Abdessamad Derraz 7edd438af1 fix: refuse a target no platform has
With --all, an unknown --target skipped every platform that has targets and ran the others (BizHawk, MiSTer) unfiltered, so verify, slots and the pipeline exited 0 on a typo. A platform without targets is now left out like the rest, and a target no platform has is refused.
2026-10-20 03:27:41 +02:00
Abdessamad Derraz 8fd8437351 refactor: share the core file lookup with verify
verify replayed the builder's placement of core files with a bare name lookup, without the archive index, the data directories or the core's own choice of copy. It compared other bytes than the pack's (blueMSX's MSX2P.rom from bios/ against the data tree's copy) and let an unresolved file take a path the builder leaves free. ExtraResolver now holds the builder's lookup and both call it.
2026-10-19 23:08:12 +02:00
Abdessamad Derraz 7d1e9f8816 fix: keep RomM files where its player reads them
RomM lists a platform's firmware folder without descending (base_handler.py:1002, recursive=False) and EmulatorJS writes the chosen file, or each zip member, under its bare name. 543 files of the RomM pack sat below a folder where no core could read them. The registry now marks RomM's folder flat, and such files are omitted with the reason below_platform_folder in the manifest and the installer.
2026-10-19 06:57:36 +02:00
Abdessamad Derraz 4231902a6c docs: write prose without dashes or semicolons
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.

validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
2026-10-19 06:47:31 +02:00
Abdessamad Derraz 04deb95a35 docs: give every command-line option a help text 2026-10-19 04:23:34 +02:00
Abdessamad Derraz 6d860a0df6 fix: refuse --verify-packs misuse before locking 2026-10-19 04:20:05 +02:00
Abdessamad Derraz 5ce2b44a57 fix: fill placeholders in emulator packs
LRPS2 declares only <bios>.bin, so its emulator pack shipped no PS2 BIOS. The images in the folder another PS2 profile proves by hash, of the declared size, now take the placeholder's place in the pack and in verify.
2026-10-19 04:11:19 +02:00
Abdessamad Derraz 953e943d2d fix: count data trees apart in emulator packs 2026-10-18 20:41:39 +02:00
Abdessamad Derraz 9a417dde2a fix: fail --verify-packs when no pack was checked 2026-10-18 20:39:14 +02:00
Abdessamad Derraz ff3428e1f1 fix: resolve manifest core files like the pack
The manifest resolved core files without the data directories: blueMSX's pack carried its own MSX2P.rom while the manifest named another dump for the installer.
2026-10-18 19:46:03 +02:00
Abdessamad Derraz 4246d9ec66 fix: name a target only where it narrows the pack 2026-10-18 19:44:47 +02:00