Commit Graph
317 Commits
Author SHA1 Message Date
Abdessamad Derraz d63c881648 chore: drop label comments across scripts
CommentsSayWhy read three scripts while 22 comments in 13 other files
named the next statement. They are removed or replaced by the reason,
and the test reads every tracked script and install.py.
2026-10-21 10:06:39 +02:00
Abdessamad Derraz 6c2204a7a8 chore: merge tooling follow-up
# Conflicts:
#	tests/test_audit_regressions.py
2026-10-21 06:50:20 +02:00
Abdessamad Derraz 378a9e4da5 chore: merge audit fixes (integrity)
# Conflicts:
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/retropie.json
#	install/rocknix.json
#	install/romm.json
#	scripts/generate_pack.py
#	wiki/tools.md
2026-10-21 06:48:42 +02:00
Abdessamad Derraz b8baf92f1d fix: give atomic writes the umask mode
mkstemp creates its scratch 0600 and os.replace kept that mode, so
database.json, the manifests, the packs, the recipe snapshots and the
large-file cache were readable by their owner only. set_plain_mode
gives the scratch 0o666 less the umask, as open() would, before each
rename. A test walks every function that renames a mkstemp file.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 9f1dbaa6c2 chore: merge audit fixes (packcli)
# Conflicts:
#	README.md
#	scripts/generate_pack.py
#	scripts/pipeline.py
2026-10-21 05:51:12 +02:00
Abdessamad Derraz 53da2a95d2 chore: merge audit fixes (docs)
# Conflicts:
#	tests/test_doc_surfaces.py
#	wiki/installer.md
#	wiki/testing-guide.md
2026-10-21 05:50:25 +02:00
Abdessamad Derraz f4a225488d chore: merge audit fixes (scrapers) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 738577b5a4 fix: run verify and generate_pack from any dir
Started outside the repository root, both ended on a FileNotFoundError
traceback: the database and the data registry name files from the root.
They now move to the root after reading the paths the caller typed, which
stay relative to where they were typed.
2026-10-21 05:25:29 +02:00
Abdessamad Derraz 76618af460 docs: shorten two script docstrings
_enrich_hashes repeated "ground truth" five times with capitalised
insistence for a three-step lookup, and the MAME ZIP rebuild listed
three guarantees of which two were the same.
2026-10-21 05:24:34 +02:00
Abdessamad Derraz 4fa598645e fix: print pack and lock errors on stderr
The lock refusal, "no pack for", "no manifest for" and "no pack to
verify" went to stdout, so a run redirected to a log exited 1 with
nothing on the terminal. A test reads both scripts and refuses an ERROR
line printed to stdout.
2026-10-21 05:22:05 +02:00
Abdessamad Derraz 0061041466 fix: claim the pack output before refreshing data
generate_pack refreshed the data directories before probing its output
lock. The refresh waits for every run holding a tree, and a pipeline holds
them all: a second build sat silent for hours instead of being refused.
The output lock is now taken first, and a refresh that has to wait for a
tree names it.
2026-10-21 05:21:54 +02:00
Abdessamad Derraz a160a5bc6b fix: name the hash formats --from-md5 accepts
A value of another length was looked up as an MD5 and reported NOT FOUND,
a SHA-256 was dropped, and an input with no hash ended on "No valid
hashes found in input". Digests are now read by length (CRC32, MD5 and
its 29-digit Batocera form, SHA-1, SHA-256), SHA-256 is looked up in the
database, and any other value is refused with the formats accepted.
2026-10-21 05:20:59 +02:00
Abdessamad Derraz bc68e517cb fix: suggest the closest system for --platform
--platform retroarch --system sony-playstaton listed the first ten systems
of the alphabet. The warning now names the closest systems and the
command that lists the platform's own, as --system alone already did.
2026-10-21 05:19:27 +02:00
Abdessamad Derraz 959934d43c refactor: compute the gap report once per build
The region grouping and phase 2 of each pack and manifest called
find_undeclared_files with the same arguments: 30 reports for the 15
manifests of --all --manifest, 2.8 s, now 15 and 1.7 s. The 15 manifests
are byte-identical before and after.
2026-10-21 05:18:53 +02:00
Abdessamad Derraz 38af19091d fix: compare pack paths, not only counts
Step 5 compared (ok, total) and the core file count. RetroArch, Lakka
and RetroPie read 526/526 on both sides while verify credited four paths
below SGB1.sfc and SGB2.sfc that the pack never writes: a declared file
blocks them as a directory. verify now places the platform files in the
builder's order with the same path rules, and a user_provided or
external declaration keeps its slot on both sides, so no core file takes
it in the report only.

verify --destinations writes the paths each pack holds by the report,
and step 5 compares them with the archive members (notes and data tree
members set aside). Every emulator profile, in each build it has, is
built in memory (PackRecord, nothing read or written) and compared with
verify --emulator path by path. Cost measured here: about 1 s per
platform archive and 11 s for the 930 emulator packs.
2026-10-21 05:18:02 +02:00
Abdessamad Derraz f7d561e42f fix: store already-compressed pack members
Every member was deflated, archives and PNGs included. On the 2026-10-20
RetroArch pack the .zip, .7z, .chd, .png, .ogg and .pk4 members are 2.1 GB
that took 28 s to deflate again for 9.6 MB saved (0.45 %), and about 16 GB
for 72 MB across all packs. Those formats are now stored. The choice reads
the member name only, so a pack stays byte-reproducible.
2026-10-21 05:10:00 +02:00
Abdessamad Derraz ae874534e2 fix: reread a contradicted hash from disk
_stable_hashes retried through the hash cache, so both retries returned
the first reading and the file was read once before the manifest run
stopped. The retries now read the file again.
2026-10-21 05:07:06 +02:00
Abdessamad Derraz 8008439033 fix: keep manifests and target lists apart
Both are written as <platform>.json. --manifest-targets aimed at install/
replaced install/retroarch.json with a target table, and --manifest aimed
at install/targets/ did the reverse, both with exit 0. Each mode now stops
when the output directory holds the other kind and names the directory to
use.
2026-10-21 05:06:36 +02:00
Abdessamad Derraz fb639647e6 fix: resolve a loose archive entry by its members
gngeopi and neo-emu declare neogeo.zip loose and, apart, the ROMs they
open inside it. The archive pass took the copy holding them, the loose
entry resolved by name to another copy, and both verify --emulator and
the emulator pack reported the archive missing while the ZIP held it.
with_archive_members() gives the loose entry the members in verify, the
pack, slots and the gap report.
2026-10-21 05:05:54 +02:00
Abdessamad Derraz 5fc2423ca5 fix: apply --required-only to data directories
The Required packs and manifests carried every data tree, the FreeIntv
overlays included. A data_directories entry now accepts required: false,
and the platform, emulator and manifest walks leave such a tree out under
--required-only. FreeIntv reads its overlays only with the multi-screen
option and draws a placeholder keypad without one (src/libretro.c:294-296,
350-369).
2026-10-21 05:05:49 +02:00
Abdessamad Derraz 83d49d93a8 fix: name every --all-variants pack in its check
--verify-packs after an --all-variants build rebuilt only the full pack's
name and opened it alone. The five narrowed packs of the same run passed
without a line. Each one is now named with the hash check the build ran
on its members.
2026-10-21 05:03:42 +02:00
Abdessamad Derraz 7095aa61e7 fix: make maintainer tool output actionable
--emulator accepts a buildbot core name from a profile's cores:, and a
mistyped target gets the closest names instead of every target. diff_truth
shows destinations and reports unpaired entries as evidence, not removals.
The verify summary counts discrepancies, a shared destination names both
systems, a truth hash conflict names both cores, and pipeline steps carry
their place in the real run instead of N/8. Every GitHub API caller takes
the token from github_token(), which falls back to gh auth token.
2026-10-21 04:42:01 +02:00
Abdessamad Derraz 6a852737a8 fix: one file under two spellings keeps both cores
BlastEm opens 32X_M_BIOS.bin and PicoDrive 32X_M_BIOS.BIN. On a case-insensitive platform the manifest kept the first and dropped the second core's claim, so an install narrowed to PicoDrive (RetroArch on Switch) lost the 32X BIOS its pack carries. The claim now joins the entry kept.
2026-10-20 11:27:08 +02:00
Abdessamad Derraz 71b59eb2a3 docs: rewrite half-dash asides in comments
Dropping the double-dash asides left 72 comments and docstrings with
half a dash glued to the next word ("shape -a size or a size range -is
all"). They now read as a colon, a comma or parentheses, and a prose
style test names any that comes back.
2026-10-20 09:17:10 +02:00
Abdessamad Derraz 6df087e3e9 fix: a platform's own data trees leave its pack
The platform layer names data trees for its systems: retroarch.yml carries dolphin-sys and ppsspp-assets, and Lakka inherits them, while Lakka's core packages install both trees into the system directory. A profile records that with installed_on on its data directory, and the pack and the install manifest now leave those trees out for that platform.
2026-10-20 06:36:32 +02:00
Abdessamad Derraz ddc7432825 feat: packs leave out what the emulator installs
A file or tree installed with the emulator is already on the user's machine, so packs, install manifests and verify's pack count leave it out. bundled is scoped by mode like any entry: Dolphin's Sys and PPSSPP's assets ship with the standalone releases, not with the libretro cores, so they now follow the libretro build only and still reach RetroArch packs. PrimeHack, standalone only, ships its own Sys.
2026-10-20 06:04:10 +02:00
Abdessamad Derraz 7edd438af1 fix: refuse a target no platform has
With --all, an unknown --target skipped every platform that has targets and ran the others (BizHawk, MiSTer) unfiltered, so verify, slots and the pipeline exited 0 on a typo. A platform without targets is now left out like the rest, and a target no platform has is refused.
2026-10-20 03:27:41 +02:00
Abdessamad Derraz 8fd8437351 refactor: share the core file lookup with verify
verify replayed the builder's placement of core files with a bare name lookup, without the archive index, the data directories or the core's own choice of copy. It compared other bytes than the pack's (blueMSX's MSX2P.rom from bios/ against the data tree's copy) and let an unresolved file take a path the builder leaves free. ExtraResolver now holds the builder's lookup and both call it.
2026-10-19 23:08:12 +02:00
Abdessamad Derraz 7d1e9f8816 fix: keep RomM files where its player reads them
RomM lists a platform's firmware folder without descending (base_handler.py:1002, recursive=False) and EmulatorJS writes the chosen file, or each zip member, under its bare name. 543 files of the RomM pack sat below a folder where no core could read them. The registry now marks RomM's folder flat, and such files are omitted with the reason below_platform_folder in the manifest and the installer.
2026-10-19 06:57:36 +02:00
Abdessamad Derraz 4231902a6c docs: write prose without dashes or semicolons
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.

validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
2026-10-19 06:47:31 +02:00
Abdessamad Derraz 04deb95a35 docs: give every command-line option a help text 2026-10-19 04:23:34 +02:00
Abdessamad Derraz 6d860a0df6 fix: refuse --verify-packs misuse before locking 2026-10-19 04:20:05 +02:00
Abdessamad Derraz 5ce2b44a57 fix: fill placeholders in emulator packs
LRPS2 declares only <bios>.bin, so its emulator pack shipped no PS2 BIOS. The images in the folder another PS2 profile proves by hash, of the declared size, now take the placeholder's place in the pack and in verify.
2026-10-19 04:11:19 +02:00
Abdessamad Derraz 953e943d2d fix: count data trees apart in emulator packs 2026-10-18 20:41:39 +02:00
Abdessamad Derraz 9a417dde2a fix: fail --verify-packs when no pack was checked 2026-10-18 20:39:14 +02:00
Abdessamad Derraz ff3428e1f1 fix: resolve manifest core files like the pack
The manifest resolved core files without the data directories: blueMSX's pack carried its own MSX2P.rom while the manifest named another dump for the installer.
2026-10-18 19:46:03 +02:00
Abdessamad Derraz 4246d9ec66 fix: name a target only where it narrows the pack 2026-10-18 19:44:47 +02:00
Abdessamad Derraz 9c154a29b9 fix: join system names with a separator they lack 2026-10-18 19:43:40 +02:00
Abdessamad Derraz c2e3d64343 fix: keep only the system's entries in --system
A multi-system profile shipped every entry: the Lynx system pack carried 34 PlayStation, Saturn and Apple II files from mednafen. The entry's own system decides, as the platform pack and the installer already do.
2026-10-18 19:39:28 +02:00
Abdessamad Derraz 1c7b2e85a4 fix: resolve nameless md5 entries as verify does 2026-10-18 18:18:19 +02:00
Abdessamad Derraz 34c8b925d7 fix: count a shared destination once, filled
verify kept the worst verdict on the raw destination, the builder the first one it read. Both now key on the sanitized path and let a declaration that holds a file fill one that finds nothing.
2026-10-18 18:17:28 +02:00
Abdessamad Derraz 8e067c1c3a fix: name the packs that fail verification 2026-10-18 16:21:54 +02:00
Abdessamad Derraz def1e80232 fix: fail a named pack that is not built 2026-10-18 16:20:33 +02:00
Abdessamad Derraz 10cfdb4de0 feat: install the data trees the pack carries
Manifests list each data directory's source archive and the hash of
every file; the installer rebuilds the tree from the archive and keeps
only the files whose hash matches.
2026-10-18 16:18:39 +02:00
Abdessamad Derraz dbd47886ce fix: name release assets as GitHub serves them 2026-10-18 14:49:00 +02:00
Abdessamad Derraz 9ea1630caa feat: install case twins where case is kept
The RetroArch list is folded for Windows and macOS. Its manifest
now carries the names a case-keeping directory also needs, and the
installer adds them after probing the BIOS directory.
2026-10-18 13:52:37 +02:00
Abdessamad Derraz df597cbe7d fix: hold data trees while packs are verified 2026-10-18 13:46:11 +02:00
Abdessamad Derraz c597df2a32 fix: swap in the pack after adding its manifest 2026-10-18 13:45:00 +02:00
Abdessamad Derraz 1f942770d5 fix: rank only held files in a slot 2026-10-18 13:43:41 +02:00
Abdessamad Derraz bfa6c51ff5 fix: report an archive another profile masks 2026-10-18 13:00:38 +02:00