Commit Graph
312 Commits
Author SHA1 Message Date
Abdessamad Derraz b8baf92f1d fix: give atomic writes the umask mode
mkstemp creates its scratch 0600 and os.replace kept that mode, so
database.json, the manifests, the packs, the recipe snapshots and the
large-file cache were readable by their owner only. set_plain_mode
gives the scratch 0o666 less the umask, as open() would, before each
rename. A test walks every function that renames a mkstemp file.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 9f1dbaa6c2 chore: merge audit fixes (packcli)
# Conflicts:
#	README.md
#	scripts/generate_pack.py
#	scripts/pipeline.py
2026-10-21 05:51:12 +02:00
Abdessamad Derraz 53da2a95d2 chore: merge audit fixes (docs)
# Conflicts:
#	tests/test_doc_surfaces.py
#	wiki/installer.md
#	wiki/testing-guide.md
2026-10-21 05:50:25 +02:00
Abdessamad Derraz f4a225488d chore: merge audit fixes (scrapers) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 738577b5a4 fix: run verify and generate_pack from any dir
Started outside the repository root, both ended on a FileNotFoundError
traceback: the database and the data registry name files from the root.
They now move to the root after reading the paths the caller typed, which
stay relative to where they were typed.
2026-10-21 05:25:29 +02:00
Abdessamad Derraz 76618af460 docs: shorten two script docstrings
_enrich_hashes repeated "ground truth" five times with capitalised
insistence for a three-step lookup, and the MAME ZIP rebuild listed
three guarantees of which two were the same.
2026-10-21 05:24:34 +02:00
Abdessamad Derraz 4fa598645e fix: print pack and lock errors on stderr
The lock refusal, "no pack for", "no manifest for" and "no pack to
verify" went to stdout, so a run redirected to a log exited 1 with
nothing on the terminal. A test reads both scripts and refuses an ERROR
line printed to stdout.
2026-10-21 05:22:05 +02:00
Abdessamad Derraz 0061041466 fix: claim the pack output before refreshing data
generate_pack refreshed the data directories before probing its output
lock. The refresh waits for every run holding a tree, and a pipeline holds
them all: a second build sat silent for hours instead of being refused.
The output lock is now taken first, and a refresh that has to wait for a
tree names it.
2026-10-21 05:21:54 +02:00
Abdessamad Derraz a160a5bc6b fix: name the hash formats --from-md5 accepts
A value of another length was looked up as an MD5 and reported NOT FOUND,
a SHA-256 was dropped, and an input with no hash ended on "No valid
hashes found in input". Digests are now read by length (CRC32, MD5 and
its 29-digit Batocera form, SHA-1, SHA-256), SHA-256 is looked up in the
database, and any other value is refused with the formats accepted.
2026-10-21 05:20:59 +02:00
Abdessamad Derraz bc68e517cb fix: suggest the closest system for --platform
--platform retroarch --system sony-playstaton listed the first ten systems
of the alphabet. The warning now names the closest systems and the
command that lists the platform's own, as --system alone already did.
2026-10-21 05:19:27 +02:00
Abdessamad Derraz 959934d43c refactor: compute the gap report once per build
The region grouping and phase 2 of each pack and manifest called
find_undeclared_files with the same arguments: 30 reports for the 15
manifests of --all --manifest, 2.8 s, now 15 and 1.7 s. The 15 manifests
are byte-identical before and after.
2026-10-21 05:18:53 +02:00
Abdessamad Derraz f7d561e42f fix: store already-compressed pack members
Every member was deflated, archives and PNGs included. On the 2026-10-20
RetroArch pack the .zip, .7z, .chd, .png, .ogg and .pk4 members are 2.1 GB
that took 28 s to deflate again for 9.6 MB saved (0.45 %), and about 16 GB
for 72 MB across all packs. Those formats are now stored. The choice reads
the member name only, so a pack stays byte-reproducible.
2026-10-21 05:10:00 +02:00
Abdessamad Derraz ae874534e2 fix: reread a contradicted hash from disk
_stable_hashes retried through the hash cache, so both retries returned
the first reading and the file was read once before the manifest run
stopped. The retries now read the file again.
2026-10-21 05:07:06 +02:00
Abdessamad Derraz 8008439033 fix: keep manifests and target lists apart
Both are written as <platform>.json. --manifest-targets aimed at install/
replaced install/retroarch.json with a target table, and --manifest aimed
at install/targets/ did the reverse, both with exit 0. Each mode now stops
when the output directory holds the other kind and names the directory to
use.
2026-10-21 05:06:36 +02:00
Abdessamad Derraz 5fc2423ca5 fix: apply --required-only to data directories
The Required packs and manifests carried every data tree, the FreeIntv
overlays included. A data_directories entry now accepts required: false,
and the platform, emulator and manifest walks leave such a tree out under
--required-only. FreeIntv reads its overlays only with the multi-screen
option and draws a placeholder keypad without one (src/libretro.c:294-296,
350-369).
2026-10-21 05:05:49 +02:00
Abdessamad Derraz 83d49d93a8 fix: name every --all-variants pack in its check
--verify-packs after an --all-variants build rebuilt only the full pack's
name and opened it alone. The five narrowed packs of the same run passed
without a line. Each one is now named with the hash check the build ran
on its members.
2026-10-21 05:03:42 +02:00
Abdessamad Derraz 7095aa61e7 fix: make maintainer tool output actionable
--emulator accepts a buildbot core name from a profile's cores:, and a
mistyped target gets the closest names instead of every target. diff_truth
shows destinations and reports unpaired entries as evidence, not removals.
The verify summary counts discrepancies, a shared destination names both
systems, a truth hash conflict names both cores, and pipeline steps carry
their place in the real run instead of N/8. Every GitHub API caller takes
the token from github_token(), which falls back to gh auth token.
2026-10-21 04:42:01 +02:00
Abdessamad Derraz 6a852737a8 fix: one file under two spellings keeps both cores
BlastEm opens 32X_M_BIOS.bin and PicoDrive 32X_M_BIOS.BIN. On a case-insensitive platform the manifest kept the first and dropped the second core's claim, so an install narrowed to PicoDrive (RetroArch on Switch) lost the 32X BIOS its pack carries. The claim now joins the entry kept.
2026-10-20 11:27:08 +02:00
Abdessamad Derraz 71b59eb2a3 docs: rewrite half-dash asides in comments
Dropping the double-dash asides left 72 comments and docstrings with
half a dash glued to the next word ("shape -a size or a size range -is
all"). They now read as a colon, a comma or parentheses, and a prose
style test names any that comes back.
2026-10-20 09:17:10 +02:00
Abdessamad Derraz 6df087e3e9 fix: a platform's own data trees leave its pack
The platform layer names data trees for its systems: retroarch.yml carries dolphin-sys and ppsspp-assets, and Lakka inherits them, while Lakka's core packages install both trees into the system directory. A profile records that with installed_on on its data directory, and the pack and the install manifest now leave those trees out for that platform.
2026-10-20 06:36:32 +02:00
Abdessamad Derraz ddc7432825 feat: packs leave out what the emulator installs
A file or tree installed with the emulator is already on the user's machine, so packs, install manifests and verify's pack count leave it out. bundled is scoped by mode like any entry: Dolphin's Sys and PPSSPP's assets ship with the standalone releases, not with the libretro cores, so they now follow the libretro build only and still reach RetroArch packs. PrimeHack, standalone only, ships its own Sys.
2026-10-20 06:04:10 +02:00
Abdessamad Derraz 7edd438af1 fix: refuse a target no platform has
With --all, an unknown --target skipped every platform that has targets and ran the others (BizHawk, MiSTer) unfiltered, so verify, slots and the pipeline exited 0 on a typo. A platform without targets is now left out like the rest, and a target no platform has is refused.
2026-10-20 03:27:41 +02:00
Abdessamad Derraz 8fd8437351 refactor: share the core file lookup with verify
verify replayed the builder's placement of core files with a bare name lookup, without the archive index, the data directories or the core's own choice of copy. It compared other bytes than the pack's (blueMSX's MSX2P.rom from bios/ against the data tree's copy) and let an unresolved file take a path the builder leaves free. ExtraResolver now holds the builder's lookup and both call it.
2026-10-19 23:08:12 +02:00
Abdessamad Derraz 7d1e9f8816 fix: keep RomM files where its player reads them
RomM lists a platform's firmware folder without descending (base_handler.py:1002, recursive=False) and EmulatorJS writes the chosen file, or each zip member, under its bare name. 543 files of the RomM pack sat below a folder where no core could read them. The registry now marks RomM's folder flat, and such files are omitted with the reason below_platform_folder in the manifest and the installer.
2026-10-19 06:57:36 +02:00
Abdessamad Derraz 4231902a6c docs: write prose without dashes or semicolons
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.

validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
2026-10-19 06:47:31 +02:00
Abdessamad Derraz 04deb95a35 docs: give every command-line option a help text 2026-10-19 04:23:34 +02:00
Abdessamad Derraz 6d860a0df6 fix: refuse --verify-packs misuse before locking 2026-10-19 04:20:05 +02:00
Abdessamad Derraz 5ce2b44a57 fix: fill placeholders in emulator packs
LRPS2 declares only <bios>.bin, so its emulator pack shipped no PS2 BIOS. The images in the folder another PS2 profile proves by hash, of the declared size, now take the placeholder's place in the pack and in verify.
2026-10-19 04:11:19 +02:00
Abdessamad Derraz 953e943d2d fix: count data trees apart in emulator packs 2026-10-18 20:41:39 +02:00
Abdessamad Derraz 9a417dde2a fix: fail --verify-packs when no pack was checked 2026-10-18 20:39:14 +02:00
Abdessamad Derraz ff3428e1f1 fix: resolve manifest core files like the pack
The manifest resolved core files without the data directories: blueMSX's pack carried its own MSX2P.rom while the manifest named another dump for the installer.
2026-10-18 19:46:03 +02:00
Abdessamad Derraz 4246d9ec66 fix: name a target only where it narrows the pack 2026-10-18 19:44:47 +02:00
Abdessamad Derraz 9c154a29b9 fix: join system names with a separator they lack 2026-10-18 19:43:40 +02:00
Abdessamad Derraz c2e3d64343 fix: keep only the system's entries in --system
A multi-system profile shipped every entry: the Lynx system pack carried 34 PlayStation, Saturn and Apple II files from mednafen. The entry's own system decides, as the platform pack and the installer already do.
2026-10-18 19:39:28 +02:00
Abdessamad Derraz 1c7b2e85a4 fix: resolve nameless md5 entries as verify does 2026-10-18 18:18:19 +02:00
Abdessamad Derraz 34c8b925d7 fix: count a shared destination once, filled
verify kept the worst verdict on the raw destination, the builder the first one it read. Both now key on the sanitized path and let a declaration that holds a file fill one that finds nothing.
2026-10-18 18:17:28 +02:00
Abdessamad Derraz 8e067c1c3a fix: name the packs that fail verification 2026-10-18 16:21:54 +02:00
Abdessamad Derraz def1e80232 fix: fail a named pack that is not built 2026-10-18 16:20:33 +02:00
Abdessamad Derraz 10cfdb4de0 feat: install the data trees the pack carries
Manifests list each data directory's source archive and the hash of
every file; the installer rebuilds the tree from the archive and keeps
only the files whose hash matches.
2026-10-18 16:18:39 +02:00
Abdessamad Derraz dbd47886ce fix: name release assets as GitHub serves them 2026-10-18 14:49:00 +02:00
Abdessamad Derraz 9ea1630caa feat: install case twins where case is kept
The RetroArch list is folded for Windows and macOS. Its manifest
now carries the names a case-keeping directory also needs, and the
installer adds them after probing the BIOS directory.
2026-10-18 13:52:37 +02:00
Abdessamad Derraz df597cbe7d fix: hold data trees while packs are verified 2026-10-18 13:46:11 +02:00
Abdessamad Derraz c597df2a32 fix: swap in the pack after adding its manifest 2026-10-18 13:45:00 +02:00
Abdessamad Derraz 1f942770d5 fix: rank only held files in a slot 2026-10-18 13:43:41 +02:00
Abdessamad Derraz bfa6c51ff5 fix: report an archive another profile masks 2026-10-18 13:00:38 +02:00
Abdessamad Derraz 92fb885341 fix: accept platform names as the site prints them 2026-10-18 08:49:41 +02:00
Abdessamad Derraz b23f5b04ad fix: refuse a --bios-dir that does not exist 2026-10-18 08:39:27 +02:00
Abdessamad Derraz 4240c98359 fix: name a missing database in one error line 2026-10-18 08:34:09 +02:00
Abdessamad Derraz 1305516e87 docs: say --offline cuts release asset downloads 2026-10-18 07:30:56 +02:00
Abdessamad Derraz ae3dc63bc3 fix: accept RomM files on size and any hash
RomM's verify_file_hashes compares the size, then md5, sha1 or crc. We
compared md5 alone, so a fixture entry with one mistyped hash was left out
of the pack RomM would accept it from. The rule is declared in romm.yml
and read by verify, the builder and the manifest.
2026-10-18 07:24:33 +02:00