Commit Graph
223 Commits
Author SHA1 Message Date
Abdessamad Derraz 377e53f4c0 feat: show load_from, built_by and pin fields
load_from, found_by, on_mismatch, built_by, pin_frozen and
version_source were in the profiles and on no page: Dolphin's
gba_bios.bin showed a path under system/ although the libretro core reads
it from its save directory, and mame0278 did not say only Recalbox ships
it.
2026-10-21 10:05:18 +02:00
Abdessamad Derraz 6c2204a7a8 chore: merge tooling follow-up
# Conflicts:
#	tests/test_audit_regressions.py
2026-10-21 06:50:20 +02:00
Abdessamad Derraz 378a9e4da5 chore: merge audit fixes (integrity)
# Conflicts:
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/retropie.json
#	install/rocknix.json
#	install/romm.json
#	scripts/generate_pack.py
#	wiki/tools.md
2026-10-21 06:48:42 +02:00
Abdessamad Derraz 57ac252482 feat: check the profiles a branch adds or edits
profile_sync gains --changed (added or edited profiles, uncommitted
ones included) and --added, both against the merge base of --base
(origin/main by default). CONTRIBUTING runs --changed --strict instead
of one --emulator call per new profile, and the CI job runs --added
in place of its own diff loop.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 35b4cacb5d chore: merge audit fixes (installer)
# Conflicts:
#	install.ps1
#	install.sh
2026-10-21 06:02:02 +02:00
Abdessamad Derraz 1c4686ab93 feat: place RomM firmware by its own template
RomM 5.4 reads each platform's firmware where filesystem.structure.firmware
in config.yml says, bios/{platform} by default, and suggests {platform}/bios
for a {platform}/roms library (rommapp/romm 5.4.0,
backend/config/config_manager.py:58,101-109,205-225,564-568,1105-1124).
The installer only wrote bios/<platform>/. It now reads the template from
config.yml beside the library, takes --romm-firmware when that file is
elsewhere, refuses a platform-first library with no template, and detects
RomM by /romm/config/config.yml too.
2026-10-21 05:53:23 +02:00
Abdessamad Derraz 734643b8fc chore: merge audit fixes (site)
# Conflicts:
#	wiki/testing-guide.md
2026-10-21 05:53:15 +02:00
Abdessamad Derraz 9f1dbaa6c2 chore: merge audit fixes (packcli)
# Conflicts:
#	README.md
#	scripts/generate_pack.py
#	scripts/pipeline.py
2026-10-21 05:51:12 +02:00
Abdessamad Derraz 53da2a95d2 chore: merge audit fixes (docs)
# Conflicts:
#	tests/test_doc_surfaces.py
#	wiki/installer.md
#	wiki/testing-guide.md
2026-10-21 05:50:25 +02:00
Abdessamad Derraz 43c6df58ee chore: merge audit fixes (region)
# Conflicts:
#	install.ps1
#	install.sh
2026-10-21 05:48:59 +02:00
Abdessamad Derraz a67265e903 fix: let the site search land on cards
The search splits a page at its headings. Emulator cards were titled in
bold and platform systems folded into blocks under one heading, so a
pasted hash opened the top of a page or the systems overview of a
100-block platform page. Each file card is now a level 4 heading kept
out of the table of contents, and each platform system a heading.

Card lines that list other pages (platforms, packs, source citations,
collection paths) leave the search index: 24.7 MB down to 18.1 MB, the
names, hashes and notes still searchable.
2026-10-21 05:47:31 +02:00
Abdessamad Derraz f4a225488d chore: merge audit fixes (scrapers) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz eb44e3f898 chore: merge audit fixes (profiles) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 73d73e0f2a chore: merge audit fixes (profsync) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz e03fe338ee chore: merge audit fixes (tests) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 89c533e9ba fix: detect Batocera and Knulli by build files
No Batocera build writes /etc/batocera-version: batocera-system.mk:21-22
(batocera.linux 38845725, unchanged since batocera-30) writes
/usr/share/batocera/batocera.version. Knulli (knulli-linux 57afa662) adds
OS_NAME=knulli to os-release, writes /usr/share/knulli/knulli.version and
removes /usr/share/batocera, and no Knulli source writes /etc/knulli-release.
Neither was detected. A Knulli board is named and not narrowed by
Batocera's per-board lists, and an unlisted board is named.
2026-10-21 05:44:27 +02:00
Abdessamad Derraz 5858ebf53e docs: restate the recipe store counts
The reimport kept the sets holding missing catalog dumps: 24,101 entries
in 2,148 distinct recipes.
2026-10-21 05:38:58 +02:00
Abdessamad Derraz edda080c59 fix: read muOS RetroArch from its own config
muOS runs RetroArch on a config copied from its default, which reads
/run/muos/storage/bios, bound to MUOS/bios on the second card when that
card holds one, and loads cores from /opt/muos/share/core (MustardOS/internal
f41f4a89, script/control/retroarch.sh:8-20, script/device/bind.sh:116-150,
retroarch.default.cfg:2806,3151). The installer wrote to /mnt/mmc/MUOS/bios
and --installed-cores found no core. The config now decides both.
2026-10-21 05:35:45 +02:00
Abdessamad Derraz bdcc8f0fb6 fix: lower the pipeline floor to Python 3.9
The 3.12 guard rested on generate_readme f-strings that have since been
rewritten. Every script compiles under CPython 3.9.25, 3.10 and 3.11, and
only generate_pack's parenthesized context managers fail under 3.8. Under
3.9 the offline pipeline (--skip-packs) completed with no change to the
committed artifacts, the 15 manifests matched 3.12 byte for byte, and the
RomM pack was identical. CI now compiles every script under a real 3.9.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz 15cb52d551 fix: find EmuDeck Windows where its scripts look
functions/all.ps1:4-14 (emudeck-we 037460ef) reads the APPDATA
settings.ps1, else the one under the profile, and takes C:\Emulation when
neither names a folder. Only the first was read and a forced platform fell
back to %USERPROFILE%\Emulation\bios. A network install names its share
through $VolName (JSONtoPS1.ps1:166-172), and a UTF-16 settings file is
decoded.
2026-10-21 05:31:35 +02:00
Abdessamad Derraz cdb2cfa6f0 feat: find RetroBat through the folder it records
RetroBat.exe writes its folder to HKCU\SOFTWARE\RetroBat
LatestKnownInstallPath on every start (retrobat-executable 702da66a,
Program.cs:57 and 99, RegistryTools.cs:15-50). Only C:\RetroBat and the
profile folder were looked at, so an install on D: was missed and a forced
--platform retrobat wrote into C:\RetroBat\bios. The value is read
through winreg, or reg.exe under WSL, before the fixed folders.
2026-10-21 05:28:35 +02:00
Abdessamad Derraz 8f86842117 feat: keep romsets holding a missing catalog dump
The recipe snapshot kept only the archives a profile or platform names,
so provenance_report --missing named no set for dumps a public romset
carries (fm7740sx holds the FM77AV40SX DICROM, EXTSUB and KANJI2 ROMs).
The importer now also keeps the sets holding a catalog entry the
collection lacks. Reimported from mame0289 and the cached FBNeo DATs:
MAME 1726 to 2148 recipes, FBNeo 104 to 111, none dropped.
2026-10-21 05:27:42 +02:00
Abdessamad Derraz eb385a4039 docs: note the profile verdict check in the guide 2026-10-21 05:27:09 +02:00
Abdessamad Derraz 309231afc2 docs: say what --skip-packs still runs
The help said "Only regenerate DB and verify" while steps 1b, 4d, 7 and
8 still ran and rewrote derivable.json, README.md and docs/. It now names
the steps it skips and points to --skip-docs.
2026-10-21 05:27:07 +02:00
Abdessamad Derraz b464ed9202 fix: count prose that holds the pin as review work
The triage, --changed-only and --strict counted refs only. Prose that
--realign-prose would move or refuse blocks every recale and bump, and
blastem read 0 to review. Each report now carries that reading. The
triage also gets a header, a closing count, and one sampling notice per
run instead of one per profile.
2026-10-21 05:24:17 +02:00
Abdessamad Derraz 5e70fe7ff5 docs: state each verification rule once
architecture.md drew path_suffix ahead of SHA1 and listed a core
resolution that missed the retroarch list entry and built_by. Both now
link to the one statement that matches common.py. The Recalbox table
read hashMatchMandatory into verify.py, which never reads it, and
md5_composite hashes member contents, not member MD5s
(Zip.cpp:71-96). The validation index is built from the platform's
resolved profiles only. The requirement table gains the fields that
make a platform YAML reversible, and pages that name a platform list
say what changes rather than judging it.
2026-10-21 05:22:38 +02:00
Abdessamad Derraz 9edf785513 docs: refresh installer and release pages
The installer output example showed 5865 files and no Not installable
line, while the RetroArch manifest gives 8069 and omits 11. Exit code 2
also comes from every option argparse refuses. RETROBIOS_REF holds a
file list only until the next release deletes its tag, and large files
always come from large-files. The release commit now carries the
manifests the pipeline regenerates. A RetroPie 4.8 image ships Python
3.7.3, below the bootstrap's floor, and the pages say what to use.
2026-10-21 05:22:28 +02:00
Abdessamad Derraz 32bda05be3 docs: align the testing and profiling guides
The test skeleton indexed verify_platform's dict result as a list and
stopped on KeyError: 0. The verification discipline and the profiling
guide hedged rules the project holds as firm. The profile field table
marked as required fields the schema does not require, and now names
the pin that profile_sync --strict needs.
2026-10-21 05:22:20 +02:00
Abdessamad Derraz 253a41ebc8 feat: list the names --emulator accepts
An unknown --emulator name pointed at --list-cores and counted 441 names
while --list-cores prints 189: the buildbot names --emulator accepts
(mednafen_psx for beetle_psx) were listed nowhere. --list-emulators prints
each accepted name beside the emulator it stands for, and the hint names
it.
2026-10-21 05:22:08 +02:00
Abdessamad Derraz a160a5bc6b fix: name the hash formats --from-md5 accepts
A value of another length was looked up as an MD5 and reported NOT FOUND,
a SHA-256 was dropped, and an input with no hash ended on "No valid
hashes found in input". Digests are now read by length (CRC32, MD5 and
its 29-digit Batocera form, SHA-1, SHA-256), SHA-256 is looked up in the
database, and any other value is refused with the formats accepted.
2026-10-21 05:20:59 +02:00
Abdessamad Derraz 7bcd41e0c4 fix: let taiwan and asia-ntsc answer each other
kronos tags the ST-V BIOS epr-19854.ic8 taiwan, mednafen_stv tags it
asia-ntsc, the SMPC area Mednafen names for Taiwan (ss.cpp:1119,
2363-2364). As siblings under asia they never matched, so a request in
one spelling dropped the file tagged in the other, and --region
taiwan,us dropped the Saturn sega_101.bin. AREA_TERRITORIES relates
each area to the territory it serves, china for asia-pal included, in
region.py and its installer copy.
2026-10-21 05:20:08 +02:00
Abdessamad Derraz 38af19091d fix: compare pack paths, not only counts
Step 5 compared (ok, total) and the core file count. RetroArch, Lakka
and RetroPie read 526/526 on both sides while verify credited four paths
below SGB1.sfc and SGB2.sfc that the pack never writes: a declared file
blocks them as a directory. verify now places the platform files in the
builder's order with the same path rules, and a user_provided or
external declaration keeps its slot on both sides, so no core file takes
it in the report only.

verify --destinations writes the paths each pack holds by the report,
and step 5 compares them with the archive members (notes and data tree
members set aside). Every emulator profile, in each build it has, is
built in memory (PackRecord, nothing read or written) and compared with
verify --emulator path by path. Cost measured here: about 1 s per
platform archive and 11 s for the 930 emulator packs.
2026-10-21 05:18:02 +02:00
Abdessamad Derraz 8a19d647a8 fix: pass -v and every short flag to install.py
A [Parameter()] attribute made install.ps1 an advanced script: its common
parameters took -v (-Verbose), -vb, -d and -ea before install.py saw them,
and the scriptblock form printed PowerShell's own VERBOSE lines. The
wrapper now forwards $args as typed.
2026-10-21 05:16:52 +02:00
Abdessamad Derraz 1392f00c8d fix: replace a link at a standalone copy's name
A link already at the destination was skipped without a word and counted
as "dir not found" while the folder existed, so the emulator kept the file
the link pointed at. The link now gives way to the file, written beside it
and renamed over the name, and a name leading out of the folder is a named
failure.
2026-10-21 05:16:16 +02:00
Abdessamad Derraz 9c2f49720d feat: flag a changed .info of a profiled core
The coreinfo area compared .info names only: a firmwareN_path added to a
profiled core's .info came out OK. check_freshness now compares each
profiled .info's git blob with platforms/_core_info.json, recorded on
2026-10-21 from the current listing, and --record-coreinfo CORE records
one as reviewed.
2026-10-21 05:12:34 +02:00
Abdessamad Derraz fbc8b1877e fix: build the site in pull request validation
mkdocs was installed by the deploy alone, after the merge, and wiki/
was not a validation path: a dead link or a stray heading failed the
deploy on main with nothing in front of it. The test job now runs the
deploy's site build, and every site input triggers validation.
2026-10-21 05:08:37 +02:00
Abdessamad Derraz 5fc2423ca5 fix: apply --required-only to data directories
The Required packs and manifests carried every data tree, the FreeIntv
overlays included. A data_directories entry now accepts required: false,
and the platform, emulator and manifest walks leave such a tree out under
--required-only. FreeIntv reads its overlays only with the multi-screen
option and draws a placeholder keypad without one (src/libretro.c:294-296,
350-369).
2026-10-21 05:05:49 +02:00
Abdessamad Derraz c87c73d0a8 fix: replace the system icon cache whole
The cache was truncated and dumped in place. A concurrent build read
half a file, took it for empty, and an offline run then wrote {} over
every recorded icon. The file is now replaced through a scratch file,
merged with what another run recorded meanwhile, and left alone when
nothing new was learned.
2026-10-21 05:05:46 +02:00
Abdessamad Derraz 83d49d93a8 fix: name every --all-variants pack in its check
--verify-packs after an --all-variants build rebuilt only the full pack's
name and opened it alone. The five narrowed packs of the same run passed
without a line. Each one is now named with the hash check the build ran
on its members.
2026-10-21 05:03:42 +02:00
Abdessamad Derraz 82acd55ebc fix: guide installer users with their own command
Names are matched on letters and digits, so the site's display names
work, and a typo gets the closest names instead of a full list. Hints
and advice give back the command the user ran (curl or PowerShell form,
every flag kept), --dest alone follows detection and says whose list it
installs, Ctrl+C at a question exits 130, and --cores-dir names a core
folder. install.ps1 runs in its own scope so irm | iex leaves the
session alone, and an override that causes a fingerprint mismatch is
named. --check lists hash mismatches first, missing required files are
grouped by system, and --help ends with examples.
2026-10-21 04:47:50 +02:00
Abdessamad Derraz 7095aa61e7 fix: make maintainer tool output actionable
--emulator accepts a buildbot core name from a profile's cores:, and a
mistyped target gets the closest names instead of every target. diff_truth
shows destinations and reports unpaired entries as evidence, not removals.
The verify summary counts discrepancies, a shared destination names both
systems, a truth hash conflict names both cores, and pipeline steps carry
their place in the real run instead of N/8. Every GitHub API caller takes
the token from github_token(), which falls back to gh auth token.
2026-10-21 04:42:01 +02:00
Abdessamad Derraz 6ee138a819 docs: point users to --check and exact pack files
Troubleshooting, the FAQ and the download page sent people who only ran
the installer to verify.py, which reads the repository, not their BIOS
folder. Download links now name the exact asset of the recorded release,
the --dest example names the BIOS folder on the card, the slice layout
left with the last release that used it, and the shown verify output is
what the tool prints.
2026-10-21 04:34:04 +02:00
Abdessamad Derraz 621f4fcdee fix: give fork pull requests the hash report
A fork's GITHUB_TOKEN is read-only, so the comment never arrived and a
valid file turned red. The report now goes to the job summary for every
pull request, and the comment and labels run only where the token can
write. validate_pr points an oversized file to an issue instead of a
platform setting a contributor cannot use.
2026-10-21 04:25:25 +02:00
Abdessamad Derraz 348c56e94c docs: put the project's working rules in the wiki
Principles, collection rules and the inverted model in architecture.md, research and citation rules in profiling.md, platform and scraper rules in their pages, tool conventions in tools.md, test and commit expectations in testing-guide.md, installer constraints in installer.md. Each rule sits where a reader looks for it, with its reason. Also fixes the stale release-job line, the suite command without -t ., and the unsourceable definition.
2026-10-21 04:03:31 +02:00
Abdessamad Derraz 392aacb207 fix: hash a large download through its own handle
On Windows the partial copy of a file of 16 MB or more is locked with msvcrt.locking, a LockFile region: the locking process cannot access that region through a second handle (Microsoft LockFile remarks). Hashing reopened the path, the read failed with a sharing violation, and every large file was reported as 'folder not writable' (issue 82). Both digests are now read in one pass through the handle that holds the lock. A read-only file already at the destination had the same misleading report: on Windows its attribute is now cleared before the verified copy replaces it, and a refusal that remains is named for what it is.
2026-10-21 03:46:14 +02:00
Abdessamad Derraz cf54730a17 fix: lay out RetroPie standalone packages
RetroPie's YAML listed no standalone_cores, so every package was laid
out as a libretro core and the standalone files of amiberry, dolphin,
hatari, mame and ppsspp never reached the pack. Packages without the
lr- prefix (bar the retroarch frontend) are now standalone, and names
shipped both ways are dual-mode.
2026-10-20 23:04:42 +02:00
Abdessamad Derraz b2b8dcd62c feat: read platform launchers in verify and packs
A names_searched list is one place the launcher fills with the first name it finds, so a core file it would never open once another name of the list is held is not reported missing. A place no declared file and no packed core file fills receives, under the launcher's name, a file the core accepts by content. The layer is applied inside find_undeclared_files, which verify and the pack builder both read, and a test reads their source to keep it there.
2026-10-20 21:26:48 +02:00
Abdessamad Derraz 8660892d60 docs: list the RetroPie and RetroDECK tests 2026-10-20 21:13:17 +02:00
Abdessamad Derraz 15ffeddfe3 feat: read RetroPie launchers from scriptmodules
The scraper walks each package's functions the way RetroPie runs them in one shell (assignments, arrays, for loops, brace expansion, $md_id, calls into other packages) and records every link, copy, move and setting that names $biosdir, the addEmulator flags (sdltrs -romfile3), the BIOS read outside $biosdir (basilisk --rom, gngeopi -B) and the launch scripts that exit without their file (fs-uae kick13.rom, reicast dc_boot.bin).

Each package also gets a package record with its rp_module_repo and the systems it registers, which maps a package to the build it compiles. The schema gains package, repo, systems and the moves and package kinds. platforms/retropie.yml is regenerated at 46128cd2: only the launchers section is new.
2026-10-20 21:13:09 +02:00
Abdessamad Derraz fe9a081cfb feat: read Recalbox launchers from configgen
The scraper now records, from configgen at the pinned tag, the names each Recalbox launcher tries and where it hands them: the Hatari TOS cascades (standalone, hatari and hatarib cores), GSplus, Amiberry kickstarts, the PCSX2 region choice, the Dolphin IPL presence test, xemu, the mupen64plus 64DD IPL, the Mini vMac boot disk and the o2em_bios option. The system_directory libretroRetroarch gives each core, and the per-system overrides after it, are passes_path records.
2026-10-20 20:59:51 +02:00