Commit Graph
77 Commits
Author SHA1 Message Date
Abdessamad Derraz 57ac252482 feat: check the profiles a branch adds or edits
profile_sync gains --changed (added or edited profiles, uncommitted
ones included) and --added, both against the merge base of --base
(origin/main by default). CONTRIBUTING runs --changed --strict instead
of one --emulator call per new profile, and the CI job runs --added
in place of its own diff loop.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 9f1dbaa6c2 chore: merge audit fixes (packcli)
# Conflicts:
#	README.md
#	scripts/generate_pack.py
#	scripts/pipeline.py
2026-10-21 05:51:12 +02:00
Abdessamad Derraz 53da2a95d2 chore: merge audit fixes (docs)
# Conflicts:
#	tests/test_doc_surfaces.py
#	wiki/installer.md
#	wiki/testing-guide.md
2026-10-21 05:50:25 +02:00
Abdessamad Derraz f4a225488d chore: merge audit fixes (scrapers) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 73d73e0f2a chore: merge audit fixes (profsync) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 5858ebf53e docs: restate the recipe store counts
The reimport kept the sets holding missing catalog dumps: 24,101 entries
in 2,148 distinct recipes.
2026-10-21 05:38:58 +02:00
Abdessamad Derraz bdcc8f0fb6 fix: lower the pipeline floor to Python 3.9
The 3.12 guard rested on generate_readme f-strings that have since been
rewritten. Every script compiles under CPython 3.9.25, 3.10 and 3.11, and
only generate_pack's parenthesized context managers fail under 3.8. Under
3.9 the offline pipeline (--skip-packs) completed with no change to the
committed artifacts, the 15 manifests matched 3.12 byte for byte, and the
RomM pack was identical. CI now compiles every script under a real 3.9.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz 8f86842117 feat: keep romsets holding a missing catalog dump
The recipe snapshot kept only the archives a profile or platform names,
so provenance_report --missing named no set for dumps a public romset
carries (fm7740sx holds the FM77AV40SX DICROM, EXTSUB and KANJI2 ROMs).
The importer now also keeps the sets holding a catalog entry the
collection lacks. Reimported from mame0289 and the cached FBNeo DATs:
MAME 1726 to 2148 recipes, FBNeo 104 to 111, none dropped.
2026-10-21 05:27:42 +02:00
Abdessamad Derraz 309231afc2 docs: say what --skip-packs still runs
The help said "Only regenerate DB and verify" while steps 1b, 4d, 7 and
8 still ran and rewrote derivable.json, README.md and docs/. It now names
the steps it skips and points to --skip-docs.
2026-10-21 05:27:07 +02:00
Abdessamad Derraz b464ed9202 fix: count prose that holds the pin as review work
The triage, --changed-only and --strict counted refs only. Prose that
--realign-prose would move or refuse blocks every recale and bump, and
blastem read 0 to review. Each report now carries that reading. The
triage also gets a header, a closing count, and one sampling notice per
run instead of one per profile.
2026-10-21 05:24:17 +02:00
Abdessamad Derraz 5e70fe7ff5 docs: state each verification rule once
architecture.md drew path_suffix ahead of SHA1 and listed a core
resolution that missed the retroarch list entry and built_by. Both now
link to the one statement that matches common.py. The Recalbox table
read hashMatchMandatory into verify.py, which never reads it, and
md5_composite hashes member contents, not member MD5s
(Zip.cpp:71-96). The validation index is built from the platform's
resolved profiles only. The requirement table gains the fields that
make a platform YAML reversible, and pages that name a platform list
say what changes rather than judging it.
2026-10-21 05:22:38 +02:00
Abdessamad Derraz a160a5bc6b fix: name the hash formats --from-md5 accepts
A value of another length was looked up as an MD5 and reported NOT FOUND,
a SHA-256 was dropped, and an input with no hash ended on "No valid
hashes found in input". Digests are now read by length (CRC32, MD5 and
its 29-digit Batocera form, SHA-1, SHA-256), SHA-256 is looked up in the
database, and any other value is refused with the formats accepted.
2026-10-21 05:20:59 +02:00
Abdessamad Derraz 9c2f49720d feat: flag a changed .info of a profiled core
The coreinfo area compared .info names only: a firmwareN_path added to a
profiled core's .info came out OK. check_freshness now compares each
profiled .info's git blob with platforms/_core_info.json, recorded on
2026-10-21 from the current listing, and --record-coreinfo CORE records
one as reviewed.
2026-10-21 05:12:34 +02:00
Abdessamad Derraz 5fc2423ca5 fix: apply --required-only to data directories
The Required packs and manifests carried every data tree, the FreeIntv
overlays included. A data_directories entry now accepts required: false,
and the platform, emulator and manifest walks leave such a tree out under
--required-only. FreeIntv reads its overlays only with the multi-screen
option and draws a placeholder keypad without one (src/libretro.c:294-296,
350-369).
2026-10-21 05:05:49 +02:00
Abdessamad Derraz 83d49d93a8 fix: name every --all-variants pack in its check
--verify-packs after an --all-variants build rebuilt only the full pack's
name and opened it alone. The five narrowed packs of the same run passed
without a line. Each one is now named with the hash check the build ran
on its members.
2026-10-21 05:03:42 +02:00
Abdessamad Derraz 7095aa61e7 fix: make maintainer tool output actionable
--emulator accepts a buildbot core name from a profile's cores:, and a
mistyped target gets the closest names instead of every target. diff_truth
shows destinations and reports unpaired entries as evidence, not removals.
The verify summary counts discrepancies, a shared destination names both
systems, a truth hash conflict names both cores, and pipeline steps carry
their place in the real run instead of N/8. Every GitHub API caller takes
the token from github_token(), which falls back to gh auth token.
2026-10-21 04:42:01 +02:00
Abdessamad Derraz 348c56e94c docs: put the project's working rules in the wiki
Principles, collection rules and the inverted model in architecture.md, research and citation rules in profiling.md, platform and scraper rules in their pages, tool conventions in tools.md, test and commit expectations in testing-guide.md, installer constraints in installer.md. Each rule sits where a reader looks for it, with its reason. Also fixes the stale release-job line, the suite command without -t ., and the unsourceable definition.
2026-10-21 04:03:31 +02:00
Abdessamad Derraz 545e3f3ba1 feat: record each export change before and after
The page named the entries a change touched and never the values, so
a maintainer could not check a correction. The report now carries
changes, one object per change with kind, entry, field, old and new,
read from the platform entry and from the value the render writes
(written() per format: record value for System.dat and RomM, one md5
where a single one is checked). The existing label lists are derived
from it. The page writes "md5 X becomes Y" and names each required
flag with its new value. A test parses the original and updated
System.dat and es_bios.xml and checks the list equals the diff.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 0b4f98ce79 feat: report entries no examined core opens
The export keeps every entry a platform declares. An entry the truth
does not pair was still read in 1189 of 2776 cases (alias, hash, a
type: directory tree, a core outside the resolved set), so absence of
a pairing proves nothing. Each report now lists, under
removal_candidates, the unpaired entries that no profile of the cores
the platform runs answers by name, path tail, hash, directory or data
directory, with the cores examined and a reason. A core the platform
names without a profile, or a ROM set under a core that opens sets by
name, leaves the verdict undetermined. The exports page states the
count and lists the entries per platform.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 0a309c79a8 chore: record the two new derivable dumps
The MAME 0.289 device sets made two more catalogued dumps derivable
from held ones: derivable.json lists 67, and the tools page now says
so.
2026-10-20 19:35:57 +02:00
Abdessamad Derraz 8de45b6f45 feat: generate MAME device sets from -listxml
A machine that instantiates a device loads the device's ROMs from <shortname>.zip (src/emu/device.cpp:140-152), and a missing required ROM stops that machine (romload.cpp:529-570, 667-671). The BIOS root scan never sees these sets. mame_hash_scraper --devices --listxml reads them from the -listxml of the release mame.yml pins, refuses another release, and cites each set at the ROM_START block holding its ROMs. The BIOS root merge also wrote refs without the upstream prefix and would have rewritten all 171 of them in mame.yml.
2026-10-20 15:12:23 +02:00
Abdessamad Derraz ef8b372810 chore: drop copies only an install provides
A file that arrives with the emulator is already on the user's machine
and the packs leave it out. 4558 collected copies (1.4 GB) were reached
only by such entries, with no platform, other profile entry or install
manifest needing them. They are removed, and two release-asset copies
leave the ignore list. bundled_only.py now lists every copy of a listed
file, since the database keeps one path per content, and pipeline step
4d fails while one remains.
2026-10-20 10:17:08 +02:00
Abdessamad Derraz ca13e6036d feat: publish the corrected platform files
export_native now writes into exports/ by default, and beside each
platform's corrected file a report naming every change, where the run
printed five per kind. A new site page lists each platform with its
original file, a download of the corrected one and every corrected,
filled or withheld value. Each platform page links to its row.
2026-10-20 09:20:02 +02:00
Abdessamad Derraz 9b630a39bb fix: hold the new profiles to the shared checks
BizHawk's NDS firmware slot accepts six dumps, two of which the collection holds, and the profile named one. Kega Fusion's US Mega CD BIOS covers the Americas. Sonic 3 A.I.R. data is installed by Batocera in unpacked form. izApple2 downloads its ProDOS images when a card is added, which is no install. The Lakka list now equals a case-keeping RetroArch install less what Lakka's packages install, and the profile hash contradiction count leaves out entries no pack ships. The pipeline table names step 8b.
2026-10-20 08:12:03 +02:00
Abdessamad Derraz bb19a6d5cf feat: profile what RetroBat and RetroPie run
100 emulator names the two platforms run had no profile, so their packs carried nothing for them. Each was read from the platform's own launcher or scriptmodule: the same program as an existing profile takes the name in cores:, a different build gets its own profile read from that version, a launcher or self-contained game goes to remove_cores, and the rest are new profiles written from their source or shipped binary. The files they need that the collection lacked are collected from the upstream distributions. Bungie and splitwolf trees are read by path and stay out of dedup.
2026-10-20 06:14:10 +02:00
Abdessamad Derraz 36e30beac0 docs: give the troubleshooting topics headings
Twenty-one topics were bold lines ending in a colon: MkDocs gave them no anchor to link from an issue and left them out of the page contents. They are headings now, and the prose test refuses the form.
2026-10-20 06:08:03 +02:00
Abdessamad Derraz a07821d810 feat: list files only an emulator install provides
A file installed with its emulator is already on the user's machine and the packs leave it out. bundled_only.py lists the collected copies nothing else needs: no platform declares them, no entry of any build needs them, no install manifest ships them.
2026-10-20 06:05:20 +02:00
Abdessamad Derraz 810396941d feat: write the release notes from the record
The pack table, its parts sentence and the install commands were typed by hand from figures release.json already held. A copied cell or an old collection total at the top is how issue 81 happened. release_notes.py renders them and appends the hand-written account of what changed.
2026-10-20 04:51:56 +02:00
Abdessamad Derraz 8bd2d319b2 feat: the pipeline runs the schema check first
PR validation runs validate_schemas.py --source-only and the pipeline did not, so a profile with an unquoted numeric core name passed every step and failed CI on push. It is step 0 now, skipped where jsonschema is not installed.
2026-10-20 04:50:04 +02:00
Abdessamad Derraz 18e093182d docs: name the registries behind assets and dedup
The guides said files over 50 MB become release assets and that RPG Maker and ScummVM were the trees kept out of dedup. .gitignore decides the first, four committed files are over 50 MB, and NODEDUP_DIRS holds eleven trees.
2026-10-20 04:35:02 +02:00
Abdessamad Derraz 4231902a6c docs: write prose without dashes or semicolons
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.

validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
2026-10-19 06:47:31 +02:00
Abdessamad Derraz 583031eda5 refactor: plain documentation theme for the site
No gradient banner, no coloured stripes or dots beside cards, no lifting cards, pill badges or small-caps headers. File blocks are separated by a rule, tables keep the theme's own size, and headings no longer repeat the site name the header bar already shows. Classes nothing emits were removed.
2026-10-19 05:34:52 +02:00
Abdessamad Derraz 4bfe4eef2c docs: name both methods a scraper must define 2026-10-19 04:16:52 +02:00
Abdessamad Derraz 4521edca59 fix: name every pin a bump changes 2026-10-18 19:33:11 +02:00
Abdessamad Derraz 8c326bc31e docs: list region_audit and the slots flags 2026-10-18 16:10:06 +02:00
Abdessamad Derraz 6dc373fcfd fix: refuse two output forms in diff_truth 2026-10-18 08:40:01 +02:00
Abdessamad Derraz 3116a57617 docs: document every option the scripts accept
Fifty options appeared in no page. fileless_audit also gains --all, so
its example sweeps instead of naming three settled profiles, and it and
download.py --info now fail on an unknown name or an unreachable source.
2026-10-18 07:29:16 +02:00
Abdessamad Derraz 23ed3e92ab docs: mend a torn sentence in the tools page 2026-10-18 07:25:06 +02:00
Abdessamad Derraz 731faee6e1 docs: tie the tools page counts to their artifacts 2026-10-18 07:20:34 +02:00
Abdessamad Derraz 2977615f99 docs: list every exporter and target scraper 2026-10-18 07:19:58 +02:00
Abdessamad Derraz d6d03617db fix: exit 1 when a hash lookup finds nothing
The documented example looked up a test fixture hash, printed NOT FOUND
and exited 0. The docs now use a hash the database holds and say which
line forms a hash file accepts.
2026-10-18 07:19:01 +02:00
Abdessamad Derraz e089cb86e3 docs: align tools and release pages with the code 2026-10-17 21:20:26 +02:00
Abdessamad Derraz 7fe65936e0 fix: apply verify --strict in every mode 2026-10-17 18:26:57 +02:00
Abdessamad Derraz 5cc45794f4 docs: show installer runs as they really print 2026-10-17 15:06:04 +02:00
Abdessamad Derraz 0fc71793a9 feat: mark catalog gaps a held dump contains 2026-10-17 13:40:05 +02:00
Abdessamad Derraz f6b808b1d9 feat: let verify fail on a missing required file 2026-10-17 12:36:35 +02:00
Abdessamad Derraz a4da6d205f feat: profile the MAME releases recalbox builds
Recalbox ships libretro/mame built at 0.258 and at 0.278, and no
profile described either release. mame_hash_scraper gains --release,
which reads the BIOS root sets at one mamedev tag and writes a profile
pinned to it; mame0258 and mame0278 are generated from it, every
source_ref anchored at its tag.
2026-10-17 03:31:11 +02:00
Abdessamad Derraz 42950a553c feat: refresh data trees once per online pipeline
Step 2 refreshed the data directories, then the pack build and the
manifest build refreshed them again, so a tree tracking a moving
branch could differ between the pack and the manifest. generate_pack
gains --no-refresh-data, which packs the cached trees and still
fetches release assets; the online pipeline passes it to both builds.
2026-10-17 01:59:20 +02:00
Abdessamad Derraz 0408c5d359 fix: state and check the 3.12 build floor 2026-10-16 16:05:10 +02:00
Abdessamad Derraz 0c39fc479d fix: drop the --include-extras flag nothing read 2026-10-06 12:27:21 +02:00