Commit Graph
60 Commits
Author SHA1 Message Date
Abdessamad Derraz 545e3f3ba1 feat: record each export change before and after
The page named the entries a change touched and never the values, so
a maintainer could not check a correction. The report now carries
changes, one object per change with kind, entry, field, old and new,
read from the platform entry and from the value the render writes
(written() per format: record value for System.dat and RomM, one md5
where a single one is checked). The existing label lists are derived
from it. The page writes "md5 X becomes Y" and names each required
flag with its new value. A test parses the original and updated
System.dat and es_bios.xml and checks the list equals the diff.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 0b4f98ce79 feat: report entries no examined core opens
The export keeps every entry a platform declares. An entry the truth
does not pair was still read in 1189 of 2776 cases (alias, hash, a
type: directory tree, a core outside the resolved set), so absence of
a pairing proves nothing. Each report now lists, under
removal_candidates, the unpaired entries that no profile of the cores
the platform runs answers by name, path tail, hash, directory or data
directory, with the cores examined and a reason. A core the platform
names without a profile, or a ROM set under a core that opens sets by
name, leaves the verdict undetermined. The exports page states the
count and lists the entries per platform.
2026-10-20 20:00:54 +02:00
Abdessamad Derraz 0a309c79a8 chore: record the two new derivable dumps
The MAME 0.289 device sets made two more catalogued dumps derivable
from held ones: derivable.json lists 67, and the tools page now says
so.
2026-10-20 19:35:57 +02:00
Abdessamad Derraz 8de45b6f45 feat: generate MAME device sets from -listxml
A machine that instantiates a device loads the device's ROMs from <shortname>.zip (src/emu/device.cpp:140-152), and a missing required ROM stops that machine (romload.cpp:529-570, 667-671). The BIOS root scan never sees these sets. mame_hash_scraper --devices --listxml reads them from the -listxml of the release mame.yml pins, refuses another release, and cites each set at the ROM_START block holding its ROMs. The BIOS root merge also wrote refs without the upstream prefix and would have rewritten all 171 of them in mame.yml.
2026-10-20 15:12:23 +02:00
Abdessamad Derraz ef8b372810 chore: drop copies only an install provides
A file that arrives with the emulator is already on the user's machine
and the packs leave it out. 4558 collected copies (1.4 GB) were reached
only by such entries, with no platform, other profile entry or install
manifest needing them. They are removed, and two release-asset copies
leave the ignore list. bundled_only.py now lists every copy of a listed
file, since the database keeps one path per content, and pipeline step
4d fails while one remains.
2026-10-20 10:17:08 +02:00
Abdessamad Derraz ca13e6036d feat: publish the corrected platform files
export_native now writes into exports/ by default, and beside each
platform's corrected file a report naming every change, where the run
printed five per kind. A new site page lists each platform with its
original file, a download of the corrected one and every corrected,
filled or withheld value. Each platform page links to its row.
2026-10-20 09:20:02 +02:00
Abdessamad Derraz 9b630a39bb fix: hold the new profiles to the shared checks
BizHawk's NDS firmware slot accepts six dumps, two of which the collection holds, and the profile named one. Kega Fusion's US Mega CD BIOS covers the Americas. Sonic 3 A.I.R. data is installed by Batocera in unpacked form. izApple2 downloads its ProDOS images when a card is added, which is no install. The Lakka list now equals a case-keeping RetroArch install less what Lakka's packages install, and the profile hash contradiction count leaves out entries no pack ships. The pipeline table names step 8b.
2026-10-20 08:12:03 +02:00
Abdessamad Derraz bb19a6d5cf feat: profile what RetroBat and RetroPie run
100 emulator names the two platforms run had no profile, so their packs carried nothing for them. Each was read from the platform's own launcher or scriptmodule: the same program as an existing profile takes the name in cores:, a different build gets its own profile read from that version, a launcher or self-contained game goes to remove_cores, and the rest are new profiles written from their source or shipped binary. The files they need that the collection lacked are collected from the upstream distributions. Bungie and splitwolf trees are read by path and stay out of dedup.
2026-10-20 06:14:10 +02:00
Abdessamad Derraz 36e30beac0 docs: give the troubleshooting topics headings
Twenty-one topics were bold lines ending in a colon: MkDocs gave them no anchor to link from an issue and left them out of the page contents. They are headings now, and the prose test refuses the form.
2026-10-20 06:08:03 +02:00
Abdessamad Derraz a07821d810 feat: list files only an emulator install provides
A file installed with its emulator is already on the user's machine and the packs leave it out. bundled_only.py lists the collected copies nothing else needs: no platform declares them, no entry of any build needs them, no install manifest ships them.
2026-10-20 06:05:20 +02:00
Abdessamad Derraz 810396941d feat: write the release notes from the record
The pack table, its parts sentence and the install commands were typed by hand from figures release.json already held. A copied cell or an old collection total at the top is how issue 81 happened. release_notes.py renders them and appends the hand-written account of what changed.
2026-10-20 04:51:56 +02:00
Abdessamad Derraz 8bd2d319b2 feat: the pipeline runs the schema check first
PR validation runs validate_schemas.py --source-only and the pipeline did not, so a profile with an unquoted numeric core name passed every step and failed CI on push. It is step 0 now, skipped where jsonschema is not installed.
2026-10-20 04:50:04 +02:00
Abdessamad Derraz 18e093182d docs: name the registries behind assets and dedup
The guides said files over 50 MB become release assets and that RPG Maker and ScummVM were the trees kept out of dedup. .gitignore decides the first, four committed files are over 50 MB, and NODEDUP_DIRS holds eleven trees.
2026-10-20 04:35:02 +02:00
Abdessamad Derraz 4231902a6c docs: write prose without dashes or semicolons
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.

validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
2026-10-19 06:47:31 +02:00
Abdessamad Derraz 583031eda5 refactor: plain documentation theme for the site
No gradient banner, no coloured stripes or dots beside cards, no lifting cards, pill badges or small-caps headers. File blocks are separated by a rule, tables keep the theme's own size, and headings no longer repeat the site name the header bar already shows. Classes nothing emits were removed.
2026-10-19 05:34:52 +02:00
Abdessamad Derraz 4bfe4eef2c docs: name both methods a scraper must define 2026-10-19 04:16:52 +02:00
Abdessamad Derraz 4521edca59 fix: name every pin a bump changes 2026-10-18 19:33:11 +02:00
Abdessamad Derraz 8c326bc31e docs: list region_audit and the slots flags 2026-10-18 16:10:06 +02:00
Abdessamad Derraz 6dc373fcfd fix: refuse two output forms in diff_truth 2026-10-18 08:40:01 +02:00
Abdessamad Derraz 3116a57617 docs: document every option the scripts accept
Fifty options appeared in no page. fileless_audit also gains --all, so
its example sweeps instead of naming three settled profiles, and it and
download.py --info now fail on an unknown name or an unreachable source.
2026-10-18 07:29:16 +02:00
Abdessamad Derraz 23ed3e92ab docs: mend a torn sentence in the tools page 2026-10-18 07:25:06 +02:00
Abdessamad Derraz 731faee6e1 docs: tie the tools page counts to their artifacts 2026-10-18 07:20:34 +02:00
Abdessamad Derraz 2977615f99 docs: list every exporter and target scraper 2026-10-18 07:19:58 +02:00
Abdessamad Derraz d6d03617db fix: exit 1 when a hash lookup finds nothing
The documented example looked up a test fixture hash, printed NOT FOUND
and exited 0. The docs now use a hash the database holds and say which
line forms a hash file accepts.
2026-10-18 07:19:01 +02:00
Abdessamad Derraz e089cb86e3 docs: align tools and release pages with the code 2026-10-17 21:20:26 +02:00
Abdessamad Derraz 7fe65936e0 fix: apply verify --strict in every mode 2026-10-17 18:26:57 +02:00
Abdessamad Derraz 5cc45794f4 docs: show installer runs as they really print 2026-10-17 15:06:04 +02:00
Abdessamad Derraz 0fc71793a9 feat: mark catalog gaps a held dump contains 2026-10-17 13:40:05 +02:00
Abdessamad Derraz f6b808b1d9 feat: let verify fail on a missing required file 2026-10-17 12:36:35 +02:00
Abdessamad Derraz a4da6d205f feat: profile the MAME releases recalbox builds
Recalbox ships libretro/mame built at 0.258 and at 0.278, and no
profile described either release. mame_hash_scraper gains --release,
which reads the BIOS root sets at one mamedev tag and writes a profile
pinned to it; mame0258 and mame0278 are generated from it, every
source_ref anchored at its tag.
2026-10-17 03:31:11 +02:00
Abdessamad Derraz 42950a553c feat: refresh data trees once per online pipeline
Step 2 refreshed the data directories, then the pack build and the
manifest build refreshed them again, so a tree tracking a moving
branch could differ between the pack and the manifest. generate_pack
gains --no-refresh-data, which packs the cached trees and still
fetches release assets; the online pipeline passes it to both builds.
2026-10-17 01:59:20 +02:00
Abdessamad Derraz 0408c5d359 fix: state and check the 3.12 build floor 2026-10-16 16:05:10 +02:00
Abdessamad Derraz 0c39fc479d fix: drop the --include-extras flag nothing read 2026-10-06 12:27:21 +02:00
Abdessamad Derraz 5c89ac6edc docs: name jsonschema where the build needs it 2026-10-06 09:46:10 +02:00
Abdessamad Derraz 52f682cdd6 feat: publish a large pack as zips, not byte ranges 2026-10-04 18:09:24 +02:00
Abdessamad Derraz b81199ff26 feat: keep cached platform originals in step 2026-10-04 16:36:35 +02:00
Abdessamad Derraz 691ccbfca7 feat: make the native export round-trip a platform file 2026-09-05 17:32:48 +02:00
Abdessamad Derraz c738073f66 fix: move the refs and the pin together or not at all
Recaling refs while the pin stays put produces exactly the state the
all-or-nothing rule exists to prevent: a profile whose refs describe one
revision and whose source_commit names another. The tool manufactured it
on mariani, where three prose runs it could not rewrite kept
bump_commit refusing while eleven refs had already moved.

Asking for both writes is now atomic. The work happens on a copy, which
is promoted only when the pin follows, and a rebase is refused outright
when something visible beforehand will hold the pin: an annotated ref,
one under a mode key, or a prose run whose tokens cannot be located well
enough to rewrite. Where the block only appears after the write, the
copy is discarded and the profile is named on stderr rather than left
half moved.

mariani is back on its pin and stays at four refs to read, which is
honest: three of them have to be rewritten by hand before anything can
advance.
2026-09-04 18:55:15 +02:00
Abdessamad Derraz d124f6c516 feat: watch the profiles that declare no files
An empty file list is the one assertion here that ages unwatched. Nothing
can go missing and no ref can drift, so nothing notices when a core that
embedded everything grows a path. virtualjaguar said "No external BIOS
files are required or loaded by this core" while its source had grown
eleven filenames read from the system directory, and only a reading
found it.

fileless_audit looks for the request itself, the system directory ask, in
the sources each profile already cites. Over the 151 fileless profiles it
named eleven, of which two were covered by data_directories, six carried
an exclusion_note, and three had nothing written down at all: craft
writes its world database in that directory, dice stores the answer in a
variable no other file in the tree names, lutro hands it to the Lua game.
Each now says so.

The check settles: declared files, a declared directory, or a written
answer all end it, so what it reports is the set nobody has read yet. A
test holds the corpus at zero.
2026-09-04 18:13:53 +02:00
Abdessamad Derraz 1a1f19be2d feat: check a MAME ref against the set it names
profile_sync follows content, so a ref that drifted still anchors where
the cited text went. That is drift detection working, and it cannot
answer the only question a MAME romset ref asks: does this line declare
this set. It flagged five refs in one driver file where nineteen were
stale, the fourteen others having been relocatable somewhere plausible.

mame_ref_audit asks the stronger question and found ninety-two across
the four profiles whose upstream still moves: mame 66, mamearcade 18,
mamemess 6, groovymame 2. Each had exactly one declaration to point at.
The frozen generations, mame2009 through mame2016, come out clean, which
is the check saying it finds drift only where drift can happen.

The set name is argument 1 of the machine macro. Matching it anywhere on
the line matches every clone naming it as parent, which is most of a
driver, and comments are stripped first because a declaration can sit
behind one. A set no machine declares is reported as not judgeable, not
wrong: device archives take their DEFINE_DEVICE_TYPE shortname.
2026-09-04 17:48:24 +02:00
Abdessamad Derraz ece637d52d feat: let a profile state that its upstream is gone
Saying those profiles would stay open no matter what was wrong. A dead
forge is a verifiable fact, and a fact can be recorded: upstream_gone
carries why, and the profile stops being an open problem. The refs
describe the last revision anyone could reach, which is all any reader
can ask of them.

The declaration is guarded rather than trusted, because a forge can come
back and a profile that keeps asserting a death nobody rechecks is worse
than one that fails loudly. Declared and unreachable reports as a
recorded fact; declared and answering reports as the contradiction it
is, and the profile has to be read again.

yuzu and suyu carry GitHub's 451. citron carries the loss of
git.citron-emu.org and the squatter now sitting on the .com. Each names
what was probed and when, so the next reader retraces it rather than
repeating it.
2026-09-04 17:25:23 +02:00
Abdessamad Derraz f45050aa5c docs: say what a gone forge and a mirror do 2026-09-04 15:36:10 +02:00
Abdessamad Derraz 3891d897a6 fix: report the writes a dry run would make
--backfill-commits and --realign-prose have always printed what they
would write. --rebase-refs and --bump-commit took the flag and printed
nothing, so the only way to read a plan was to let it happen, and the
recale and the pin had to be done in two full network passes with
--force in between.

Both now plan. The plan runs the production write path over a throwaway
copy rather than a parallel branch, so it cannot drift from the write,
and the planned bump reads the text the recale would have left: a pin
held back by prose the same pass would move is no longer reported as
blocked. One pass does both, recale before bump on each profile.

bump_commit also stopped announcing a rewrite of the pin to the value it
already held. On the corpus that was 126 of 232 announcements, which
buried the 106 profiles that did move.
2026-09-04 14:41:37 +02:00
Abdessamad Derraz 1a96853aee feat: sign the release checksum list
SHA256SUMS.txt sat beside the artifacts it vouches for, so whoever could
rewrite a release rewrote the list with it. The packs were already
reproducible, which answers corruption and lets a third party rebuild an
archive byte for byte; nothing answered a rewritten release.

The list is now signed with an ed25519 key kept for this alone, and the
public half is allowed_signers at the repository root, so verification
does not go through the release page: ssh-keygen -Y verify against the
committed file, then sha256sum --check. Rehearsed on all three outcomes:
a good signature, a tampered pack caught by the sums, a rewritten list
caught by the signature.

The release steps sign and upload the signature, the README points a
downloader at the procedure, and the reproducibility section says what
each half proves. Rotation keeps retired lines so past releases stay
verifiable. Three tests hold the trust root, the signing step and the
documented principal in agreement.
2026-09-04 14:01:05 +02:00
Abdessamad Derraz 626c012a6b docs: document split packs and region filtering 2026-09-04 10:18:14 +02:00
Abdessamad Derraz c00314d479 docs: align the wiki with what the code does
The pages described workflows, an installer pinning and a CI permission
model that are not the ones in the tree. Corrected: the release process,
the CI table, the resolution chain and its statuses, the hash-mismatch
policy per verification mode, the archive convention, and the romset
recipe store.

The FAQ keeps the project's own reading of the legal question rather
than a version that reaches no conclusion.
2026-08-10 13:37:20 +02:00
Abdessamad Derraz 7755367787 docs: document tag-pinned profile handling 2026-08-08 13:17:52 +02:00
Abdessamad Derraz 09d586e387 docs: document the ref anchoring strategies 2026-08-08 12:42:29 +02:00
Abdessamad Derraz de19473f2c docs: document upstream and exporter base modules 2026-08-08 12:15:48 +02:00
Abdessamad Derraz ccb9b3e94f feat: add profile sync against upstream 2026-08-08 11:17:21 +02:00