Commit Graph
1377 Commits
Author SHA1 Message Date
Abdessamad Derraz b89383ba68 fix: open the gap page on what is missing
The page opened on a slogan and four tiles that read Verified (100%)
for 5,437 of 5,438 files and Missing 0, above a table of red cells. It
now opens on what it covers and one sentence that says every count of
the platforms' own lists and points to the emulator files below.
2026-10-21 10:16:14 +02:00
Abdessamad Derraz 69965ebb25 feat: link each platform and system to its data
An emulator page linked its YAML and api/v1 JSON, platform and system
pages nothing. Each platform page now links its YAML, its own
api/v1/platforms/<id>.json and its install manifest, and each system
page its api/v1/systems/<slug>.json, both validated by the envelope
schema.
2026-10-21 10:15:11 +02:00
Abdessamad Derraz 2239df669f fix: address a hashed file card by its hash
An emulator card's anchor changed when a second entry with the same
path was added, so cited links landed at the page top. A card whose
entry declares a hash is now addressed by it, and the first card of a
path keeps the plain file-<path> id as an alias.
2026-10-21 10:12:26 +02:00
Abdessamad Derraz 73ff57468f fix: page descriptions and urls match the page
A core that loads no file no longer promises paths and hashes in its
snippet, a description is cut at a word and keeps the underscores of
its code spans, and the JSON-LD url is encoded as mkdocs encodes the
canonical link and the sitemap.
2026-10-21 10:09:26 +02:00
Abdessamad Derraz fac8685819 fix: give both hashes in the untested table
A row said the declared hash was contradicted by the local file and
named neither. The table now prints the hashes the platform declares
and the path and hashes of the copy the collection holds.
2026-10-21 10:06:52 +02:00
Abdessamad Derraz 377e53f4c0 feat: show load_from, built_by and pin fields
load_from, found_by, on_mismatch, built_by, pin_frozen and
version_source were in the profiles and on no page: Dolphin's
gba_bios.bin showed a path under system/ although the libretro core reads
it from its save directory, and mame0278 did not say only Recalbox ships
it.
2026-10-21 10:05:18 +02:00
Abdessamad Derraz e65ece743d fix: name where a held profile entry sits
164 entries wore the in repo badge with no copy named: they are held as
a ZIP member or in a data directory, which the database does not index.
held_copy() finds that copy and the card names it. storage: large_file
no longer counts as held by itself: kenji-nx's user-picked firmware
package read as held although nothing collects it.
2026-10-21 10:02:50 +02:00
Abdessamad Derraz 45db77cfd0 fix: link a system to its maker's folder
The database keeps one path per content, so a system placed by the
copy deduplication kept linked to another maker: nintendo-nes and
nintendo-fds to Arcade > FBNeo, sega-mastersystem and
sinclair-zx-spectrum to Other > zesarux. Every folder holding a copy now
votes, an id that spells one console folder links to it, and a maker's
folder outranks the folders that file copies by emulator.
2026-10-21 09:54:56 +02:00
Abdessamad Derraz bc2233e725 fix: write truth systems in a stable order
Systems a core covers without a file were added while walking a set,
so the hash seed placed them and two runs wrote dist/truth/<platform>.yml
in two orders. The normalized system map was built from a set too.
2026-10-21 06:57:49 +02:00
Abdessamad Derraz 6c2204a7a8 chore: merge tooling follow-up
# Conflicts:
#	tests/test_audit_regressions.py
2026-10-21 06:50:20 +02:00
Abdessamad Derraz 768b42e807 fix: fold case in the site gap report too 2026-10-21 06:50:10 +02:00
Abdessamad Derraz 378a9e4da5 chore: merge audit fixes (integrity)
# Conflicts:
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/retropie.json
#	install/rocknix.json
#	install/romm.json
#	scripts/generate_pack.py
#	wiki/tools.md
2026-10-21 06:48:42 +02:00
Abdessamad Derraz 4ba935374e fix: keep pack order outside contested paths
Ranking contested core claims sorted the whole gap report, which
reordered every manifest. Only the claims on one contested path move now,
grouped where the first of them stood, and two spellings of a path are
one group where the filesystem folds case (RetroBat's 64DD_IPL_DEV.bin).

An entry without a destination (EmuDeck's system:md5 whitelist) no
longer counts as a pack path in verify, and step 5 prints one status for
counts and paths together. A full offline run compared all 12 archives
and 930 emulator packs path by path, every one equal.
2026-10-21 06:47:44 +02:00
Abdessamad Derraz 85a44d9bf7 chore: merge amiberry and hatari re-profile
# Conflicts:
#	README.md
#	emulators/amiberry.yml
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/rocknix.json
2026-10-21 06:21:30 +02:00
Abdessamad Derraz 76e44a74b4 refactor: drop comments that restate the code
Thirty one-line labels in verify.py, generate_site.py and
generate_truth.py named the statement under them ("# Load registry",
"# Copy branding assets"). Comments that give a reason stay, as do the
module section headers. A test reads the three files for new ones.
2026-10-21 06:20:50 +02:00
Abdessamad Derraz 3613c0927b docs: open the download page on its content
The which-pack page announced itself ("This page picks the right pack
for a setup") before naming a pack. The sentence goes, the pointer to
Getting started stays.
2026-10-21 06:18:59 +02:00
Abdessamad Derraz 566d36275f fix: count alias readers in a file's requirers
Ymir reads sega_101.bin or saturn_bios.bin, so the sega_101.bin entry
carries saturn_bios.bin as an alias. kronos, yabause and yabasanshiro
open saturn_bios.bin and fall back to sega_101.bin: they were in that
entry's _cores but not its _required_by, while their saturn_bios.bin
entry had them requiring it. A reader that requires one of an entry's
aliases now requires the entry. No reader is added, and same-name
entries kept apart by path are left alone. No export changes.
2026-10-21 06:18:32 +02:00
Abdessamad Derraz 5ce7ab8386 fix: self-check a ref in every repository
A prose citation of the port's own lines was read only in the upstream file that a source_ref attributes the path to, and a range past that file's end came back GONE although the port carries it.
2026-10-21 06:18:00 +02:00
Abdessamad Derraz f19a8f99d3 fix: read composite zip md5s apart in diff_truth
Recalbox hashes a .zip by its members. diff_truth set that composite
against the profile's container md5 and reported neocdz.zip and
neocd.zip as mismatches. _hash_mismatch now drops it through
comparable(), the rule pair_entries and the export already apply.
2026-10-21 06:12:08 +02:00
Abdessamad Derraz b8baf92f1d fix: give atomic writes the umask mode
mkstemp creates its scratch 0600 and os.replace kept that mode, so
database.json, the manifests, the packs, the recipe snapshots and the
large-file cache were readable by their owner only. set_plain_mode
gives the scratch 0o666 less the umask, as open() would, before each
rename. A test walks every function that renames a mkstemp file.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 66937d21d7 docs: install the site packages for contributors
The contributor install line left out mkdocs-material and
pymdown-extensions, so pipeline step 8b (render site) was skipped.
The ranges are the ones deploy-site.yml installs, and a test reads
them from the workflow.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 57ac252482 feat: check the profiles a branch adds or edits
profile_sync gains --changed (added or edited profiles, uncommitted
ones included) and --added, both against the merge base of --base
(origin/main by default). CONTRIBUTING runs --changed --strict instead
of one --emulator call per new profile, and the CI job runs --added
in place of its own diff loop.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 1b3763e58f feat: read the voidpoint.io gitlab
eduke32 and fury declare https://voidpoint.io/terminx/eduke32 and were
skipped as an unsupported host. The instance serves the GitLab v4 API
(/api/v4/projects/terminx%2Feduke32 answers 200) and /-/raw/ paths.
2026-10-21 06:09:33 +02:00
Abdessamad Derraz b784a73d5c fix: name a repository the forge renamed
GitHub, GitLab and Gitea redirect a renamed repository, and every call
follows the redirect: AloysHF/NicaiEmu answers as AloysHF/CoolbarEmu
and the profile read as current. profile_sync now reads the name the
forge answers under and counts a move as review work.
2026-10-21 06:09:31 +02:00
Abdessamad Derraz d6fc66274f fix: let check_freshness read the profile report
check_freshness ran profile_sync with --json and the detection flags,
which profile_sync refuses, so --profiles always reported an unreadable
report. The version check now lives on the report and travels in the
JSON and markdown outputs; check_freshness asks for --json
--check-version only.
2026-10-21 06:09:15 +02:00
Abdessamad Derraz 734643b8fc chore: merge audit fixes (site)
# Conflicts:
#	wiki/testing-guide.md
2026-10-21 05:53:15 +02:00
Abdessamad Derraz a1a9da59af chore: merge audit fixes (export)
# Conflicts:
#	exports/recalbox.json
#	exports/retropie.json
2026-10-21 05:53:05 +02:00
Abdessamad Derraz 9f1dbaa6c2 chore: merge audit fixes (packcli)
# Conflicts:
#	README.md
#	scripts/generate_pack.py
#	scripts/pipeline.py
2026-10-21 05:51:12 +02:00
Abdessamad Derraz 53da2a95d2 chore: merge audit fixes (docs)
# Conflicts:
#	tests/test_doc_surfaces.py
#	wiki/installer.md
#	wiki/testing-guide.md
2026-10-21 05:50:25 +02:00
Abdessamad Derraz 43c6df58ee chore: merge audit fixes (region)
# Conflicts:
#	install.ps1
#	install.sh
2026-10-21 05:48:59 +02:00
Abdessamad Derraz a67265e903 fix: let the site search land on cards
The search splits a page at its headings. Emulator cards were titled in
bold and platform systems folded into blocks under one heading, so a
pasted hash opened the top of a page or the systems overview of a
100-block platform page. Each file card is now a level 4 heading kept
out of the table of contents, and each platform system a heading.

Card lines that list other pages (platforms, packs, source citations,
collection paths) leave the search index: 24.7 MB down to 18.1 MB, the
names, hashes and notes still searchable.
2026-10-21 05:47:31 +02:00
Abdessamad Derraz f4a225488d chore: merge audit fixes (scrapers) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 73d73e0f2a chore: merge audit fixes (profsync) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz aa36ec544d fix: name every gap verify counts
verify --platform printed N optional core files and N game data files
missing and named none, even with --verbose. --verbose now lists them as
MISSING (optional) and MISSING (game data), and without it the count line
says --verbose lists them, as the line about cores with no external file
already did.
2026-10-21 05:40:11 +02:00
Abdessamad Derraz 28435230a7 fix: read each named core's own entry in recalbox
An element's mandatory flag came from the one truth entry its md5 paired
with. saturn/saturn_bios.bin names yabause, yabasanshiro and kronos,
which require it, and was paired with the beetle_saturn entry, which
does not. Each named core's entries for the file, found by name, now
decide, except a namesake elsewhere in the same tree (bluemsx KANJI.rom
under Shared Roms is not the one under MSX2+ - European).
2026-10-21 05:39:17 +02:00
Abdessamad Derraz 82a3e44afd fix: settle a slot in any case where case folds
declaration_settles tested the exact-case name before the slot, so on a
filesystem that folds case a declaration spelled otherwise never settled
the core's file: RetroBat's 32X_G_BIOS.BIN left BlastEm's 32X_G_BIOS.bin
undeclared, and verify counted ten such entries as core files. A slot a
case-insensitive platform fills now settles the entry whatever the
spelling (case_folded_slots), the slot arbitration judging its bytes as
for the exact name. Lakka and RetroPie, which keep both names, are
unchanged. Measured: 10 entries settle on RetroBat, 21 on RetroArch.

generate_site.py builds its gap report through cross_reference() without
the folded slots; it needs folded_declared_slots() passed in the same way
as the command line now does.
2026-10-21 05:38:11 +02:00
Abdessamad Derraz 184c822e6e fix: name the file each layer reads in slots.py
Each content-checked line ended with a verdict on the platform's list,
printed at pipeline step 3b. The line now says which emulators read the
other file, and the closing count says the rest follow the platform's
list. The module source carries no such verdict, a test reads it.
2026-10-21 05:33:50 +02:00
Abdessamad Derraz bdcc8f0fb6 fix: lower the pipeline floor to Python 3.9
The 3.12 guard rested on generate_readme f-strings that have since been
rewritten. Every script compiles under CPython 3.9.25, 3.10 and 3.11, and
only generate_pack's parenthesized context managers fail under 3.8. Under
3.9 the offline pipeline (--skip-packs) completed with no change to the
committed artifacts, the 15 manifests matched 3.12 byte for byte, and the
RomM pack was identical. CI now compiles every script under a real 3.9.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz ceb8b752e6 fix: count bundled gaps apart in each heading
cross_reference.py --emulator abuse printed "Gaps: 3 missing" above three
[BUNDLED] files, then "0 missing (need to be sourced)" in the total. The
heading and the total now read one count: bundled files appear as
"N bundled", never among the missing.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz c05d29d941 fix: a held name must carry the declared hash
entry_source took a name or path hit before reading the hashes the entry
declares, so the gap report and the site counted as held a file whose
only homonym has other bytes: zquestclassic's ending.mid and gameover.mid
matched Hydra Castle Labyrinth's. A name or path hit now counts only
when its record carries one of the declared hashes, and the declared
content anywhere in bios/ comes before a data/ copy. Measured over every
profile entry: 254 change, 248 bundled ones (report and site only),
three unsourceable SC-55 revisions now read missing, three data hits
read bios by hash.
2026-10-21 05:32:19 +02:00
Abdessamad Derraz 0dab8f4706 fix: credit file readers through the data trees
FileConsumers resolved profile entries without the data directory
registry the builder uses. Dolphin's hashless codehandler.bin then took
Dolphin-MMJR's copy by name, and the site listed Dolphin and Triforce as
its readers while every pack serves data/dolphin-sys. blueMSX's RS232.ROM
and MSX2P.rom were credited to bios/ copies the same way. The registry
is now read by default, as the builder reads it.

With it, bundled_only.py --strict (pipeline step 4d) names
bios/Other/dolphin-mmjr/codehandler.bin: only dolphin-mmjr's bundled
entry, which already carries its sha1 and size, reaches that copy.
2026-10-21 05:29:42 +02:00
Abdessamad Derraz 5e95574143 fix: the strongest proof takes a shared core path
A destination no platform declares went to the first claimant in profile
order whatever the proof: fbalpha2012 asks for hiscore.dat by name and
took the root of six packs over cores that pin it by sha1. Claims on a
contested path are now ranked by what the collection holds for each (a
held declared hash, then a path or every required archive member, then a
name, then a contradicted hash), and profile order only breaks ties.
Measured: 14 slots change file across the platforms, among them
hiscore.dat, kick13.rom on RetroPie, cgb_boot.bin and dmg_boot.bin on
EmuDeck, and ROCKNIX neogeo.zip and neocdz.zip.
2026-10-21 05:28:10 +02:00
Abdessamad Derraz 8f86842117 feat: keep romsets holding a missing catalog dump
The recipe snapshot kept only the archives a profile or platform names,
so provenance_report --missing named no set for dumps a public romset
carries (fm7740sx holds the FM77AV40SX DICROM, EXTSUB and KANJI2 ROMs).
The importer now also keeps the sets holding a catalog entry the
collection lacks. Reimported from mame0289 and the cached FBNeo DATs:
MAME 1726 to 2148 recipes, FBNeo 104 to 111, none dropped.
2026-10-21 05:27:42 +02:00
Abdessamad Derraz 309231afc2 docs: say what --skip-packs still runs
The help said "Only regenerate DB and verify" while steps 1b, 4d, 7 and
8 still ran and rewrote derivable.json, README.md and docs/. It now names
the steps it skips and points to --skip-docs.
2026-10-21 05:27:07 +02:00
Abdessamad Derraz 738577b5a4 fix: run verify and generate_pack from any dir
Started outside the repository root, both ended on a FileNotFoundError
traceback: the database and the data registry name files from the root.
They now move to the root after reading the paths the caller typed, which
stay relative to where they were typed.
2026-10-21 05:25:29 +02:00
Abdessamad Derraz 76618af460 docs: shorten two script docstrings
_enrich_hashes repeated "ground truth" five times with capitalised
insistence for a three-step lookup, and the MAME ZIP rebuild listed
three guarantees of which two were the same.
2026-10-21 05:24:34 +02:00
Abdessamad Derraz 0457bd8f40 fix: resolve the pipeline output dir once
Every step runs from the repository root, but the lock and the purge of
stale packs read --output-dir from the caller's directory. Started from
scripts/, the run locked and purged scripts/dist/ and wrote to dist/,
leaving the stale packs there. The directory is now made absolute before
the lock, dist/ at the root by default.
2026-10-21 05:24:25 +02:00
Abdessamad Derraz 82d5ff77cf docs: sentence-case and split README paragraphs
Quick Install was the one Title Case heading left in the published
text. The Files sentence was glued to the paragraph above it, two
paragraphs before the table whose column it describes. It now names
the column and sits just above the table.
2026-10-21 05:24:25 +02:00
Abdessamad Derraz 8eda797562 fix: a hashless entry is a path in any mode
Whether the frontend compares content was decided per platform. EmuDeck
(md5) declares neocdz.zip with no hash and checkBIOS.sh never names it,
yet the proven claims of geolith and FBNeo were dropped and the pack
shipped a neocdz.zip without 000-lo.lo. checked_by_path() decides per
entry: no hash of the kind the frontend compares (any of md5, sha1,
crc32 under RomM's rule) and no zipped_file means a path. Measured: the
EmuDeck neocdz.zip slot and three RetroBat DS slots (dsi_bios7.bin,
dsi_bios9.bin, firmware.bin) now take the file the cores prove.
2026-10-21 05:24:19 +02:00
Abdessamad Derraz b464ed9202 fix: count prose that holds the pin as review work
The triage, --changed-only and --strict counted refs only. Prose that
--realign-prose would move or refuse blocks every recale and bump, and
blastem read 0 to review. Each report now carries that reading. The
triage also gets a header, a closing count, and one sampling notice per
run instead of one per profile.
2026-10-21 05:24:17 +02:00