Commit Graph
114 Commits
Author SHA1 Message Date
Abdessamad Derraz 08f174bf32 docs: name the part marker in the release steps 2026-10-23 07:22:46 +02:00
Abdessamad Derraz 2927c49734 docs: split project reports, pin checker links
Reading this project's own reports gets its own page. Checker code is linked at the revision each platform list was read from. Installer moves to Your setup, legal to Start here, build internals to Contribute, and Reference pages carry a group.
2026-10-23 02:42:19 +02:00
Abdessamad Derraz 123d698e43 feat: build the site with Astro Starlight
generate_site.py writes the Starlight project in website/ and nothing
else: mkdocs.yml, the MkDocs hooks, theme overrides and stylesheet go,
and the wiki is written in Starlight's syntax. Both workflows set up
Node with the official actions/setup-node pinned by commit, run npm ci
from the committed lockfile, build with the strict link validator and
check the result with validate_site.py.

The search weighs a page by what it answers: Start here pages first,
a system page's cards and file names before a guide citing them, deep
guides, launchers, aliases and frozen copies after. Status badges,
hash blocks and the export change lists leave the index. The ranking
sits at the centre of a range where the 13 queries of search_gate.py
all answer, in the search box itself.

839 pages, every URL and 45,611 ids match the MkDocs build. The tests
read the Starlight content and syntax.
2026-10-22 18:47:22 +02:00
Abdessamad Derraz f18cfa1f5c chore: merge the archive recipe follow-ups 2026-10-22 15:10:14 +02:00
Abdessamad Derraz 06e37b36f7 feat: collect the tyrian files emuelec declares
EmuELEC declares 19 files of the Tyrian 2.1 distribution (setup.ini,
netterm.int, the net*.pcx screens...) that bios/ did not hold. They
resolved only at the data directory step, against data/opentyrian, the
cache refresh_data_dirs.py fills from the opentyrian-data.zip asset. verify
gave 544/572 in a checkout holding that cache and 525/572 in one without
it, for the same commit.

The 19 files are taken from the freeware tyrian21.zip (camanis.net, sha1
29827de9), each equal to the cached copy. netset.pcx and tyrset.pcx are
the same bytes under two names, so the folder joins NODEDUP_DIRS. A test
resolves every platform entry with the data directory caches present and
fails on any that only a cache answers.
2026-10-22 15:08:15 +02:00
Abdessamad Derraz be3324f869 feat: read guides and wiki nav from data
Platform guides, device groups, terminal tabs, pack README manuals and
logo credits move from Python tables to platforms/_guides.yml, checked
against the registry on load. A folder row without a path is the
registry's extract_to, written once.

Wiki pages place themselves in the navigation with order and nav_title
in their front matter. Generated docs/ and mkdocs.yml are unchanged.
2026-10-22 15:06:52 +02:00
Abdessamad Derraz 274fc37673 chore: merge the site weight and speed work 2026-10-22 13:46:50 +02:00
Abdessamad Derraz be5be996db feat: hold the site to page and index budgets
validate_site.py fails a page over 1 MB of HTML, a search index over
8.5 MB (1.7 MB gzip), a skipped heading level, a table header without
text, a script from another host and a diagram drawn at runtime. The
limits sit above what the site now measures (0.88 MB, 7.2 MB, 1.43 MB),
each with its reason, and each has a test.
2026-10-22 13:13:55 +02:00
Abdessamad Derraz 2d2addd6d2 fix: align the suite with the site layout
The redesign moved the technical questions to technical-faq.md, made the Get your files rows generated from each platform key and reduced getting-started.md to links. Tests now read those places: the pack-row test reads the rendered page. The new site scripts go through run_cli and refuse abbreviated options. The three maintenance image tools are exempt by name from the build dependency rule: their output is committed and the build only reads it. siteguides.py now triggers the site deploy, and the recipe counts match the store.
2026-10-22 13:11:07 +02:00
Abdessamad Derraz 6353ee53df feat: lay the site out as a reading funnel
Five tabs from Start here to Contribute, a level on every page, a device picker on Get your files that lists every registered platform, and platform pages that open on where the files go, the command, the pack and how to check. FAQ and troubleshooting keep the questions asked before installing, the technical ones move to their own page, and getting started points to the pages that took its content.
2026-10-22 11:50:53 +02:00
Abdessamad Derraz aed958ccd1 docs: document the game data filter 2026-10-22 06:19:13 +02:00
Abdessamad Derraz 841017c887 chore: merge the minui family platforms 2026-10-22 04:09:17 +02:00
Abdessamad Derraz c7be31699f docs: document the minui family platforms 2026-10-22 04:04:40 +02:00
Abdessamad Derraz 44a9c8d9bb docs: list knulli, amberelec and emuelec
The download page drops a row whose platform has no pack in the
recorded release instead of linking a release that lacks it.
2026-10-22 03:42:25 +02:00
Abdessamad Derraz 2d74524d59 feat: file mame software lists and floppy samples
The -listxml names the software lists each machine and slot card
declares and the floppy drives it can instantiate. --devices now writes
one entry per hash/<list>.xml the release ships under the systems of its
placed machines (111 lists, 666 stay with hash/), and files each floppy
sample under the systems whose drives select its form factor
(floppy.cpp:1428-1473, 1759-1767).

Machines a registered platform declares under one of the profile's
systems seed the walk too: bbcb, cdimono1, cdimono2, dragon32, ep64 and
ep128. Unplaced device sets go from 903 to 861.
2026-10-21 12:24:37 +02:00
Abdessamad Derraz d6a0639330 chore: merge audit fixes (packrules)
# Conflicts:
#	scripts/packextras.py
2026-10-21 11:52:28 +02:00
Abdessamad Derraz cd589b2742 chore: merge audit fixes (mame) 2026-10-21 11:48:30 +02:00
Abdessamad Derraz 56e8cbe046 feat: file mame device sets by their machines
The -listxml names, for every machine, its parent (romof), the devices of its default configuration (device_ref) and the cards its slots offer (slotoption). A machine takes the system of the first set on its romof chain a written entry lists, and passes it to each device it can instantiate. A device set reached from one system gets system:, from several the new per-entry systems:, from none neither, and keeps serving every system of the profile. The gap report, the core extras and slots read the entry's own systems, so generate_pack --system and the installer pick by them.
2026-10-21 11:33:39 +02:00
Abdessamad Derraz 3fe88332e7 feat: accept platform system names in --system
generate_pack and verify read --system beside --platform through common.resolve_system_arg, built on platform_system_names, the mapping the manifest's system_names comes from. verify --platform --system now reports on the narrowed pack. Unknown names are refused with suggestions before the database is read.
2026-10-21 10:52:40 +02:00
Abdessamad Derraz 9a0501d9d3 fix: count an uncached release asset as fetched
verify resolved offline, so a release asset absent from bios/ and the
cache read as missing while an online build fetched and packed it. verify
now reads the database record of the registered path and counts such a
declaration present, listed as fetched at build time. --offline keeps the
old count, and the offline pipeline passes it to its verify step.
2026-10-21 10:50:03 +02:00
Abdessamad Derraz 055627f39c chore: merge round 2 fixes (tooling) 2026-10-21 10:31:04 +02:00
Abdessamad Derraz 6409757c39 chore: merge round 2 fixes (docs)
# Conflicts:
#	CONTRIBUTING.md
#	scripts/check_release_assets.py
#	scripts/generate_readme.py
2026-10-21 10:31:04 +02:00
Abdessamad Derraz 648db6bcbf fix: merge mame devices when the pin moves
Step 2a advanced mame.yml's core_version and upstream_commit to a new
release and left the device ROM sets on the previous one. A plain run
that moves the pin now fetches that release's listxml into .cache/dats
and runs the device merge, which checks version and commit itself.
2026-10-21 10:06:13 +02:00
Abdessamad Derraz a74a56abc4 feat: name the profile a missing dump would fill
The acquisition report never read a profile, so XM7's required DICROM.ROM
sat among 2 287 targets at the rank of dumps no core reads. A missing
entry that a profile reads and the collection lacks is now listed first
with the profile, matched by hash or, for a hashless entry with a size,
by name and size. 29 entries on the current snapshots.
2026-10-21 10:05:10 +02:00
Abdessamad Derraz d16f8fd17a fix: accept downloads by cryptographic hash only
A crc32 and a size can be met by crafted bytes, so a remote download
matched on them could land in bios/. verify_content now requires every
sha256, sha1 and md5 the entry declares to match, and an entry declaring
none is reported as not verifiable from the network and nothing is
written. crc32 and size still find a file the database already holds.
2026-10-21 10:04:20 +02:00
Abdessamad Derraz 6c076884ed fix: let auto_fetch prove sha256 and crc32 entries
verify_content read sha1 then md5 only, so an entry pinned by sha256 or
crc32 could be searched and never accepted, and one without any hash was
announced as searched. The strongest declared hash now decides, crc32
counts with its size, the database lookup reads sha256 and crc32, and an
entry with nothing to prove a download by is reported, not searched.
2026-10-21 10:02:05 +02:00
Abdessamad Derraz e9f945857c docs: describe the native export without verdicts 2026-10-21 10:01:46 +02:00
Abdessamad Derraz f3586001ab fix: fail freshness on unknown, name the refresher
check_freshness exited 0 when no upstream answered. An unknown row now
holds the exit code, and a profile skipped by nature is reported skipped
so it does not hold it forever. --only names the valid areas, a stale
native row names its platform, and the summary names refresh_stale.py.
2026-10-21 10:00:36 +02:00
Abdessamad Derraz cc57d44de5 fix: order inherited rescrapes, end on verify
refresh_stale ran a parent's rescrape and its children's truth and
export jobs side by side, so a child read the parent mid-rewrite. A job
now starts once the rescrapes of its ancestors have ended, and a child
no row names gets its export rewritten after them. A rescrape chain ends
on verify.py --platform, whose summary closes the report.
2026-10-21 09:58:58 +02:00
Abdessamad Derraz 563ce1ebf2 fix: read fbneo dats at one checked commit
The FBNeo import recorded the listing's blob sha for bytes fetched from a
branch URL a CDN caches, and --fetch <revision> still read the tip. The
revision is now resolved to a commit, listed and downloaded there, each
DAT is held against its blob sha, and a DAT read at a named revision is
labelled with it without replacing the tip's recorded blobs.
2026-10-21 09:55:50 +02:00
Abdessamad Derraz 12cc71b634 fix: hold the pipeline's fixed outputs for a run
The pipeline locked only --output-dir while it also writes database.json,
install/, README.md and docs/. A run on a scratch output directory could
start beside the run on dist/ and rewrite them. Every run now holds the
repository's .lock first.
2026-10-21 09:53:15 +02:00
Abdessamad Derraz 43337c6719 fix: match release packs by exact name
release_record rebuilt nothing from the name builder: a substring match
took RetroArch_v1.22.2_AndroidArm64V8A_BIOS_Pack.zip for the RetroArch
pack. The record now refuses any pack that is not a registered platform's
full pack and stores the platforms each pack serves, which the readers
use instead of the pack name.
2026-10-21 09:51:55 +02:00
Abdessamad Derraz f2628c1170 docs: drop hand-kept --derive figures from tools 2026-10-21 09:51:51 +02:00
Abdessamad Derraz 6877512af7 feat: carry new upstream names and hashes in json
--detect-new-files and --watch-hashes were printed only by the text report and refused with --json, so check_freshness --profiles never saw a name or hash the cited code gained. Both now live on the report, and check_freshness passes them and marks such a profile stale.
2026-10-21 09:51:14 +02:00
Abdessamad Derraz 6c2204a7a8 chore: merge tooling follow-up
# Conflicts:
#	tests/test_audit_regressions.py
2026-10-21 06:50:20 +02:00
Abdessamad Derraz 378a9e4da5 chore: merge audit fixes (integrity)
# Conflicts:
#	install/batocera.json
#	install/emudeck.json
#	install/lakka.json
#	install/recalbox.json
#	install/retroarch.json
#	install/retrobat.json
#	install/retrodeck.json
#	install/retropie.json
#	install/rocknix.json
#	install/romm.json
#	scripts/generate_pack.py
#	wiki/tools.md
2026-10-21 06:48:42 +02:00
Abdessamad Derraz 57ac252482 feat: check the profiles a branch adds or edits
profile_sync gains --changed (added or edited profiles, uncommitted
ones included) and --added, both against the merge base of --base
(origin/main by default). CONTRIBUTING runs --changed --strict instead
of one --emulator call per new profile, and the CI job runs --added
in place of its own diff loop.
2026-10-21 06:09:55 +02:00
Abdessamad Derraz 9f1dbaa6c2 chore: merge audit fixes (packcli)
# Conflicts:
#	README.md
#	scripts/generate_pack.py
#	scripts/pipeline.py
2026-10-21 05:51:12 +02:00
Abdessamad Derraz 53da2a95d2 chore: merge audit fixes (docs)
# Conflicts:
#	tests/test_doc_surfaces.py
#	wiki/installer.md
#	wiki/testing-guide.md
2026-10-21 05:50:25 +02:00
Abdessamad Derraz f4a225488d chore: merge audit fixes (scrapers) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 73d73e0f2a chore: merge audit fixes (profsync) 2026-10-21 05:44:32 +02:00
Abdessamad Derraz 5858ebf53e docs: restate the recipe store counts
The reimport kept the sets holding missing catalog dumps: 24,101 entries
in 2,148 distinct recipes.
2026-10-21 05:38:58 +02:00
Abdessamad Derraz bdcc8f0fb6 fix: lower the pipeline floor to Python 3.9
The 3.12 guard rested on generate_readme f-strings that have since been
rewritten. Every script compiles under CPython 3.9.25, 3.10 and 3.11, and
only generate_pack's parenthesized context managers fail under 3.8. Under
3.9 the offline pipeline (--skip-packs) completed with no change to the
committed artifacts, the 15 manifests matched 3.12 byte for byte, and the
RomM pack was identical. CI now compiles every script under a real 3.9.
2026-10-21 05:32:50 +02:00
Abdessamad Derraz 8f86842117 feat: keep romsets holding a missing catalog dump
The recipe snapshot kept only the archives a profile or platform names,
so provenance_report --missing named no set for dumps a public romset
carries (fm7740sx holds the FM77AV40SX DICROM, EXTSUB and KANJI2 ROMs).
The importer now also keeps the sets holding a catalog entry the
collection lacks. Reimported from mame0289 and the cached FBNeo DATs:
MAME 1726 to 2148 recipes, FBNeo 104 to 111, none dropped.
2026-10-21 05:27:42 +02:00
Abdessamad Derraz 309231afc2 docs: say what --skip-packs still runs
The help said "Only regenerate DB and verify" while steps 1b, 4d, 7 and
8 still ran and rewrote derivable.json, README.md and docs/. It now names
the steps it skips and points to --skip-docs.
2026-10-21 05:27:07 +02:00
Abdessamad Derraz b464ed9202 fix: count prose that holds the pin as review work
The triage, --changed-only and --strict counted refs only. Prose that
--realign-prose would move or refuse blocks every recale and bump, and
blastem read 0 to review. Each report now carries that reading. The
triage also gets a header, a closing count, and one sampling notice per
run instead of one per profile.
2026-10-21 05:24:17 +02:00
Abdessamad Derraz 5e70fe7ff5 docs: state each verification rule once
architecture.md drew path_suffix ahead of SHA1 and listed a core
resolution that missed the retroarch list entry and built_by. Both now
link to the one statement that matches common.py. The Recalbox table
read hashMatchMandatory into verify.py, which never reads it, and
md5_composite hashes member contents, not member MD5s
(Zip.cpp:71-96). The validation index is built from the platform's
resolved profiles only. The requirement table gains the fields that
make a platform YAML reversible, and pages that name a platform list
say what changes rather than judging it.
2026-10-21 05:22:38 +02:00
Abdessamad Derraz a160a5bc6b fix: name the hash formats --from-md5 accepts
A value of another length was looked up as an MD5 and reported NOT FOUND,
a SHA-256 was dropped, and an input with no hash ended on "No valid
hashes found in input". Digests are now read by length (CRC32, MD5 and
its 29-digit Batocera form, SHA-1, SHA-256), SHA-256 is looked up in the
database, and any other value is refused with the formats accepted.
2026-10-21 05:20:59 +02:00
Abdessamad Derraz 38af19091d fix: compare pack paths, not only counts
Step 5 compared (ok, total) and the core file count. RetroArch, Lakka
and RetroPie read 526/526 on both sides while verify credited four paths
below SGB1.sfc and SGB2.sfc that the pack never writes: a declared file
blocks them as a directory. verify now places the platform files in the
builder's order with the same path rules, and a user_provided or
external declaration keeps its slot on both sides, so no core file takes
it in the report only.

verify --destinations writes the paths each pack holds by the report,
and step 5 compares them with the archive members (notes and data tree
members set aside). Every emulator profile, in each build it has, is
built in memory (PackRecord, nothing read or written) and compared with
verify --emulator path by path. Cost measured here: about 1 s per
platform archive and 11 s for the 930 emulator packs.
2026-10-21 05:18:02 +02:00
Abdessamad Derraz 9c2f49720d feat: flag a changed .info of a profiled core
The coreinfo area compared .info names only: a firmwareN_path added to a
profiled core's .info came out OK. check_freshness now compares each
profiled .info's git blob with platforms/_core_info.json, recorded on
2026-10-21 from the current listing, and --record-coreinfo CORE records
one as reviewed.
2026-10-21 05:12:34 +02:00