Links to this site get shared on Discord, Reddit and forums, where a
page with no Open Graph tags renders as a grey rectangle. The pages
already carry a per-page title and description, so a theme override
fills the tags from those; mkdocs-material would otherwise only emit
them through its social plugin, which pulls in Pillow and CairoSVG to
render a preview image these pages do not need.
Decorated site pages carry the generation stamp twice: once as the
markdown footer and once as a rendered element. write_if_changed knew
only the first, so every page was rewritten on every run for the clock
alone.
The comparison is what makes the deploy-site freshness guard a real
staleness check rather than a guaranteed failure, and it had no tests.
generate_emulator_page carried a 228-line loop body rendering one file
row from thirty-odd optional fields, which put its complexity at 141,
more than twice the next function in the repository. The row renderer
and its badge strip are now their own functions and the page function
sits at 68.
_forge_sources went from 18 to 32 when it learned to pair each
repository with its own revision; the pairing is now its own function
and the caller is back under the threshold.
Verified by regenerating the site and diffing: all 505 pages are
identical apart from their generation timestamps.
The job carried if: false, which blocks workflow_dispatch as well as
push, so there was no way to cut a release through CI at all and the
comment described a temporary state that had become permanent.
Releasing is deliberate: the push trigger is gone and the job runs when
someone dispatches it. The seven-day rate limit stays as the guard
against dispatching twice, and concurrency no longer cancels a run that
may be midway through uploading assets.
The cached hashes were rebuilt by iterating a set, so their order in
each database entry followed set hashing rather than a declared one. A
run with a warm cache rewrote all 7,850 entries with no content change.
The order is now the one compute_hashes returns, and a test holds a
warm-cache run byte-identical to a --force rehash.
validate_pr.py inspects paths chosen by whoever opened the pull
request, and it hashed the file before deciding whether it was a
symlink. A link to /dev/zero was read until the job timed out, and a
link out of the checkout was hashed and reported as though its target
had been contributed. The shape is now settled first, and a test that
used to hang the run covers it.
The gate had no tests at all, and neither did the 3DS crypto reached
by dynamic import from validation.py: RSA PKCS#1 v1.5, AES-128-CBC and
ECDSA over GF(2^233), all written by hand. Coverage goes from 0 to 95%
on the curve, 0 to 55% on the gate, 9 to 35% on the rest. The curve
tests check against the published SEC 2 parameters rather than against
the module: the generator satisfies the curve equation and the group
order takes it to infinity.
The database drops the duplicate PS3UPDAT.PUP entry and regains
adler32 on every file. RetroPie gains an install manifest, so the
download table reports its size instead of a dash.
Each name is one a platform or the buildbot uses for a core the
profile already documents; without it the reverse index cannot reach
the profile and target filtering skips it.
A region- or target-filtered build wrote to the same filename as the
full one. Both filters now appear in the pack and manifest names, and
verify.py accepts --region for emulator and system reports so the
coverage figures come from the same selection the builder used.
A profile whose builds live in separate repositories keys source,
upstream and source_commit by build mode. The site flattened the URLs
but read a single scalar revision, so a libretro fork could be pinned
to the standalone commit, and binding the object form into SQLite
failed outright once ymir adopted it. URL and revision are now read as
pairs.
The site also published a sitemap nothing pointed at, so robots.txt is
generated alongside it.
Two builds of the same pack from the same inputs produced different
archives. Of 67 members, 65 were already identical: only README.txt
and manifest.json differed, both stamped with the wall clock by
writestr and the second carrying a generated timestamp. Generated
members now use the epoch the archive rebuilder already applies, and
the timestamp comes from the database snapshot the pack was built
from, so the same data yields the same bytes.
Install manifests skipped archived platforms, which is why RetroPie
had none; archived means upstream is no longer scraped, not that the
packs stopped shipping. A target-filtered manifest also had no record
of its filter beyond the filename, so it carries one the way a
region-filtered manifest already does.
Two entries claimed bios/Sony/PlayStation 3/PS3UPDAT.PUP with different
SHA1s. Preserving large-file entries matched on path and keyed on
SHA1, so replacing a firmware revision on disk left the old entry
pointing at a path that now serves other bytes. A preserved entry whose
path the scan has already claimed is dropped, and validate_schemas
refuses a database where one path carries two entries.
Separately, a run without --force rebuilt each cached entry from a
hand-written list of four digests and wrote it back without adler32,
so one such run stripped the digest from every file permanently. A
cache entry missing any digest is now a miss.
Loading the emulator profiles is the most expensive step of every
command here, and all forty call sites used the pure-Python scanner
while libyaml sat unused in the same wheel. One shared yaml_load picks
the C loader when pyyaml ships it: the 375 profiles parse in 0.18s
instead of 1.39s, and verify --platform retroarch drops from 2.17s to
0.73s. The loader class is the same restricted one safe_load uses.
es_bios.xml was parsed straight from the network while install.py
already refused a document declaring entities; both now share one
guard. Scrapers reach it through a single path bootstrap in the
package rather than two ad-hoc ones.
verify.py and generate_pack.py must reach the same verdict on the same
file, and CLAUDE.md calls any divergence critical, but each spelled out
mode == 'existence' in its own words. Both now ask nativemode whether
the frontend reads the file's bytes, which is the one fact the rest
follows from, and a test holds the two answers together.
The BaseScraper contract the wiki asks contributors to implement had no
caller anywhere, so nothing proved compare_with_config, has_changes or
test_connection still worked.
A pack is the platform baseline plus what its cores need, so a profile
change alters pack contents; build.yml did not watch emulators/. Its
release notes read git log -15 on a depth-1 clone, which returns one
commit, and its test step ran a single module out of nineteen.
validate.yml ignored install.sh and install.ps1, so a PR touching only
a bootstrap skipped the test that pins them to install.py.
checkout and setup-python were pinned to v6 in two workflows and v7 in
the others; jsonschema was imported by validate_schemas.py and
declared nowhere, and requires-python claimed 3.10 while both
bootstraps accept 3.8.
Every other manifest field was treated as hostile input, but
standalone_copies targets were only length-checked before being
expanded and written to: a traversal component or a symlink already
sitting at the destination sent the copy outside the directory the
user opted into. Targets are now validated like the other paths and a
symlinked destination is never followed.
RETROBIOS_BASE_URL serves the manifest and the files it declares, so
it now has to be HTTPS the way both bootstraps already require of the
installer URL; loopback stays open for the end-to-end tests.
install.ps1 left TLS at the Windows PowerShell 5.1 default, which
GitHub refuses, so the download failed before any hash was checked.
check_local read every file once per declared digest, single threaded.
One read now feeds both, across the same pool the downloads use.
RetroPie had no manifest, so the one-line installer answered 'unknown
platform' for a frontend whose packs do ship.
LICENSE covered the whole repository, so the MIT grant read as
covering 9.9 GB of third-party firmware. It now states its scope and
NOTICE describes the files it does not cover, with a removal channel.
The FAQ cited Connectix and Accolade for redistribution when both
decided intermediate copying during reverse engineering, presented fair
use as settled, read section 1201(f) as a general permission, and
listed abandonware among legal doctrines. Each claim is now stated at
the strength it actually has, and the weakest ground is named.
Four files, each with a verifiable origin.
logo.tga and logo_red.tga come from the loader's own resources/logo.h:
hook.c's fread maps FILE_LOGO_TGA to logoLL or logoLLRed depending on
which path was opened, one array per file, so the extraction is exact.
Both decode as valid RLE TGA at their declared lengths.
lgp.zip and timetrav.zip carry crc 0 in Hypseus and in Daphne upstream,
so MAME vouches for them instead. lgp maps onto MAME's set by base name,
confirmed by the five members whose bytes already agreed, and the four
that were missing came from the same set. timetrav is MAME's timetrv:
tt061891.bin, 262144 bytes, crc a3d44219.
The four Sega fonts stay open, and the profile now says why: the loader
embeds one 128x256 atlas and one metrics table answering all four paths,
so it substitutes for the originals rather than reproducing either.
A CI checkout omits every file over 50 MB, so verify and generate_pack
resolve those database entries against a disk that does not hold them and
report them missing. The generated README then stops matching the
committed one for a reason that has nothing to do with staleness.
restore_large_files.py writes them back from the release cache, matched by
SHA1 rather than by name, and only where the path is gitignored and
absent. The site workflow runs it, and refreshes the data directories, before
generating.
Sixty-nine of the seventy-one sets Hypseus Singe loads, assembled from
atoms retrieved from public archives and rebuilt with torrentzip under
the member names its source expects.
Every set is anchored: the profile already carried each member with its
CRC32, read from Hypseus' own source, so a candidate is either an exact
match or discarded. Sixty-seven verified that way. For gpworld and
firefoxa, Hypseus declares crc 0 and gives no anchor, so MAME's recipe
for the same game vouched for the bytes instead: 17/17 and 6/6.
lgp and timetrav stay out. Hypseus declares crc 0 for both, MAME covers
only 5 of lgp's 9 members and has no timetrav at all, so nothing can
vouch for them and half-verified is not verified.
The six Lindbergh font and logo files gain has_builtin: true. hook.c
serves them from resources/logo.h and resources/font.h unless built-ins
are disabled, so they are not a gap the collection can close.
The pages described workflows, an installer pinning and a CI permission
model that are not the ones in the tree. Corrected: the release process,
the CI table, the resolution chain and its statuses, the hash-mismatch
policy per verification mode, the archive convention, and the romset
recipe store.
The FAQ keeps the project's own reading of the legal question rather
than a version that reaches no conclusion.
vba_m carried md5 values taken from docs.libretro.com while its entries
declare validation: [size], which is what src/gba/GBA.cpp:3262-3272 and
src/gb/GB.cpp:2172-2192 actually check. Only the source decides a hash,
so they are gone.
The profile becomes an alias of vbam, and the buildbot name vba-m moves
into its cores: list so target filtering still resolves it.
PR validation keeps its four jobs, its comment and its labels. The two
inline schema heredocs become validate_schemas.py --source-only, the
changed-file list is NUL-separated so a path with a space survives, and
the whole suite runs instead of one module.
The offline pipeline, manifest regeneration and mkdocs build stay out of
the pull-request path: they belong to the site workflow, which only runs
on main, and stacking them on every PR does not fit the free-tier
budget.
Deploy Site gains schema validation and rendered-site validation, and
checks that the committed README and CONTRIBUTING match what the
generator produces. write_if_changed compares with the timestamp line
stripped, so that check reports staleness rather than the clock.
The metadata behind the verifier, the pack builder and the site is now
served as static files: a versioned JSON API, CSV extracts, a SQLite
snapshot, and a catalog carrying a SHA-256 for each artifact.
The gaps dataset covers both layers behind a layer column. It previously
held one row, the single platform-verification anomaly, while the page
offering it as a download led with the emulator-level count.
source_ref renders as a permalink pinned to the revision the profile
cites. When a profile declares two repositories, a path that belongs to
neither by name is left as plain code: a citation without a link still
names the file and the lines, a link to the wrong repository does not.
The table filter, focus outlines and tap targets are progressive
enhancement; the pages work without them.
The manifest is untrusted input. It is now read under a size limit and
validated before use: destinations are contained below the BIOS root,
repo_path stays inside bios/, release_asset is a basename, hashes match
their shape, declared totals match their lists, and every entry has a
download source.
Downloads stream against the declared size, are checked by SHA-256 then
SHA-1, land in a per-process temporary file and are installed with
os.replace. Copies into standalone-emulator directories are opt-in with
--standalone-copies so a detection never writes outside the selected
tree.
Both bootstraps verify install.py against an embedded SHA-256 before
running it, and require the Python version install.py actually needs.
A target that publishes no core list is a target with no filter, not a
broken manifest: rejecting it disabled --target for the whole platform.
bios/Sega/Triforce/.variants/segaboot.gcm was never a SegaBoot image:
its header reads "(C) 1999-2001 Nintendo" and it matches Redump's
"Triforce (ARCADE BOOTROM)". crediar's Dolphin fork reads the IPL from
GC/<region>/IPL_TRI.bin and SegaBoot from Triforce/segaboot.gcm, two
separate paths, so it moves to GC/USA/IPL_TRI.bin.
segaboot.gcm.e475c7dc is SegaBoot v3.11, the sha1 emulators/triforce.yml
already declared with no local file.
spec128.zip.1a524bfa is rebuilt from the FinalBurn Neo ZX Spectrum
recipe and roms already held, closing one of the pinned archives listed
as unobtainable.
Fourteen files a profiled core loads and the collection lacked.
Eight OpenH264 2.4.1 binaries for ruffle, from ciscobinary.openh264.org.
Cisco serves only the bzip2 artifact; the emulator loads the
decompressed library under the plain name. All eight sha256 match both
emulators/ruffle.yml and ruffle's own table in
video/external/src/decoder/openh264.rs.
Six Lindbergh runtimes: libCg.so and libCgGL.so from NVIDIA's
Cg-3.1_April2012_x86.tgz, libCg2.so from Cg-2.0_Jan2008_x86.tar.gz, and
the three libraries the loader vendors in libs/.
Its logo and font files are not external: hook.c:655-709 serves them
from resources/logo.h and resources/font.h unless built-ins are
disabled. segaboot and lindbergrc have no public source.
A profile's contents: block and a DAT both state, per set, the members
one emulator version expects. Torrentzip makes archive bytes a function
of that list alone, so a recipe plus the roms reproduces the archive
exactly.
MAME ships its -listxml as a release asset and FBNeo keeps its dats in
its repository, so neither needs a browser. The importer streams the
311 MB document with a sliding window, resolves romof parents in two
passes, drops undumped members, and accumulates versions instead of
replacing them. Identical recipes shared across versions are stored once
with dats listing every version that agrees: 22 MAME versions give 23679
entries for 1726 distinct recipes, 18 MB down to 2.2 MB.
Snapshots live in recipes/ because load_provenance_snapshots reads every
provenance/*.json as a dump catalogue, and a recipe is not one.
1113 archives now reproduce byte for byte, against 175 from profiles
alone, and spec128.zip is rebuilt from roms already held.
A declared hash that the local dump contradicts is not one situation. An
existence platform never reads the bytes, so withholding the file lets an
upstream list error remove something the frontend would have loaded; a
hash platform would reject it, so shipping it is pointless.
The mode now decides, at every point that had an opinion: pack building,
core complement, emulator packs, manifests, conformance and
_intentional_hash_exclusion. verify.find_undeclared_files follows, since
verify and generate_pack must agree file for file.
Also here: resolution reports which evidence matched rather than a flat
"exact", a path or filename can no longer override a declared hash, and
safe_extract_zip treats a Windows backslash as the separator it is
instead of refusing the archive.
Four archives carried member names no MAME version uses. Each is decided
against MAME 0.289 at the revision the profiles cite:
manager.zip 1 -> 01 crvision.cpp:957
sys573.zip 700a01(gchgchmp).22g -> , ksys573.cpp:3575
cedmag.zip Magnet-Master-VID-E03.BIN cedar_magnet.cpp:1040
advision.zip b225__/b8223__ prefixes cut advision.cpp:456,459
Content is untouched: every member keeps its crc32 and size. The
archives are rebuilt with torrentzip so their bytes depend only on their
contents, which is the repository convention (51 of 60 sampled archives)
and what two of these four already were.
Batocera, RetroBat and RetroDECK pin the older naming, so the upstream
archives stay verbatim under .variants/ and resolve_local_file serves
each layer the copy it expects.
YAML 1.1 reads an unquoted 01 as the integer 1 and 81 as 81, so the
manager set stopped naming the file its driver opens and the API
published "cores": [2048] as a number where consumers match strings.
Ground truth from MAME 0.289 at the revision the profile cites:
src/mame/vtech/crvision.cpp:957 loads "01" and "23".
The root cause was the scraper: _hash_merge wrote these names through an
f-string, so quoting the profiles alone would be undone on the next
refresh. _yaml_scalar now quotes every name, archive and description it
writes.
JSON Schemas for the database, install and pack manifests, target
manifests, site API envelopes and stats, plus the semantic invariants a
schema cannot express: declared totals matching their lists, no
destination both installed and omitted, database keys matching their
sha1. validate_site.py checks the rendered HTML for metadata, headings,
image alternatives, duplicate ids and unresolved local links.
Pack manifests are read from inside the generated archives, where
generate_pack writes them, rather than from a dist/ glob that matches
nothing.
Emulator and platform schemas gain additionalProperties: false, and
cores[] plus contents[].name must be strings: an unquoted 81 or 01 in
YAML parses as a number and stops matching the upstream name.
Abdessamad Derraz