verify.py reaches these through a dynamic import and nothing exercised
them, so a signature check that accepted everything would have looked
exactly like one that worked.
Console dumps are personal data and cannot be committed, but the
verifiers only read structure and signatures: the fixtures are built
from the layouts in unique_data.cpp and otp.cpp, signing with a key the
test owns and passing the matching public key in through the keys file.
That covers the region-change detection in SecureInfo_A, the embedded
LFCS in movable.sed, and the OTP path down to the sect233r1 certificate
including the pre-v5 expiry endianness.
crypto_verify goes from 9 to 89 percent. Disabling the OTP hash check
and the movable.sed magic check each fails a test.
The pack integrity tests failed whenever another run was writing
dist/, reporting a corrupt archive when the only fact established was
that somebody else was building. Which test went red depended on how
far along that build was. They skip now, the way validate_schemas
already does.
install.py fetches a file from its repo_path or from a release asset.
Resolution can land on a file the database does not index, and the
entry then shipped with neither: a line in the download list that can
only ever fail. Those are recorded as omitted instead, which is what
the installer already knows how to report, and a test holds the
committed manifests to it.
validate_schemas read dist/ while a build was writing it and reported
a half-written pack as 'File is not a zip file'. It takes the shared
lock --verify-packs uses, and says so when a build holds it.
A required-only build is narrower than the platform declares by
design, like the source-restricted variants already handled, so the
full expectation must not be applied to it. --region is also refused
alongside --manifest-targets, which carries no region dimension.
Decorated site pages carry the generation stamp twice: once as the
markdown footer and once as a rendered element. write_if_changed knew
only the first, so every page was rewritten on every run for the clock
alone.
The comparison is what makes the deploy-site freshness guard a real
staleness check rather than a guaranteed failure, and it had no tests.
The cached hashes were rebuilt by iterating a set, so their order in
each database entry followed set hashing rather than a declared one. A
run with a warm cache rewrote all 7,850 entries with no content change.
The order is now the one compute_hashes returns, and a test holds a
warm-cache run byte-identical to a --force rehash.
validate_pr.py inspects paths chosen by whoever opened the pull
request, and it hashed the file before deciding whether it was a
symlink. A link to /dev/zero was read until the job timed out, and a
link out of the checkout was hashed and reported as though its target
had been contributed. The shape is now settled first, and a test that
used to hang the run covers it.
The gate had no tests at all, and neither did the 3DS crypto reached
by dynamic import from validation.py: RSA PKCS#1 v1.5, AES-128-CBC and
ECDSA over GF(2^233), all written by hand. Coverage goes from 0 to 95%
on the curve, 0 to 55% on the gate, 9 to 35% on the rest. The curve
tests check against the published SEC 2 parameters rather than against
the module: the generator satisfies the curve equation and the group
order takes it to infinity.
A region- or target-filtered build wrote to the same filename as the
full one. Both filters now appear in the pack and manifest names, and
verify.py accepts --region for emulator and system reports so the
coverage figures come from the same selection the builder used.
A profile whose builds live in separate repositories keys source,
upstream and source_commit by build mode. The site flattened the URLs
but read a single scalar revision, so a libretro fork could be pinned
to the standalone commit, and binding the object form into SQLite
failed outright once ymir adopted it. URL and revision are now read as
pairs.
The site also published a sitemap nothing pointed at, so robots.txt is
generated alongside it.
Two builds of the same pack from the same inputs produced different
archives. Of 67 members, 65 were already identical: only README.txt
and manifest.json differed, both stamped with the wall clock by
writestr and the second carrying a generated timestamp. Generated
members now use the epoch the archive rebuilder already applies, and
the timestamp comes from the database snapshot the pack was built
from, so the same data yields the same bytes.
Install manifests skipped archived platforms, which is why RetroPie
had none; archived means upstream is no longer scraped, not that the
packs stopped shipping. A target-filtered manifest also had no record
of its filter beyond the filename, so it carries one the way a
region-filtered manifest already does.
Two entries claimed bios/Sony/PlayStation 3/PS3UPDAT.PUP with different
SHA1s. Preserving large-file entries matched on path and keyed on
SHA1, so replacing a firmware revision on disk left the old entry
pointing at a path that now serves other bytes. A preserved entry whose
path the scan has already claimed is dropped, and validate_schemas
refuses a database where one path carries two entries.
Separately, a run without --force rebuilt each cached entry from a
hand-written list of four digests and wrote it back without adler32,
so one such run stripped the digest from every file permanently. A
cache entry missing any digest is now a miss.
verify.py and generate_pack.py must reach the same verdict on the same
file, and CLAUDE.md calls any divergence critical, but each spelled out
mode == 'existence' in its own words. Both now ask nativemode whether
the frontend reads the file's bytes, which is the one fact the rest
follows from, and a test holds the two answers together.
The BaseScraper contract the wiki asks contributors to implement had no
caller anywhere, so nothing proved compare_with_config, has_changes or
test_connection still worked.
Every other manifest field was treated as hostile input, but
standalone_copies targets were only length-checked before being
expanded and written to: a traversal component or a symlink already
sitting at the destination sent the copy outside the directory the
user opted into. Targets are now validated like the other paths and a
symlinked destination is never followed.
RETROBIOS_BASE_URL serves the manifest and the files it declares, so
it now has to be HTTPS the way both bootstraps already require of the
installer URL; loopback stays open for the end-to-end tests.
install.ps1 left TLS at the Windows PowerShell 5.1 default, which
GitHub refuses, so the download failed before any hash was checked.
check_local read every file once per declared digest, single threaded.
One read now feeds both, across the same pool the downloads use.
RetroPie had no manifest, so the one-line installer answered 'unknown
platform' for a frontend whose packs do ship.
A CI checkout omits every file over 50 MB, so verify and generate_pack
resolve those database entries against a disk that does not hold them and
report them missing. The generated README then stops matching the
committed one for a reason that has nothing to do with staleness.
restore_large_files.py writes them back from the release cache, matched by
SHA1 rather than by name, and only where the path is gitignored and
absent. The site workflow runs it, and refreshes the data directories, before
generating.
The metadata behind the verifier, the pack builder and the site is now
served as static files: a versioned JSON API, CSV extracts, a SQLite
snapshot, and a catalog carrying a SHA-256 for each artifact.
The gaps dataset covers both layers behind a layer column. It previously
held one row, the single platform-verification anomaly, while the page
offering it as a download led with the emulator-level count.
source_ref renders as a permalink pinned to the revision the profile
cites. When a profile declares two repositories, a path that belongs to
neither by name is left as plain code: a citation without a link still
names the file and the lines, a link to the wrong repository does not.
The table filter, focus outlines and tap targets are progressive
enhancement; the pages work without them.
The manifest is untrusted input. It is now read under a size limit and
validated before use: destinations are contained below the BIOS root,
repo_path stays inside bios/, release_asset is a basename, hashes match
their shape, declared totals match their lists, and every entry has a
download source.
Downloads stream against the declared size, are checked by SHA-256 then
SHA-1, land in a per-process temporary file and are installed with
os.replace. Copies into standalone-emulator directories are opt-in with
--standalone-copies so a detection never writes outside the selected
tree.
Both bootstraps verify install.py against an embedded SHA-256 before
running it, and require the Python version install.py actually needs.
A target that publishes no core list is a target with no filter, not a
broken manifest: rejecting it disabled --target for the whole platform.
A profile's contents: block and a DAT both state, per set, the members
one emulator version expects. Torrentzip makes archive bytes a function
of that list alone, so a recipe plus the roms reproduces the archive
exactly.
MAME ships its -listxml as a release asset and FBNeo keeps its dats in
its repository, so neither needs a browser. The importer streams the
311 MB document with a sliding window, resolves romof parents in two
passes, drops undumped members, and accumulates versions instead of
replacing them. Identical recipes shared across versions are stored once
with dats listing every version that agrees: 22 MAME versions give 23679
entries for 1726 distinct recipes, 18 MB down to 2.2 MB.
Snapshots live in recipes/ because load_provenance_snapshots reads every
provenance/*.json as a dump catalogue, and a recipe is not one.
1113 archives now reproduce byte for byte, against 175 from profiles
alone, and spec128.zip is rebuilt from roms already held.
A declared hash that the local dump contradicts is not one situation. An
existence platform never reads the bytes, so withholding the file lets an
upstream list error remove something the frontend would have loaded; a
hash platform would reject it, so shipping it is pointless.
The mode now decides, at every point that had an opinion: pack building,
core complement, emulator packs, manifests, conformance and
_intentional_hash_exclusion. verify.find_undeclared_files follows, since
verify and generate_pack must agree file for file.
Also here: resolution reports which evidence matched rather than a flat
"exact", a path or filename can no longer override a declared hash, and
safe_extract_zip treats a Windows backslash as the separator it is
instead of refusing the archive.
YAML 1.1 reads an unquoted 01 as the integer 1 and 81 as 81, so the
manager set stopped naming the file its driver opens and the API
published "cores": [2048] as a number where consumers match strings.
Ground truth from MAME 0.289 at the revision the profile cites:
src/mame/vtech/crvision.cpp:957 loads "01" and "23".
The root cause was the scraper: _hash_merge wrote these names through an
f-string, so quoting the profiles alone would be undone on the next
refresh. _yaml_scalar now quotes every name, archive and description it
writes.
JSON Schemas for the database, install and pack manifests, target
manifests, site API envelopes and stats, plus the semantic invariants a
schema cannot express: declared totals matching their lists, no
destination both installed and omitted, database keys matching their
sha1. validate_site.py checks the rendered HTML for metadata, headings,
image alternatives, duplicate ids and unresolved local links.
Pack manifests are read from inside the generated archives, where
generate_pack writes them, rather than from a dist/ glob that matches
nothing.
Emulator and platform schemas gain additionalProperties: false, and
cores[] plus contents[].name must be strings: an unquoted 81 or 01 in
YAML parses as a number and stops matching the upstream name.
Abdessamad Derraz