A destination can be declared by more than one system, bare in one and
hash-constrained in another. generate_pack resolves that with
_preferred_entries so the constrained sibling claims the destination;
generate_manifest never did, and named whatever answered to the name.
RetroDECK's bios/d2fdc.zip is the case: declared with an md5 in the
arcade system and bare under apple-ii. The pack carried the right
archive, 262 bytes reached through the MAME clone map, while the
manifest sent install.py to a 256-byte Apple II ROM. Downloading the
ZIP and running the installer gave different files.
A test now hashes every manifest entry a platform pins and compares it
against what the platform declares: 3295 entries, on the platforms whose
frontend reads the bytes. It accepts any of several declarations for one
destination, the member-composite MD5 Recalbox pins for arcade archives,
and Batocera's 29-character prefixes. Reinstating the old manifest entry
fails it.
The contributors block is the only part of the README that comes from
the network, and a refused request returned an empty list, which deleted
the section. That happened during a pipeline run and the result was
committed; the freshness check then regenerated the section and failed
on the difference.
An unavailable list now republishes the one already there and says so.
Losing it is a worse answer than a stale one, and it makes an offline
regeneration additive rather than destructive. Reverting the change and
simulating the same outage empties the section again.
Rebuilt from the complete state: 8923 files indexed, 436 emulators, and
coverage up on every platform that gained a profile. The gap count falls
from 9 to 7, the three that closed being the C128 kernal, the Cave Story
build and Supermodel's Games.xml.
Whole pipeline green on this revision: database, provenance, recipes,
data directories, hash scrapers, truth, verify, packs, manifests,
consistency and pack integrity across the twelve platforms.
Thirty profiles from the ES-DE batch, each source-verified by the
session that wrote it and passing the schema and semantic checks. Five
declare no files at all and say why in notes referenced to the code:
NGP.emu, openbor, Plastic, Swan.emu and XeniOS.
Twenty-eight files come with them, every one declared by the profile
that needs it and matching the hash it declares, with no duplicate of
anything already held: the NetherSX2 Turnip Classic assets, the Speccy
machine ROMs, the Virtual Boy homebrew the app bundles, Snes9x EX+'s
bundled game, and sixtyforce's Overrides.plist, which carries a size
and no hash because the binary never checks one.
supermodel gains the four data files it reads from its install tree,
hakux marks its controller map unsourceable now that the asset manager
is known to read inside the package, and nethersx2 sheds the two core
aliases that belong to the Turnip profiles.
The manifests moved and the table did not. Most packs grow by about
1.4 GB: a filename-agnostic core takes any file of the declared shape in
its directory, and melonDS now finds all six regional DSi NAND images
where the anchor previously settled on one. The Android app assets that
left in the same pass were small by comparison.
bios/Other/NXEngine/Doukutsu.exe held a build whose sha1 contradicts the
one the profile declares, and it is what the RetroArch, Lakka and
RetroPie manifests ship at nxengine/Doukutsu.exe. The core opens this
binary at every launch to extract the ORG music, the PXT sounds, the
stage tile attributes, the credit bitmaps and the wavetable, so the
build matters.
The collection already held the right bytes: data/nxengine, refreshed
from the buildbot's own NXEngine asset, carries a copy matching both
declared values exactly. Same size, different build. The previous one
is kept in .variants/ under its hash; nothing declared it.
bios/Commodore/C128/vice/C128/kernal-318034-01.bin held 32768 bytes
where VICE reads 16384. It was two ROMs end to end: kernal64-325182-01
followed by the Swedish kernal the name promises. The RetroArch, Lakka
and RetroPie manifests shipped it, so the file reaching users was one
the emulator cannot load.
The second half is the genuine dump, and it matches every value
vice/src/c128/c128rom.h:104 declares: 16384 bytes, sha1 925177c9,
md5 78529222, crc32 1cf7f729. The same bytes are in the VICE tree at
the revision the profile pins. The assembly is kept in .variants/ under
its own hash, though both halves were already held separately.
Supermodel's Games.xml is collected in the same pass, from the commit
the profile pins. It is the ROM database the loader parses at startup,
so without it no set resolves and the launch returns before the
emulator is built.
The published manifests still described what the builder produced before
today's fixes. RetroArch's listed 55 files from another emulator's tree -
an Android package's banner, icons and Noto fonts - to be installed into
a BIOS directory, and omitted the six regional DSi NAND dumps a
filename-agnostic core accepts.
Across the twelve platforms: 199 files no longer listed, 36 newly
listed. Generated from the committed profile set and the committed
database so the revision stays consistent with itself.
Pinning every member's metadata made packs reproducible and took the
executable bit with it. The RetroDECK pack ships the two Voxatron engine
binaries, and extracted at 644 they cannot be run.
Git records the bit, so reading it from the source file keeps a pack the
same from any clone. Nothing else about the source's mode reaches the
archive: 2569 members ship at 644 and 942 at 755, which is what the
builder produced before the pinning.
Nothing caught this. The comparison that proved the pinning inert
checked member names, CRCs and sizes, and mode is none of those. A test
now builds a runnable payload and asserts it survives extraction.
RetroDECK rebuilds to the same bytes twice and passes its integrity
check, 2008/2008 baseline and 1551/1551 cores.
The verifier and the builder each resolved the profiles a run names, in
thirty-five lines that differed only in how they failed: one exits, the
other returns empty-handed. An alias is the same binary under another
name and a launcher only starts an emulator, so neither has requirements
of its own, and both refusals have to say the same thing.
common raises now and each caller chooses its own ending. Six tests hold
the refusals, one of them reading both sources so a copy cannot grow
back. The manifest's core-complement phase comes out of generate_manifest
in the same pass, 60 to 34.
Verified inert: manifests identical entry for entry, and the Handy pack
rebuilds to the same bytes.
generate_emulator_pack is a distinct build path from the platform packs,
and nothing asserted that it produced the same bytes twice. It does, but
only because every member write was routed through one writer: restoring
the mtime-copying write on that path alone fails the new test and no
other.
Resolving the profiles a run names, which refuses an alias or a launcher
because neither has requirements of its own, and the structural checks
on an archive that hold whatever it contains: a duplicate entry, an
absolute path, a traversal, a zero-byte member. Complexity 58 to 50 and
63 to 51.
Extracting the second surfaced a Counter it had been reading from the
enclosing module, which pyflakes caught before the tests did.
find_undeclared_files decided per entry whether a core requirement can
be a gap, through a chain that mixed two skips whose difference is easy
to lose. Some record the requirement as settled so no other profile
reconsiders it; the rest leave it open, because the same file can be
libretro-only in one profile and standalone-only in another and the key
carries no emulator.
The chain returns a named verdict now, complexity 60 to 45. Twelve tests
cover the verdicts and two more cover the distinction end to end: a
standalone-only entry seen first must not answer for the profile that
needs the file. Collapsing the two skips into one passes every other
test in the suite and fails that pair.
The markdown report wrote to a directory named after the tooling that
happened to produce it. It takes --report-dir now, defaulting to
reports/, so nothing in the tree names anything but the project.
Em-dashes replaced throughout the sources and tests, rephrased rather
than swapped for a comma where the dash carried an apposition.
Two decisions taken before a byte is written come out of generate_pack.
Which declaration wins when several claim one destination: a platform
may declare the same file bare in one system and hash-constrained in
another, and first-come dedup would let the bare one pack whatever
answers to the name. And which regional or slot alternatives the pack
leaves out, decided once over the baseline and the core extras together.
Complexity 170 to 142. The Recalbox pack rebuilds to the same bytes as
before the change, and the manifests and site are unchanged on frozen
inputs.
_collect_emulator_extras ran three passes in one body: the undeclared
files a platform's cores need, a second copy of an archive under the
subdirectory some cores read, and the scan a filename-agnostic core
allows. The last two are named now, complexity 98 to 54.
The archive-prefix pass had no test, and extracting it surfaced why that
mattered: it read an index from the enclosing scope, so on its own it
would have raised. Three tests cover it, including the case it exists to
prevent - claiming a prefixed path for an archive the collection does
not hold.
resolve_local_file is an ordered chain where the order is the policy:
content first, then a declared path, then a filename. The last steps -
the walk through the cached data directories and the shape-only match a
filename-agnostic core allows - were inline, and the predicate judging a
candidate found by name was a closure with no test of its own.
All three are named now, and the predicate is the interesting one: it
decides whether a file the walk found by filename actually satisfies
what the entry declares. Sixteen tests cover it, including the truncated
MD5 prefixes Batocera publishes and the case where one hash matches
while another is contradicted. Accepting a name match without checking
content fails nine of them.
Complexity 164 to 143. Verified inert against the previous revision on
frozen inputs.
A pack was still not a function of its inputs. ZipFile.write copies the
source file's mtime into the member: the wall clock for an archive this
build rebuilt in tmp/, the checkout time for a file from the collection.
Two consecutive builds of the Recalbox pack differed on 348 members
whose content matched byte for byte, and a pack built from a fresh clone
could never match one built from another.
Every member now goes through one writer that stamps the epoch the
archive rebuilder already uses, streaming the content so a firmware
image of several hundred megabytes is not read whole.
The pack was already covered by a two-builds-are-identical test, which
passed: its fixture held no romset, so it never reached the rebuild
path. The fixture has one now, and reverting the writer fails both that
test and the new one.
Verified on the real collection: Recalbox and RetroArch rebuild to the
same bytes twice, contents unchanged from the previous revision (1319
and 4517 members, zero CRC differences), and both still pass their
native integrity check.
The generated tree is swept rather than deleted, which puts a delete on
the build's critical path with only an end-to-end comparison behind it.
Fourteen tests hold the three behaviours that matter: front matter is
recognised and stripped for the body comparison but not for the
decoration's own, a rebuild on unchanged inputs writes nothing and keeps
its front matter, and the sweep removes a page only when nothing
produced it. Reverting any of the three fails at least two of them.
The counts were generated from a working tree holding profiles that are
still being written and are not committed, so the freshness check
regenerated different numbers from the same revision. They are read from
the committed set now, which is what the check compares against.
generate_emulator_page rendered the header, the metadata rows, the
platform block, every structured field and the file table in one body.
Four pieces come out - the scalar rows, the platform block, the
collapsible fields, and the predicate deciding whether the collection
holds a file - and the page reads as the sequence it always was.
Complexity 68 to 51, each new piece A or B rank.
Verified against the previous revision on frozen inputs: every generated
page identical.
The modules are run directly, run with -m, and imported by the tests and
the type checker. Only the first form puts this directory on the path, so
the package marker carries the bootstrap the other two need; without it
the first sibling import fails. Three tests hold the three forms open.
Three groups leave generate_site.py: the write-and-sweep bookkeeping the
whole build shares, the pinned permalinks into cited sources, and the
small renderings every page uses. Each sits below what calls it, so the
page generators stay the only composite layer.
Verified against the previous revision on frozen inputs: 571 artefacts
identical, and the rendered site still resolves all local links across
511 pages.
generate_pack.py held six responsibilities in 4744 lines. Five move out
in dependency order, so nothing above reaches back down: destinations,
core extras, resolution with its storage tiers, the notes shipped inside
a pack, and the pack verifier. generate_pack.py keeps the build and the
command line, and re-exports the rest.
Two things the move surfaced. The offline switch was a module global the
command line assigned, which a re-export would have copied and frozen at
False; it is set through a call now and not re-exported. And a facade
placed after the entry point binds too late: importing the module worked,
running it did not, so the manifest run died on a name the tests never
exercised because tests import.
Verified against the previous revision on frozen inputs: every generated
artefact is identical, save the catalogue that embeds hashes of files
carrying a build timestamp.
common.py had grown to 1833 lines by accumulation. Six coherent pieces
move out - untrusted parsing, digests, archives, generated artefacts,
release assets, dump catalogues - and common.py re-exports them, so the
sixty existing import sites keep working and migrating them stays
optional.
The site build is now reproducible, which is what made the move
checkable. It deleted its generated directories first, so every page was
new and write_if_changed had no earlier version to compare against: a
deploy republished six hundred pages for the clock alone. Directories
are swept instead, a page is removed only once nothing produces it, and
the body pass compares against the body of the file on disk rather than
against the decorated page. Two consecutive builds on the same inputs
now produce identical bytes; before, 1034 files differed.
Four sites still compared the verification mode to a literal after the
module owning that policy existed. One of them mattered: an unrecognised
mode fell through to MD5 verification while compute_severity was scoring
it as existence, so a typo in a platform YAML produced a report whose
checks and severities described different platforms.
The mode is normalized once per run and the consumers ask for what they
need. A test reads the sources and fails on a literal comparison, so the
next consumer cannot quietly grow a fifth copy.
A filename-agnostic core accepts any name for its BIOS, so the builder
scans the directory holding the candidates. It picked that directory
from a first-hit lookup by name, the one piece of evidence that lands
in another emulator's tree: five files answer to GameIndex.yaml and one
belongs to an Android package, rom1.bin is a PS2 ROM and a Roland
SC-55 ROM. One wrong match became every file beside it, flattened into
the BIOS root of platforms that do not run that emulator: 45 files in
the Recalbox pack, 170 in the RetroArch one.
Four things decide it now. A destination and the repo layout meet on a
tail, so the path index is tried from the longest tail down and never
to the bare filename; that alone corrects seven files, among them the
Japanese GameCube slot, which held the US dump. A seed has to declare
the shape it is looking for, since no shape means the whole directory.
An ambiguous name needs the profile's other files to agree before the
scan walks anywhere. What the scan emits carries the SHA-1 it selected,
so packing never resolves it by name again.
The builder and the coverage report each grouped their own candidates
before asking which regional alternatives to withdraw. The builder
grouped the platform files and the core extras; the report grouped the
platform files alone, and keyed them on an unsanitized destination.
So a region run withdrew 73 files from a recalbox pack while the report
withdrew 14, and described the other 59 as covered by a pack that would
not carry them.
platform_region_groups builds the grouping once and both sides read it.
The extras it returns are keyed by emulator, name and path: Dolphin
declares three IPL.bin that differ by path alone, and a name-keyed map
withdraws the wrong one. Manifests are byte-identical before and after.
The pack builder accepts the aliases declared in the target overrides,
so --target switch works there, but the installer's target manifests
carried only canonical names. The documented word was the one that
failed, and the installer then carried on with every file: 1911 files
and 4.1 GB where the user had asked for the 863 that target needs.
Aliases are emitted beside their canonical target, and an unknown
target now stops the run and lists what is available. A filter is
applied or refused, never ignored.
The gap analysis answered from the name index, so a core extra whose
local copy contradicts its declared hash counted as held. Under a
digest mode the builder drops exactly that file, so the coverage report
described a pack that would not contain it: seven files across
Batocera, Recalbox and RetroBat.
An entry that states what its content should be is now resolved by
content; the name still answers for entries that declare nothing to
check against, and existence mode is unchanged because there the
frontend never opens the file and the pack does carry it.
The per-emulator report counted an entry nobody can supply -- a
per-user key, a slot the user fills, a dump that was never made -- as
plainly missing. fpse-ng read 14 missing when twelve were documented as
unobtainable, which invites the wrong repair: dropping the flag,
deleting the entry, or chasing a vendor's whole install tree.
They are listed with the reason the profile records rather than hidden,
and the summary counts them apart. Platform reports are untouched.
The per-emulator report captured the status resolve_local_file returns
and then ignored it: any non-empty path became OK. An entry whose only
candidate contradicts its declared hash therefore read as covered, so a
same-named file from another system stood in for one the collection
does not hold. 36 entries across 14 profiles were affected, among them
config.ini, ROM and rom2.bin, names that collide across systems.
Emulator validation still runs first, since it names the field that
disagrees; the resolution status is consulted only when validation had
nothing to say.
export_native turns the profile data back into each platform's own
file, for a maintainer there to pick up and use, and none of the ten
exporters had a test. Coverage goes from 0 to 83%.
Writing them found the EmuDeck exporter rejecting its own output: its
export skips placeholder entries, its validate looked for them anyway
and reported the omission as a missing hash. Both sides apply the same
filter now.
The placeholder assertion initially passed for the wrong reason twice
over -- the fixture entry had no hash, so it was dropped as incomplete
rather than as a placeholder, and the assertion then read the name
while the exporters write the path. It is pinned on an unmistakable
destination now, and removing the guard fails two tests.
The requirement parser read core, hashMatchMandatory and note from
each bios element and discarded all three, which reads like a dropped
attribute: hashMatchMandatory decides whether a wrong hash is yellow or
red on Recalbox. It is not lost -- the config parser below captures it
for the YAML -- so these were leftovers, and they cost a look to
confirm that.
Two yaml imports left unused by the move to the shared loader go too.
validate_site reached complexity 43 doing three things at once. The
per-page checks and the link resolution are now separate functions and
the caller keeps only the cross-page duplicate check, which cannot run
until every page has been seen.
The first attempt left the link pass calling a closure that had moved,
and running it against the real site did not catch that: no page there
has a broken link, so the error path never ran. The unit test covering
a deliberately broken fragment did.
_diff_system reached complexity 42, and its hash-based rename fallback
is the part that stands alone: a platform is free to call a file
whatever it likes, so a name matching nothing is not yet a gap, and
counting one file as both missing and extra invents a discrepancy.
That step is now its own function at complexity 31, with the tests it
never had: pairing on any of the three digests, case folding, non-string
values, and the case where two files simply have no hashes and so are
not evidence of anything. diff_truth output is unchanged.
Regenerated with the large-file cache populated, the state CI builds
from: without it ten preserved entries resolve differently and the
counts drift by one file.
SHA1 is the primary key of this collection and there was no
.gitattributes, so git guessed. Git for Windows sets core.autocrlf=true
by default: a clone there rewrites every file git considers text,
meaning the shaders, .ini, .txt and .dat assets under bios/ arrive with
CRLF and a different hash from the one published. Verification then
fails on files nobody touched.
bios/ and data/ are exempt from normalisation, generated artefacts are
pinned to LF so a Windows checkout does not show them modified, and the
rule order is asserted rather than assumed.
A clone group is only visible while both copies are on disk, and the
run then deletes the clone, so writing only what this run saw erased
every mapping an earlier run had recorded. One real run took the map
from 69 entries to 1, and the canonical zips silently stopped answering
to the names they stand in for.
The map is merged now, the 68 lost entries are restored, and the one
whose canonical file is no longer in the collection is dropped.
The cross-reference loop body became its own function, but one exit
stayed a continue and left the module unparseable, which took the whole
test suite down with it. The other ten continues are inside genuine
inner loops and stand.
Output verified identical to the version before the split.
deduplicate reached complexity 50 by mixing the question of which
copies may go with the work of removing them. The decision is now its
own function: same name in two directories is a true duplicate,
different names only collapse as MAME device clones, and anywhere else
each name may be the one some emulator looks for.
Verified against the real collection: both versions plan the same 99
removals.
dedup.py is the only script here that deletes BIOS files and it had no
tests: the sole guard was remembering --dry-run. These cover the
protected directories where two identical copies are both load-bearing,
the canonical choice between a primary and a variant, MAME zip clones
against same-content files that must keep every name, and that a dry
run reports exactly the plan the real run executes.
Disabling the NODEDUP guard fails four of them.
generate_platform_page reached complexity 49, most of it in the block
rendering one collapsible section per system: each file shows a
different set of hashes, sizes and provenance depending on what the
platform declares. That block is now its own function and the caller
sits at 28. All 534 pages identical apart from timestamps.
generate_cross_reference rendered the same relation twice in one
function, once per platform and once per upstream, and reached
complexity 53. Each view is now its own function and the caller is a
pair of calls.
Verified by generating the site with both versions against the same
data: all 533 pages identical apart from timestamps, with a control run
of the same code twice to confirm the comparison had no drift in it.
Abdessamad Derraz