Pinning every member's metadata made packs reproducible and took the
executable bit with it. The RetroDECK pack ships the two Voxatron engine
binaries, and extracted at 644 they cannot be run.
Git records the bit, so reading it from the source file keeps a pack the
same from any clone. Nothing else about the source's mode reaches the
archive: 2569 members ship at 644 and 942 at 755, which is what the
builder produced before the pinning.
Nothing caught this. The comparison that proved the pinning inert
checked member names, CRCs and sizes, and mode is none of those. A test
now builds a runnable payload and asserts it survives extraction.
RetroDECK rebuilds to the same bytes twice and passes its integrity
check, 2008/2008 baseline and 1551/1551 cores.
The verifier and the builder each resolved the profiles a run names, in
thirty-five lines that differed only in how they failed: one exits, the
other returns empty-handed. An alias is the same binary under another
name and a launcher only starts an emulator, so neither has requirements
of its own, and both refusals have to say the same thing.
common raises now and each caller chooses its own ending. Six tests hold
the refusals, one of them reading both sources so a copy cannot grow
back. The manifest's core-complement phase comes out of generate_manifest
in the same pass, 60 to 34.
Verified inert: manifests identical entry for entry, and the Handy pack
rebuilds to the same bytes.
The markdown report wrote to a directory named after the tooling that
happened to produce it. It takes --report-dir now, defaulting to
reports/, so nothing in the tree names anything but the project.
Em-dashes replaced throughout the sources and tests, rephrased rather
than swapped for a comma where the dash carried an apposition.
Two decisions taken before a byte is written come out of generate_pack.
Which declaration wins when several claim one destination: a platform
may declare the same file bare in one system and hash-constrained in
another, and first-come dedup would let the bare one pack whatever
answers to the name. And which regional or slot alternatives the pack
leaves out, decided once over the baseline and the core extras together.
Complexity 170 to 142. The Recalbox pack rebuilds to the same bytes as
before the change, and the manifests and site are unchanged on frozen
inputs.
A pack was still not a function of its inputs. ZipFile.write copies the
source file's mtime into the member: the wall clock for an archive this
build rebuilt in tmp/, the checkout time for a file from the collection.
Two consecutive builds of the Recalbox pack differed on 348 members
whose content matched byte for byte, and a pack built from a fresh clone
could never match one built from another.
Every member now goes through one writer that stamps the epoch the
archive rebuilder already uses, streaming the content so a firmware
image of several hundred megabytes is not read whole.
The pack was already covered by a two-builds-are-identical test, which
passed: its fixture held no romset, so it never reached the rebuild
path. The fixture has one now, and reverting the writer fails both that
test and the new one.
Verified on the real collection: Recalbox and RetroArch rebuild to the
same bytes twice, contents unchanged from the previous revision (1319
and 4517 members, zero CRC differences), and both still pass their
native integrity check.
generate_pack.py held six responsibilities in 4744 lines. Five move out
in dependency order, so nothing above reaches back down: destinations,
core extras, resolution with its storage tiers, the notes shipped inside
a pack, and the pack verifier. generate_pack.py keeps the build and the
command line, and re-exports the rest.
Two things the move surfaced. The offline switch was a module global the
command line assigned, which a re-export would have copied and frozen at
False; it is set through a call now and not re-exported. And a facade
placed after the entry point binds too late: importing the module worked,
running it did not, so the manifest run died on a name the tests never
exercised because tests import.
Verified against the previous revision on frozen inputs: every generated
artefact is identical, save the catalogue that embeds hashes of files
carrying a build timestamp.
Four sites still compared the verification mode to a literal after the
module owning that policy existed. One of them mattered: an unrecognised
mode fell through to MD5 verification while compute_severity was scoring
it as existence, so a typo in a platform YAML produced a report whose
checks and severities described different platforms.
The mode is normalized once per run and the consumers ask for what they
need. A test reads the sources and fails on a literal comparison, so the
next consumer cannot quietly grow a fifth copy.
A filename-agnostic core accepts any name for its BIOS, so the builder
scans the directory holding the candidates. It picked that directory
from a first-hit lookup by name, the one piece of evidence that lands
in another emulator's tree: five files answer to GameIndex.yaml and one
belongs to an Android package, rom1.bin is a PS2 ROM and a Roland
SC-55 ROM. One wrong match became every file beside it, flattened into
the BIOS root of platforms that do not run that emulator: 45 files in
the Recalbox pack, 170 in the RetroArch one.
Four things decide it now. A destination and the repo layout meet on a
tail, so the path index is tried from the longest tail down and never
to the bare filename; that alone corrects seven files, among them the
Japanese GameCube slot, which held the US dump. A seed has to declare
the shape it is looking for, since no shape means the whole directory.
An ambiguous name needs the profile's other files to agree before the
scan walks anywhere. What the scan emits carries the SHA-1 it selected,
so packing never resolves it by name again.
The builder and the coverage report each grouped their own candidates
before asking which regional alternatives to withdraw. The builder
grouped the platform files and the core extras; the report grouped the
platform files alone, and keyed them on an unsanitized destination.
So a region run withdrew 73 files from a recalbox pack while the report
withdrew 14, and described the other 59 as covered by a pack that would
not carry them.
platform_region_groups builds the grouping once and both sides read it.
The extras it returns are keyed by emulator, name and path: Dolphin
declares three IPL.bin that differ by path alone, and a name-keyed map
withdraws the wrong one. Manifests are byte-identical before and after.
The pack builder accepts the aliases declared in the target overrides,
so --target switch works there, but the installer's target manifests
carried only canonical names. The documented word was the one that
failed, and the installer then carried on with every file: 1911 files
and 4.1 GB where the user had asked for the 863 that target needs.
Aliases are emitted beside their canonical target, and an unknown
target now stops the run and lists what is available. A filter is
applied or refused, never ignored.
install.py fetches a file from its repo_path or from a release asset.
Resolution can land on a file the database does not index, and the
entry then shipped with neither: a line in the download list that can
only ever fail. Those are recorded as omitted instead, which is what
the installer already knows how to report, and a test holds the
committed manifests to it.
validate_schemas read dist/ while a build was writing it and reported
a half-written pack as 'File is not a zip file'. It takes the shared
lock --verify-packs uses, and says so when a build holds it.
A required-only build is narrower than the platform declares by
design, like the source-restricted variants already handled, so the
full expectation must not be applied to it. --region is also refused
alongside --manifest-targets, which carries no region dimension.
Two builds of the same pack from the same inputs produced different
archives. Of 67 members, 65 were already identical: only README.txt
and manifest.json differed, both stamped with the wall clock by
writestr and the second carrying a generated timestamp. Generated
members now use the epoch the archive rebuilder already applies, and
the timestamp comes from the database snapshot the pack was built
from, so the same data yields the same bytes.
Install manifests skipped archived platforms, which is why RetroPie
had none; archived means upstream is no longer scraped, not that the
packs stopped shipping. A target-filtered manifest also had no record
of its filter beyond the filename, so it carries one the way a
region-filtered manifest already does.
A declared hash that the local dump contradicts is not one situation. An
existence platform never reads the bytes, so withholding the file lets an
upstream list error remove something the frontend would have loaded; a
hash platform would reject it, so shipping it is pointless.
The mode now decides, at every point that had an opinion: pack building,
core complement, emulator packs, manifests, conformance and
_intentional_hash_exclusion. verify.find_undeclared_files follows, since
verify and generate_pack must agree file for file.
Also here: resolution reports which evidence matched rather than a flat
"exact", a path or filename can no longer override a declared hash, and
safe_extract_zip treats a Windows backslash as the separator it is
instead of refusing the archive.
generate_db: add by_sha256 index for O(1) variant lookup.
verify: _find_best_variant uses indexed sha256 instead of O(n) scan.
validation: check_file_validation returns (reason, emulators) tuple,
attributing mismatch only to emulators whose check actually failed.
beetle_psx: remove incorrect size field for ps1_rom.bin (code does
not validate size, swanstation is sole size authority).
find_undeclared_files was enriching declared_names with DB aliases,
filtering core extras that were never packed by Phase 1 under that
name. Pass strict YAML names to _collect_emulator_extras so alias-
only files (dc_bios.bin, amiga-os-310-a1200.rom, scph102.bin, etc.)
get packed at the emulator's expected path. Also fix truth mode
output message and --all-variants --verify-packs quick-exit bypass.
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).
Run isort --profile black: normalize import ordering across all files.
Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.
3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
Move verification logic to generate_pack.py --verify-packs (single
source of truth). test_pack_integrity.py is now a thin wrapper that
calls the CLI. Pipeline step 6/8 uses the same CLI entry point.
Renumber all pipeline steps 1-8 (was skipping from 5 to 8/9).
Update generate_site.py with pack integrity test documentation.
RetroDECK: core extras with subdirectory paths (e.g. vice/C64/,
fbneo/, dc/) were placed outside bios/ because the prefix was only
inferred for bare filenames. Add _detect_extras_prefix() to infer
the dominant BIOS prefix from YAML destinations.
RomM: core extras landed flat at bios/{file} instead of the required
bios/{platform_slug}/{file}. Add _detect_slug_structure() to detect
per-system slug layouts and _map_emulator_to_slug() to route each
extra to the correct slug subfolder.
Also skip manifest writes when only the generated timestamp changed,
preventing unnecessary diffs in install/*.json.
Closes#43
FBNeo and Kronos expect BIOS archives in core-specific subdirectories
(system/fbneo/, system/kronos/). RetroArch firmware check uses .info
paths which include these prefixes, so files at root show as Missing.
Add archive_prefix field to emulator profiles. The pack code now places
archive copies in the prefixed subdirectory while keeping root copies
for cores that expect them there (e.g. Geolith for neogeo.zip).
bios_mode: agnostic (profile) and agnostic: true (file) for
emulators that accept any valid BIOS without specific filename.
find_undeclared_files skips agnostic entries, pack extras scan
includes all matching DB files by path prefix + size criteria,
resolve_local_file has agnostic fallback with rename README.
applied to pcsx2, lrps2 (bios_mode), melonds dsi_nand (file).
expand_platform_declared_names resolves platform file MD5s
through the database to recover canonical names and aliases,
eliminating false positive undeclared files when a platform
renames a file (e.g. Batocera ROM1 vs gsplus ROM).
Abdessamad Derraz