Commit Graph
782 Commits
Author SHA1 Message Date
Abdessamad Derraz 24e9b820ae feat: make packs reproducible byte for byte
Two builds of the same pack from the same inputs produced different
archives. Of 67 members, 65 were already identical: only README.txt
and manifest.json differed, both stamped with the wall clock by
writestr and the second carrying a generated timestamp. Generated
members now use the epoch the archive rebuilder already applies, and
the timestamp comes from the database snapshot the pack was built
from, so the same data yields the same bytes.

Install manifests skipped archived platforms, which is why RetroPie
had none; archived means upstream is no longer scraped, not that the
packs stopped shipping. A target-filtered manifest also had no record
of its filter beyond the filename, so it carries one the way a
region-filtered manifest already does.
2026-08-11 00:55:42 +02:00
Abdessamad Derraz dc14089932 fix: keep the database free of stale entries
Two entries claimed bios/Sony/PlayStation 3/PS3UPDAT.PUP with different
SHA1s. Preserving large-file entries matched on path and keyed on
SHA1, so replacing a firmware revision on disk left the old entry
pointing at a path that now serves other bytes. A preserved entry whose
path the scan has already claimed is dropped, and validate_schemas
refuses a database where one path carries two entries.

Separately, a run without --force rebuilt each cached entry from a
hand-written list of four digests and wrote it back without adler32,
so one such run stripped the digest from every file permanently. A
cache entry missing any digest is now a miss.
2026-08-11 00:55:32 +02:00
Abdessamad Derraz 3b8f2d75d5 perf: read yaml through the c loader
Loading the emulator profiles is the most expensive step of every
command here, and all forty call sites used the pure-Python scanner
while libyaml sat unused in the same wheel. One shared yaml_load picks
the C loader when pyyaml ships it: the 375 profiles parse in 0.18s
instead of 1.39s, and verify --platform retroarch drops from 2.17s to
0.73s. The loader class is the same restricted one safe_load uses.

es_bios.xml was parsed straight from the network while install.py
already refused a document declaring entities; both now share one
guard. Scrapers reach it through a single path bootstrap in the
package rather than two ad-hoc ones.
2026-08-11 00:55:16 +02:00
Abdessamad Derraz ab6a3bb26d refactor: single-source the native mode policy
verify.py and generate_pack.py must reach the same verdict on the same
file, and CLAUDE.md calls any divergence critical, but each spelled out
mode == 'existence' in its own words. Both now ask nativemode whether
the frontend reads the file's bytes, which is the one fact the rest
follows from, and a test holds the two answers together.

The BaseScraper contract the wiki asks contributors to implement had no
caller anywhere, so nothing proved compare_with_config, has_changes or
test_connection still worked.
2026-08-11 00:54:46 +02:00
Abdessamad Derraz 057b0c18f2 chore: widen the build triggers and pin set
A pack is the platform baseline plus what its cores need, so a profile
change alters pack contents; build.yml did not watch emulators/. Its
release notes read git log -15 on a depth-1 clone, which returns one
commit, and its test step ran a single module out of nineteen.

validate.yml ignored install.sh and install.ps1, so a PR touching only
a bootstrap skipped the test that pins them to install.py.

checkout and setup-python were pinned to v6 in two workflows and v7 in
the others; jsonschema was imported by validate_schemas.py and
declared nowhere, and requires-python claimed 3.10 while both
bootstraps accept 3.8.
2026-08-11 00:54:36 +02:00
Abdessamad Derraz 185cf47bbc fix: hold manifest paths inside the trust boundary
Every other manifest field was treated as hostile input, but
standalone_copies targets were only length-checked before being
expanded and written to: a traversal component or a symlink already
sitting at the destination sent the copy outside the directory the
user opted into. Targets are now validated like the other paths and a
symlinked destination is never followed.

RETROBIOS_BASE_URL serves the manifest and the files it declares, so
it now has to be HTTPS the way both bootstraps already require of the
installer URL; loopback stays open for the end-to-end tests.

install.ps1 left TLS at the Windows PowerShell 5.1 default, which
GitHub refuses, so the download failed before any hash was checked.

check_local read every file once per declared digest, single threaded.
One read now feeds both, across the same pool the downloads use.

RetroPie had no manifest, so the one-line installer answered 'unknown
platform' for a frontend whose packs do ship.
2026-08-11 00:54:27 +02:00
Abdessamad Derraz 958b988015 docs: scope the licence and temper the legal claims
LICENSE covered the whole repository, so the MIT grant read as
covering 9.9 GB of third-party firmware. It now states its scope and
NOTICE describes the files it does not cover, with a removal channel.

The FAQ cited Connectix and Accolade for redistribution when both
decided intermediate copying during reverse engineering, presented fair
use as settled, read section 1201(f) as a general permission, and
listed abandonware among legal doctrines. Each claim is now stated at
the strength it actually has, and the weakest ground is named.
2026-08-11 00:54:10 +02:00
Abdessamad Derraz 902bb4e01e refactor: require a hash before relocating a ref 2026-08-11 00:37:30 +02:00
Abdessamad Derraz 5330ed1950 fix: keep the cache usable when a path is both 2026-08-10 19:10:26 +02:00
Abdessamad Derraz 1ac569b98e feat: point five profiles at their load sites 2026-08-10 18:35:03 +02:00
Abdessamad Derraz f07b1e7487 feat: point ep128emu and eden at their load sites 2026-08-10 18:32:11 +02:00
Abdessamad Derraz 1e1dee59c2 feat: complete the prboom path prefixes 2026-08-10 18:31:07 +02:00
Abdessamad Derraz d922fec008 feat: point o2em rvvm and prboom at real paths 2026-08-10 18:30:51 +02:00
Abdessamad Derraz 847bc60a5f refactor: drop the info citations from dosbox_core 2026-08-10 18:29:29 +02:00
Abdessamad Derraz 106f5bc171 feat: point uae4arm at its own rom defines 2026-08-10 18:19:36 +02:00
Abdessamad Derraz 49f50f51b5 feat: read per-mode sources and pins 2026-08-10 18:18:09 +02:00
Abdessamad Derraz 4a69b96922 feat: point mesen-s at each boot rom array 2026-08-10 18:14:42 +02:00
Abdessamad Derraz 5320e9e187 feat: cite where melonds opens the dsi nand 2026-08-10 18:13:52 +02:00
Abdessamad Derraz f4babf77aa feat: point amiberry at the current candidate lists 2026-08-10 18:13:26 +02:00
Abdessamad Derraz a40f505f71 feat: pin the dragon professional boot roms 2026-08-10 18:12:49 +02:00
Abdessamad Derraz 64ab248802 docs: state when a file carries a region 2026-08-10 18:08:57 +02:00
Abdessamad Derraz a28440755c fix: align the resolve_head double with its signature 2026-08-10 18:03:46 +02:00
Abdessamad Derraz 2b7c5f9a64 feat: follow the branch a port was read from 2026-08-10 17:55:35 +02:00
Abdessamad Derraz 87087c74f0 feat: resolve bare paths in three profiles 2026-08-10 17:54:31 +02:00
Abdessamad Derraz e1fae87717 feat: point casloopy and epitch at their machine lines 2026-08-10 17:41:32 +02:00
Abdessamad Derraz dd7faf2b5f feat: attribute a shared path by its cited subject 2026-08-10 17:40:48 +02:00
Abdessamad Derraz 44ba5e45c3 feat: pin the mame upstream revision 2026-08-10 17:37:06 +02:00
Abdessamad Derraz 7ac19f4c9f feat: reach forward by the declared member count 2026-08-10 17:31:59 +02:00
Abdessamad Derraz 4f7aa5f853 feat: read content values from archive members 2026-08-10 17:30:29 +02:00
Abdessamad Derraz d546be33b0 feat: give lindbergh refs their directory 2026-08-10 17:29:20 +02:00
Abdessamad Derraz c29f3297e1 feat: give scummvm refs their full path 2026-08-10 17:28:18 +02:00
Abdessamad Derraz f0e023ac24 feat: annotate region on emulator profiles 2026-08-10 17:22:21 +02:00
Abdessamad Derraz 885c4c3c06 refactor: judge a self-checked entry whole 2026-08-10 17:22:00 +02:00
Abdessamad Derraz f430986361 refactor: treat a located value as a recalable move 2026-08-10 17:20:26 +02:00
Abdessamad Derraz 11677d6625 feat: complete the shadps4 lle module list 2026-08-10 17:19:45 +02:00
Abdessamad Derraz 43cf84d5ec refactor: require evidence before calling a ref stale 2026-08-10 17:17:19 +02:00
Abdessamad Derraz ba699f472c feat: add thomson mo6 machines to clk 2026-08-10 17:15:57 +02:00
Abdessamad Derraz f2670b1ad5 feat: anchor fbneo refs on their rom lines 2026-08-10 17:00:35 +02:00
Abdessamad Derraz f67642095c feat: self-check a profile already on head 2026-08-10 16:55:48 +02:00
Abdessamad Derraz ba84f5701f chore: cover region filter with split and target 2026-08-10 16:54:39 +02:00
Abdessamad Derraz dd4d0c1d35 chore: recale refs after the cap increase 2026-08-10 16:50:50 +02:00
Abdessamad Derraz 77a39b2f4e perf: reuse diffs and raise the file size cap 2026-08-10 16:48:59 +02:00
Abdessamad Derraz 77dff6377e fix: keep fbneo per-rom refs and archive names 2026-08-10 16:47:20 +02:00
Abdessamad Derraz 47182f4cb1 chore: refresh database, manifests and readme
7851 files.
2026-08-10 15:13:15 +02:00
Abdessamad Derraz 76984e8bce feat: close the last anchored gaps
Four files, each with a verifiable origin.

logo.tga and logo_red.tga come from the loader's own resources/logo.h:
hook.c's fread maps FILE_LOGO_TGA to logoLL or logoLLRed depending on
which path was opened, one array per file, so the extraction is exact.
Both decode as valid RLE TGA at their declared lengths.

lgp.zip and timetrav.zip carry crc 0 in Hypseus and in Daphne upstream,
so MAME vouches for them instead. lgp maps onto MAME's set by base name,
confirmed by the five members whose bytes already agreed, and the four
that were missing came from the same set. timetrav is MAME's timetrv:
tt061891.bin, 262144 bytes, crc a3d44219.

The four Sega fonts stay open, and the profile now says why: the loader
embeds one 128x256 atlas and one metrics table answering all four paths,
so it substitutes for the originals rather than reproducing either.
2026-08-10 15:13:15 +02:00
Abdessamad Derraz 8ef73daca3 chore: refresh database, manifests and readme
7847 files. Regenerated after the laserdisc acquisition.
2026-08-10 14:34:34 +02:00
Abdessamad Derraz d588ebbde4 chore: restore large files before ci coverage checks
A CI checkout omits every file over 50 MB, so verify and generate_pack
resolve those database entries against a disk that does not hold them and
report them missing. The generated README then stops matching the
committed one for a reason that has nothing to do with staleness.

restore_large_files.py writes them back from the release cache, matched by
SHA1 rather than by name, and only where the path is gitignored and
absent. The site workflow runs it, and refreshes the data directories, before
generating.
2026-08-10 14:34:27 +02:00
Abdessamad Derraz b76edd2034 feat: add laserdisc romsets for hypseus singe
Sixty-nine of the seventy-one sets Hypseus Singe loads, assembled from
atoms retrieved from public archives and rebuilt with torrentzip under
the member names its source expects.

Every set is anchored: the profile already carried each member with its
CRC32, read from Hypseus' own source, so a candidate is either an exact
match or discarded. Sixty-seven verified that way. For gpworld and
firefoxa, Hypseus declares crc 0 and gives no anchor, so MAME's recipe
for the same game vouched for the bytes instead: 17/17 and 6/6.

lgp and timetrav stay out. Hypseus declares crc 0 for both, MAME covers
only 5 of lgp's 9 members and has no timetrav at all, so nothing can
vouch for them and half-verified is not verified.

The six Lindbergh font and logo files gain has_builtin: true. hook.c
serves them from resources/logo.h and resources/font.h unless built-ins
are disabled, so they are not a gap the collection can close.
2026-08-10 14:34:15 +02:00
Abdessamad Derraz 0d4a4b3355 chore: refresh database, manifests and readme
7779 files. Regenerated after the archive renames, the Triforce filing
and the fourteen acquired runtimes.
2026-08-10 13:37:25 +02:00
Abdessamad Derraz c00314d479 docs: align the wiki with what the code does
The pages described workflows, an installer pinning and a CI permission
model that are not the ones in the tree. Corrected: the release process,
the CI table, the resolution chain and its statuses, the hash-mismatch
policy per verification mode, the archive convention, and the romset
recipe store.

The FAQ keeps the project's own reading of the legal question rather
than a version that reaches no conclusion.
2026-08-10 13:37:20 +02:00