From c00314d479152a2d3b0ef061d486f559af56bfa9 Mon Sep 17 00:00:00 2001 From: Abdessamad Derraz <3028866+Abdess@users.noreply.github.com> Date: Mon, 10 Aug 2026 13:37:20 +0200 Subject: [PATCH] docs: align the wiki with what the code does The pages described workflows, an installer pinning and a CI permission model that are not the ones in the tree. Corrected: the release process, the CI table, the resolution chain and its statuses, the hash-mismatch policy per verification mode, the archive convention, and the romset recipe store. The FAQ keeps the project's own reading of the legal question rather than a version that reaches no conclusion. --- wiki/architecture.md | 65 +++++++++++++++++++------------ wiki/faq.md | 7 ++++ wiki/getting-started.md | 15 ++++++-- wiki/release-process.md | 30 ++++++++++----- wiki/testing-guide.md | 70 +++++++++++++++++++--------------- wiki/tools.md | 78 +++++++++++++++++++++++++++++++++----- wiki/troubleshooting.md | 13 ++++++- wiki/verification-modes.md | 46 +++++++++++++--------- 8 files changed, 230 insertions(+), 94 deletions(-) diff --git a/wiki/architecture.md b/wiki/architecture.md index 16584c02..21497fc6 100644 --- a/wiki/architecture.md +++ b/wiki/architecture.md @@ -95,6 +95,9 @@ graph LR The pack combines platform baseline (layer 1) with core requirements (layer 3). Neither too much (no files from unused cores) nor too few (no missing files for active cores). +A same-named local file that contradicts an explicit hash is recorded as an +unsafe omission on hash-verifying platforms; existence platforms ship it and +report the divergence, because their code never reads the bytes. The emulator's source code serves as ground truth for what files are needed, what names they use, and what validation the emulator performs. Platform YAML @@ -148,7 +151,9 @@ RetroArch and Lakka share the same files and `base_destination` (`system/`), so they produce one combined pack (`RetroArch_Lakka_BIOS_Pack.zip`). RetroPie uses `BIOS/` as base path, so it gets a separate pack. With `--target`, the fingerprint includes target cores so platforms -with different hardware filters get separate packs. +with different hardware filters get separate packs. Emulator and system packs +also retain system identity, `variant_group` and requested region; same-named +regional files are never collapsed merely because their display label matches. ## Storage tiers @@ -175,10 +180,25 @@ Files with `hle_fallback: true` are downgraded to INFO when missing When a file passes platform verification (MD5 match) but fails emulator-level validation (wrong CRC32, wrong size), a DISCREPANCY is reported. The pack generator searches the repo for a variant that satisfies both. -If none exists, the platform version is kept. +If none exists, the platform version is kept and the discrepancy stays +reported. A filename or destination never overrides a declared content hash +during resolution; whether a mismatch is packed is then decided by the +platform's own verification mode. ## Security +- install metadata is fetched from the same revision as the installer that + reads it, and validated before use; +- destinations are normalized and contained below the selected BIOS root; +- downloads are bounded by declared and global size limits, verified with + SHA256/SHA1, written to unique temporary siblings and atomically replaced; +- ZIP extraction rejects traversal, absolute paths, duplicates, links, special + files, encryption, excessive expansion and suspicious compression ratios; +- standalone-emulator copies require explicit `--standalone-copies` consent; +- CI writes only what its job needs: `validate.yml` holds `pull-requests: write` + for the validation comment and labels, `deploy-site.yml` holds the Pages + deploy identity, and the release job stays disabled behind `if: false`. + - `safe_extract_zip()` prevents zip-slip path traversal attacks - `deterministic_zip` rebuilds MAME ZIPs so same ROMs always produce the same hash - `crypto_verify.py` and `sect233r1.py` verify 3DS RSA-2048 signatures and AES-128-CBC integrity @@ -268,31 +288,27 @@ When a clone ZIP was deduplicated, `resolve_local_file` uses this map to find th ## Install manifests `generate_pack.py --manifest` produces JSON manifests in `install/` for each platform. -These contain file lists with SHA1 hashes, platform detection config, and standalone copy +These contain downloadable file lists with SHA256 and SHA1 hashes, explicit +unsafe/unavailable omissions, platform detection config, and standalone copy instructions. `install/targets/` contains per-architecture core availability. The cross-platform installer (`install.py`) uses these manifests to auto-detect the -user's platform, filter files by hardware target, and download with SHA1 verification. +user's platform, filter files by hardware target, and download with SHA256/SHA1, +size, path-containment and atomic-write checks. ## Tests -14 test files, 664 tests total: +The suite is discovered dynamically rather than maintained as a hand-counted +subset: -| File | Tests | Coverage | -|------|-------|----------| -| `test_e2e.py` | 218 | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters | -| `test_profile_sync.py` | 189 | ref anchoring, guarded profile writes, detection, triage | -| `test_install.py` | 70 | `install.py` platform detection, config-file parsing, manifest handling | -| `test_upstream.py` | 56 | forge URL parsing, cache, revision resolution, tree comparison | -| `test_provenance.py` | 29 | Logiqx/Redump parsers, DAT pack import, provenance join, coverage report | -| `test_mame_parser.py` | 25 | BIOS root set detection, ROM block parsing, macro expansion | -| `test_hash_merge.py` | 17 | MAME/FBNeo YAML merge, diff detection, formatting preservation | -| `test_fbneo_parser.py` | 16 | BIOS set detection, ROM info parsing | -| `test_deterministic_zip.py` | 12 | streaming rebuild, metadata normalisation, entry ordering, source CRC | -| `test_artifact_lock.py` | 10 | writer/writer and writer/reader exclusion, reader sharing, release on error | -| `test_pack_integrity.py` | 8 | extract ZIP packs to disk, verify paths + hashes per platform's native mode | -| `test_torrentzip.py` | 8 | TorrentZip builder against real MAME romsets | -| `test_large_file_cache.py` | 5 | concurrent downloads, temporary file residue, hash rejection | -| `test_no_case_collisions.py` | 1 | guard against case-colliding paths in `bios/` | +| File | Coverage | +|------|----------| +| `test_e2e.py` | resolution, verification, packs, inheritance, targets, truth and exporters | +| `test_profile_sync.py`, `test_upstream.py` | source anchoring, forge parsing, revisions, cache and guarded writes | +| `test_install.py`, `test_audit_regressions.py` | automatic wrappers, manifest boundaries, pipeline exits, strong hashes and safe archives | +| `test_region.py` | region vocabulary, ranking, per-system grouping and repository conformance | +| `test_site_exports.py`, `test_site_validation.py` | versioned exports, SQLite, metadata, accessibility structure and complete local-link graph | +| parser/provenance modules | MAME, FBNeo, DAT import, merge and provenance joins | +| artifact modules | deterministic ZIPs, locking, large-file cache, pack integrity and portable paths | ```bash python -m unittest discover tests -v # full suite @@ -311,13 +327,14 @@ pattern and how to add a test. | Workflow | File | Trigger | Role | |----------|------|---------|------| | Build & Release | `build.yml` | push to main (bios/, platforms/) + manual | restore large files, build packs, create GitHub release | -| Deploy Site | `deploy-site.yml` | push to main (platforms, emulators, wiki, scripts) + manual | generate site, build with MkDocs, deploy to GitHub Pages | -| PR Validation | `validate.yml` | pull request on `bios/`/`platforms/` | validate BIOS hashes, schema check, run tests, auto-label PR | +| Deploy Site | `deploy-site.yml` | push to main (platforms, emulators, wiki, scripts) + manual | validate contracts, generate site, build with MkDocs, validate rendered HTML, deploy to Pages | +| PR Validation | `validate.yml` | pull request on bios/, platforms/, emulators/, schemas/, scripts/, tests/ | validate BIOS hashes, schema check, run the full test suite, auto-label PR | | Weekly Sync | `watch.yml` | cron (Monday 6 AM UTC) + manual | scrape upstream sources, detect changes, create update PR | Build workflow has a 7-day rate limit between releases and keeps the 3 most recent. +The release job stays disabled (`if: false`) until pack generation is validated +in production. See the [release process](release-process.md). ## License See `LICENSE` at repo root. Files are provided for personal backup and archival. - diff --git a/wiki/faq.md b/wiki/faq.md index 165c507b..b68fc580 100644 --- a/wiki/faq.md +++ b/wiki/faq.md @@ -33,6 +33,13 @@ Two main reasons: Platforms that verify by MD5 accept specific hashes. If yours doesn't match any known hash, it may be a bad dump or an uncommon revision. +A filename is never allowed to stand in for a declared hash. When a file entry +carries a hash, resolution matches on content and reports `hash_mismatch` if the +only same-named copy disagrees. What happens next depends on the platform: an +MD5 or SHA1 platform would reject the file, so the pack omits it and says why; +an existence platform only checks presence, so the file ships and the divergence +is reported instead. + ## Why are there files that aren't BIOS? A file earns its place when an emulator loads it from disk and does not bundle it. That covers actual BIOS ROMs, console firmware, arcade BIOS sets, and also game data like `prboom.wad` for the PrBoom core or soundfonts for EasyRPG. diff --git a/wiki/getting-started.md b/wiki/getting-started.md index bd166636..151870ec 100644 --- a/wiki/getting-started.md +++ b/wiki/getting-started.md @@ -10,9 +10,10 @@ Three ways to get BIOS files in place, from easiest to most manual. ### Option 1: the installer (recommended) -One command, nothing to clone. It fetches `install.py`, detects the platform -and its BIOS directory, downloads only what is missing, and verifies SHA1 -checksums before writing anything. +One command, nothing to clone. The bootstrap verifies `install.py` against the +SHA-256 embedded in it, then the installer detects the platform and its BIOS +directory, downloads only what is missing or incorrect, checks size and +SHA-256/SHA-1 before writing, and installs each file atomically. ```bash # Linux / macOS / Steam Deck @@ -39,8 +40,16 @@ python install.py --list-platforms # supported platforms and what was detect python install.py --list-targets # hardware targets for a platform python install.py --jobs 4 # parallel downloads (default 8) python install.py --verbose +python install.py --standalone-copies # opt in to extra standalone-emulator paths ``` +The default flow writes inside the detected platform tree only. Copies into +separate standalone-emulator directories are opt-in, so discovery cannot cause +unexpected writes elsewhere on the machine. +Entries the collection cannot satisfy are reported as safely omitted and the +run continues; the installer never substitutes a same-named file for one a +hash-verifying platform would reject. + Arguments pass through the one-liner too, which is how you target an SD card mounted on another machine: diff --git a/wiki/release-process.md b/wiki/release-process.md index 8c12a274..bd2d1505 100644 --- a/wiki/release-process.md +++ b/wiki/release-process.md @@ -72,7 +72,19 @@ The list is the set of inputs the site is generated from. Adding a new input to `mkdocs.yml`) 5. Run `generate_readme.py` (rebuilds README.md and CONTRIBUTING.md) 6. `mkdocs build --strict` to produce the static site -7. Upload artifact, deploy to GitHub Pages +7. Run `validate_site.py` on the rendered HTML (metadata, headings, image + alternatives, duplicate ids, local links and fragments) +8. Require the committed README and CONTRIBUTING to match what the generator + just produced. `write_if_changed()` compares content with the timestamp line + stripped, so a run that only moves the clock leaves the files untouched and + the check stays meaningful +9. Upload artifact, deploy to GitHub Pages + +Data contracts are validated with `scripts/validate_schemas.py` before the site +is generated: `database.json`, the install and target manifests, the site API +envelopes and the stats file, plus the semantic invariants those schemas cannot +express (declared totals matching their lists, no destination both installed +and omitted). The site is deployed via the `github-pages` environment using the official `actions/deploy-pages` action. Pages deployments are queued rather than @@ -90,8 +102,8 @@ so a major theme release cannot change the site without a deliberate bump. ## validate.yml - PR Validation -**Trigger.** Pull requests that modify `bios/**`, `platforms/**` or -`emulators/**`. +**Trigger.** Pull requests that modify `bios/**`, `platforms/**`, +`emulators/**`, `schemas/**`, `scripts/**`, `tests/**` or `install.py`. **Concurrency.** Per-PR group, cancel in-progress. @@ -101,13 +113,13 @@ Four parallel jobs: `validate_pr.py --markdown` on each, and posts the validation report as a PR comment (hash verification, database match status). -**validate-configs.** Validates every platform YAML against -`schemas/platform.schema.json` and every emulator profile against -`schemas/emulator.schema.json`, using `jsonschema`. Fails if any file does not -match. `*.old.yml` files are skipped: they are hash-scraper backups, not -profiles. +**validate-configs.** Runs `python scripts/validate_schemas.py --source-only`, +which validates every platform YAML against `schemas/platform.schema.json` and +every emulator profile against `schemas/emulator.schema.json`. Both schemas set +`additionalProperties: false`, so a typo in a field name fails the job instead +of being silently ignored. -**run-tests.** Runs `python -m unittest tests.test_e2e -v`. Must pass before +**run-tests.** Runs `python -m unittest discover tests -v`. Must pass before merge. **label-pr.** Auto-labels the PR based on changed paths: diff --git a/wiki/testing-guide.md b/wiki/testing-guide.md index a225cbff..db0c8144 100644 --- a/wiki/testing-guide.md +++ b/wiki/testing-guide.md @@ -2,9 +2,10 @@ This page covers how to run, understand, and extend the test suite. -10 modules, 400 tests. No network access anywhere. Most modules build synthetic -fixtures in a temp directory; the three that read the working tree skip cleanly -when the data they need is absent. +The suite is discovered dynamically across focused modules, so its exact count +is reported by the test runner instead of being hand-maintained here. Tests do +not require network access. Most build synthetic fixtures under the repository's +`tmp/`; modules that read real artifacts skip cleanly when those inputs are absent. ## Running tests @@ -31,6 +32,10 @@ python -m unittest tests.test_large_file_cache -v python -m unittest tests.test_pack_integrity -v python -m unittest tests.test_torrentzip -v python -m unittest tests.test_no_case_collisions -v +python -m unittest tests.test_region -v +python -m unittest tests.test_audit_regressions -v +python -m unittest tests.test_site_exports -v +python -m unittest tests.test_site_validation -v ``` The only dependency is `pyyaml`. No test framework beyond the standard @@ -38,22 +43,26 @@ library `unittest` module. ## Modules at a glance -| Module | Tests | Fixtures | What it covers | -|--------|-------|----------|----------------| -| `test_e2e.py` | 218 | synthetic | resolution, verification, packs, cross-reference, targets, truth | -| `test_profile_sync.py` | 189 | synthetic | ref anchoring, guarded profile writes, detection, triage | -| `test_install.py` | 70 | synthetic | `install.py` detection, config parsing, manifest handling | -| `test_upstream.py` | 56 | synthetic | forge URL parsing, cache, revision resolution, tree comparison | -| `test_provenance.py` | 29 | synthetic | Logiqx/Redump parsing, DAT import, provenance join, coverage report | -| `test_mame_parser.py` | 25 | inline C | BIOS root sets, ROM blocks, macro expansion | -| `test_hash_merge.py` | 17 | synthetic | YAML hash merge, diff, formatting preservation | -| `test_fbneo_parser.py` | 16 | inline C | `BDF_BOARDROM` sets, ROM info parsing | -| `test_deterministic_zip.py` | 12 | synthetic | streaming rebuild, metadata normalisation, entry ordering, source CRC | -| `test_artifact_lock.py` | 10 | synthetic | writer/writer and writer/reader exclusion, reader sharing, release on error | -| `test_pack_integrity.py` | 8 | real packs | extract each ZIP, verify paths and hashes | -| `test_torrentzip.py` | 8 | real romsets | TorrentZip builder byte-for-byte | -| `test_large_file_cache.py` | 5 | synthetic | concurrent downloads, temporary file residue, hash rejection | -| `test_no_case_collisions.py` | 1 | real `bios/` | no case-colliding paths on Windows/macOS clones | +| Module | Fixtures | What it covers | +|--------|----------|----------------| +| `test_e2e.py` | synthetic | resolution, verification, packs, cross-reference, targets, truth | +| `test_profile_sync.py` | synthetic | ref anchoring, guarded profile writes, detection, triage | +| `test_install.py` | synthetic | platform detection, manifests, downloads and both automatic wrappers | +| `test_region.py` | synthetic + profiles | canonical vocabulary, ranking, grouping and repository conformance | +| `test_audit_regressions.py` | synthetic | pipeline exits, strong hashes, safe archives, manifest trust boundaries | +| `test_site_exports.py` | synthetic | source permalinks, API/catalog hashes, SQLite and page metadata | +| `test_site_validation.py` | rendered HTML | links, fragments, headings, alternatives and duplicate search metadata | +| `test_upstream.py` | synthetic | forge URL parsing, cache, revision resolution, tree comparison | +| `test_provenance.py` | synthetic | Logiqx/Redump parsing, DAT import, provenance join, coverage report | +| `test_mame_parser.py` | inline C | BIOS root sets, ROM blocks, macro expansion | +| `test_hash_merge.py` | synthetic | YAML hash merge, diff, formatting preservation | +| `test_fbneo_parser.py` | inline C | `BDF_BOARDROM` sets and ROM info parsing | +| `test_deterministic_zip.py` | synthetic | streaming rebuild, metadata normalisation, ordering and source CRC | +| `test_artifact_lock.py` | synthetic | writer/reader exclusion, sharing and lock release | +| `test_large_file_cache.py` | synthetic | concurrent downloads, temporary residue and hash rejection | +| `test_pack_integrity.py` | real packs | extract ZIPs and verify paths plus hashes | +| `test_torrentzip.py` | real romsets | TorrentZip builder byte-for-byte | +| `test_no_case_collisions.py` | real `bios/` | portable case-collision guard | ## Test architecture @@ -207,8 +216,9 @@ in the platform YAML: Handles inner ZIP verification for MAME/FBNeo ROM sets (checkInsideZip, md5_composite, inner ROM MD5) and path collision deduplication. -8 tests (one per active platform): RetroArch, Batocera, BizHawk, EmuDeck, -Recalbox, RetroBat, RetroDECK, RomM. +One test per distinct pack contract: RetroArch, Batocera, BizHawk, EmuDeck, +Recalbox, RetroBat, RetroDECK, RomM. Inherited aliases such as Lakka reuse the +same artifact and are not counted twice. ```bash python -m unittest tests.test_pack_integrity -v @@ -228,6 +238,7 @@ The test suite is one layer of verification. The full quality gate is: 3. No unexpected CRITICAL entries in the verify output 4. Pack file counts match verification file counts (consistency check) 5. Pack integrity passes (every declared file extractable with correct hash) +6. `mkdocs build --strict` and `python scripts/validate_site.py` pass If a change passes tests but breaks the pipeline, it's worth investigating before merging. Similarly, new CRITICAL entries in the verify output after a change usually indicate something to look into. The pipeline is designed so that all steps agree: if verify reports N files for a platform, the pack should contain exactly N files. @@ -235,18 +246,17 @@ Ideally, tests, code, and documentation ship together. When profiles and platfor ## CI integration -The `validate.yml` workflow runs `test_e2e` on every pull request that touches -`bios/` or `platforms/` files. The test job (`run-tests`) runs in parallel -with BIOS validation, schema validation, and auto-labeling. `build.yml` runs -the same module before building release packs. +The `validate.yml` workflow runs `python -m unittest discover tests -v` on every +pull request that touches `bios/`, `platforms/`, `emulators/`, `schemas/`, +`scripts/`, `tests/` or `install.py`. The test job (`run-tests`) runs in parallel +with BIOS validation, schema validation, and auto-labeling. -CI runs `test_e2e` only, not the whole suite: the other modules either need -build artifacts (`test_pack_integrity` needs packs in `dist/`) or duplicate -work the workflow already does. Run `python -m unittest discover tests` locally -before pushing. +Modules that need real artifacts skip themselves when those artifacts are absent, +so `test_pack_integrity` is a no-op in CI (no `dist/`) and a real check locally. +Run `python -m unittest discover tests` before pushing. Tests must pass before merge. If a test fails in CI, reproduce locally with: ```bash -python -m unittest tests.test_e2e -v 2>&1 | head -50 +python -m unittest discover tests -v ``` diff --git a/wiki/tools.md b/wiki/tools.md index 174b0843..644327d8 100644 --- a/wiki/tools.md +++ b/wiki/tools.md @@ -145,7 +145,7 @@ python scripts/generate_pack.py --all --verify-packs --output-dir dist/ # extra # Data refresh python scripts/generate_pack.py --all --refresh-data # force re-download data dirs -python scripts/generate_pack.py --all --offline # skip the refresh entirely +python scripts/generate_pack.py --all --offline # cache-only; never open the network # Install manifests (consumed by install.py) python scripts/generate_pack.py --all --manifest --output-dir install/ @@ -156,6 +156,12 @@ Packs include platform baseline files plus files required by the platform's core When a file passes platform verification but fails emulator validation, the tool searches for a variant that satisfies both. If none exists, the platform version is kept and the discrepancy is reported. +What happens when a declared hash and the local dump disagree follows the +platform's own verification mode. A hash platform would reject the file, so it +is omitted and recorded in the pack report and the installer manifest. An +existence platform reads no bytes, so the file is packed and the divergence is +reported instead: an upstream list error must not remove a file the frontend +would have loaded. **Granular options:** @@ -328,6 +334,9 @@ python scripts/refresh_data_dirs.py --registry path/to/_data_dirs.yml | `provenance_report.py` | Dump-catalog coverage and acquisition targets (see above) | | `generate_readme.py` | Generate README.md and CONTRIBUTING.md from database | | `generate_site.py` | Generate all MkDocs site pages (this documentation) | +| `validate_site.py` | Validate rendered metadata, headings, image alternatives, JSON-LD, local resources, links and fragments | +| `romset_recipes.py` | Identify which emulator version an arcade archive matches, and rebuild a pinned archive from ROMs already held | +| `scraper/romset_dat_importer.py` | Fetch and import per-set recipes from MAME `-listxml` or FBNeo DATs into `recipes/` | | `deterministic_zip.py` | Rebuild MAME BIOS ZIPs deterministically (same ROMs = same hash) | | `torrentzip.py` | Build TorrentZip archives for MAME/FBNeo ROM sets (archive bytes depend only on contents) | | `crypto_verify.py` | 3DS RSA signature and AES crypto verification | @@ -343,18 +352,69 @@ Cross-platform BIOS installer for end users: # Python installer (auto-detects platform) python install.py -# Shell one-liner (Linux/macOS) -bash scripts/download.sh retroarch ~/RetroArch/system/ -bash scripts/download.sh --list +# One-line automatic install (Linux/macOS) +curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -# Or via install.sh wrapper (detects curl/wget, runs install.py) -bash install.sh +# One-line automatic install (Windows PowerShell) +irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex ``` `install.py` auto-detects the user's platform by checking config files, -downloads the matching BIOS pack from GitHub releases with SHA1 verification, -and extracts files to the correct directory. `install.ps1` provides -equivalent functionality for Windows/PowerShell. +downloads the missing files from an immutable release revision, verifies their +SHA-256 and SHA-1, and replaces each destination atomically. The shell and +Both one-liners verify the downloaded Python installer against the SHA-256 +embedded in their bootstrap before running it. Copies into standalone-emulator +directories are opt-in through `--standalone-copies`. Manifest entries with no +available payload are reported as safely omitted; they are never replaced by a +same-named file. + +`scripts/download.sh` remains available for downloading a prebuilt platform +ZIP when a manually reviewed pack release contains it; it is separate from the +per-file automatic installer above. + +## Romset recipes + +A profile's `contents:` block lists the member names and CRC32s one emulator +version expects inside an archive. TorrentZip makes archive bytes a function of +that list alone, so a recipe plus the ROM bytes reproduces the archive exactly. + +```bash +python scripts/romset_recipes.py --identify # which version each archive is +python scripts/romset_recipes.py --missing # pinned archives we lack +python scripts/romset_recipes.py --missing --write # write what can be rebuilt +``` + +Recipes come from two places. Profiles document a few hundred sets by hand. A +DAT documents every set of one emulator version at once, and both upstreams +publish theirs without a browser: MAME ships its `-listxml` as a release asset, +FBNeo keeps its DATs in the repository. + +```bash +python -m scripts.scraper.romset_dat_importer --source mame --fetch mame0289 +python -m scripts.scraper.romset_dat_importer --source mame --fetch mame0250 +python -m scripts.scraper.romset_dat_importer --source fbneo --fetch +``` + +Snapshots land in `recipes/`, not `provenance/`: the latter holds dump catalogues +and a recipe is not one. Versions accumulate rather than replace each other, +because a platform pins the archive of whichever version its list was built +against. Only sets a profile or platform references are kept, a `romof` parent's +members are merged into the child, and members with no CRC32 are dropped: those +are undumped and a real romset does not carry them. + +The store is compacted: 22 MAME versions produce 23,679 entries but only 1,726 +distinct recipes, since most sets do not change between releases. Each is kept +once, `dats` listing every version that agrees and `dat` naming the earliest, so +an identification reads as "unchanged since that version". + +`--identify` answers what an archive on disk actually is: with 22 MAME versions +and the FBNeo DATs imported, 1,113 archives are reproduced byte for byte. `--missing` takes the container MD5s the platforms pin, keeps those the +collection does not have, and tries every recipe whose ROMs are present. + +Reconstruction only works when the pinned archive is itself TorrentZip. An +archive whose bytes carry metadata unrelated to its contents cannot be derived +from ROMs by anyone; those are reported as unreproducible rather than guessed +at. Writing is explicit: the pipeline only ever reports. ## Large files diff --git a/wiki/troubleshooting.md b/wiki/troubleshooting.md index 364e84a9..129e6161 100644 --- a/wiki/troubleshooting.md +++ b/wiki/troubleshooting.md @@ -211,7 +211,8 @@ Problems with `install.py`, `install.sh`, or `download.sh`. **Network issues:** -The installer downloads packs from GitHub releases. If the download fails: +The installer fetches its manifest, then downloads only the repository or +`large-files` release objects it needs. If a download fails: - Check your internet connection - Verify that `https://github.com` is reachable @@ -238,7 +239,15 @@ python install.py --platform batocera --dest /userdata/bios/ Use `python install.py --help` to see all available platforms and options. -**Pack not found in release:** +**Manifest or file not found:** + +The one-line bootstrap verifies the downloaded installer before running it, and +the installer reads its manifests from the same revision. A 404 usually means +the platform name is wrong; check `python install.py --list-platforms`. +`RETROBIOS_REF` pins an installation to a published tag when a reproducible +file set matters more than the newest one. + +**Manual pack not found in release:** If the installer reports that no pack exists for your platform, check available releases: diff --git a/wiki/verification-modes.md b/wiki/verification-modes.md index 524897ad..3413960d 100644 --- a/wiki/verification-modes.md +++ b/wiki/verification-modes.md @@ -219,28 +219,40 @@ actionable. ## File Resolution Chain Before verification, each file entry is resolved to a local path by `resolve_local_file()`. -The function tries these steps in order, returning the first match: +The function tries these steps in order, returning the first +evidence-compatible match: | Step | Method | Returns | When it applies | |------|--------|---------|-----------------| -| 0 | Path suffix exact | `exact` | `dest_hint` matches `by_path_suffix` index (regional variants with same filename, e.g., `GC/USA/IPL.bin` vs `GC/EUR/IPL.bin`) | -| 1 | SHA1 exact | `exact` | SHA1 present in the file entry and found in database. A list-valued `sha1` from a profile is accepted | -| 1b | SHA256 exact | `exact` | SHA256 present, for profiles read from sources that publish SHA256 (e.g. MesenCE) | -| 1c | CRC32 + size exact | `exact` | Only when no stronger hash is declared (CRC-only profiles such as FBNeo and Clock Signal). CRC32 alone is weak, so the declared size has to confirm the match | -| 2 | MD5 direct lookup | `md5_exact` | MD5 present, not a `zipped_file` entry. A full 32-char MD5 is trusted on its own; a truncated MD5 also has to match the name, which prevents cross-contamination from unrelated files sharing a prefix | -| 3 | Name/alias existence | `exact` | No MD5 in entry; any file with matching name or alias exists. Prefers primary over `.variants/`, with a case-insensitive fallback for upstreams that disagree on casing | -| 4 | Name + md5_composite/MD5 | `exact` or `hash_mismatch` | Name matches, checks md5_composite for ZIPs and direct MD5 per candidate. Falls back to hash_mismatch if name matches but no hash does | -| 5 | ZIP contents index | `zip_exact` | `zipped_file` with MD5; searches inner ROM MD5 across all ZIPs when name-based resolution failed | -| 6 | MAME clone fallback | `mame_clone` | File was deduped; resolves via canonical set name (up to 3 levels deep) | -| 7 | Data directory scan | `data_dir` | Searches `data/` caches by exact path then case-insensitive basename walk | -| 8 | Agnostic fallback | `agnostic_fallback` | File entry marked `agnostic: true`; matches any file under the system path prefix within the size constraints | +| 1 | SHA1 | `sha1_exact` | A declared SHA1 identifies a database record; every other declared hash must agree with that same record | +| 2 | SHA256 | `sha256_exact` | A declared SHA256 identifies a record, again requiring all declarations to agree | +| 3 | CRC32 plus size | `crc32_exact` | Used only when no stronger hash is declared; size confirms the weaker checksum when available | +| 4 | MD5 | `md5_exact` | Direct lookup; an explicitly supported truncated MD5 also needs a compatible name | +| 5 | Path suffix | `path_exact` or a hash-exact status | Disambiguates regional paths. With any declared hash, the path is accepted only when the content matches it | +| 6 | Name or alias | `name_exact` | Existence/size resolution only when no content hash is declared; prefers primary files and supports casefold matching | +| 7 | Named candidate inspection | `md5_composite_exact`, `md5_exact`, or `hash_mismatch` | Checks composite ZIP MD5 or direct MD5. A matching name with wrong content is surfaced, never treated as exact | +| 8 | ZIP contents index | `zip_exact` | `zipped_file` with MD5; searches the inner-ROM index only after name-based resolution fails | +| 9 | MAME clone | `mame_clone` | Resolves a deduplicated clone to its canonical set only when no content hash was declared | +| 10 | Data directory | `data_dir` or `data_dir_hash_exact` | Searches exact path then case-insensitive basename; computes every declared hash before accepting an unindexed candidate | +| 11 | Agnostic fallback | `agnostic_fallback` | Size/path constrained lookup only when no content hash was declared | If no step matches, the result is `(None, "not_found")`. -The `hash_mismatch` status at step 4 means a file with the right name exists but its hash +The `hash_mismatch` status means a file with the right name or path exists but its hash does not match. This still resolves to a local path (the file is present), but verification will report it as UNTESTED with a reason string showing the expected vs actual hash prefix. +A path or filename is never allowed to mask a declared strong hash: the +identity steps run first, and a path or name is accepted only when no content +hash was declared or when the content agrees with the one that was. + +What the pack does with a `hash_mismatch` follows the platform's own mode. An +MD5 or SHA1 platform would reject those bytes, so the file is left out and +counted as an unsafe exclusion, distinct from a file the collection does not +have. An existence platform reads no bytes at all, so the file is packed and +the divergence is printed as a discrepancy: an error in an upstream BIOS list +must not remove a file the frontend would have loaded. + ## Discrepancy Detection @@ -261,8 +273,8 @@ both the platform MD5 requirement and emulator validation: The search covers files in `.variants/` (alternate hashes stored during deduplication). If a better variant is found, the pack uses it instead of the primary file. If no variant -satisfies both constraints, the platform version is kept and the discrepancy is reported -in the verification output. +satisfies both independent contracts, the platform-verified baseline is retained but the +emulator discrepancy remains explicit; it is never reported as source-compatible. ### Practical example @@ -270,5 +282,5 @@ A `scph5501.bin` file passes Batocera MD5 verification (hash matches upstream de but fails the emulator profile's size check because the profile was verified against a different revision. `_find_best_variant` scans `.variants/scph5501.bin.*` for a file that matches both the Batocera MD5 and the emulator's size expectation. If found, the -variant is used in the pack. If not, the Batocera-verified file is kept and the discrepancy -is logged. +variant is used in the pack. If not, the Batocera-verified baseline is retained and the +emulator discrepancy stays visible in verification and gap reporting.