diff --git a/README.md b/README.md index effdc82c..cda6ef09 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ BIOS and firmware files for Batocera, BizHawk, EmuDeck, Lakka, MiSTer FPGA, ROCK A pack holds the files its emulators read, as their source code names them, and leaves out what the emulators install themselves. -## Quick Install +## Quick install Copy one command into your terminal: @@ -34,12 +34,13 @@ The script auto-detects your platform, downloads only missing files, and verifie ## Download BIOS packs One pack per platform, and it holds everything the platform runs: its own BIOS list plus every file its emulator cores load. Pick your platform, download the ZIP, extract to the BIOS path. The installer above does the same file by file. At a terminal it asks whether to install everything or to choose systems, cores and regions, and `--system`, `--core`, `--region` and `--target` make the same choice without questions. For a bare minimum, build your own pack below. -Files is how many the pack holds once extracted, the figure a file manager shows for the folder. It differs from one platform to the next because a pack carries only what that platform's emulators load. The size is what the files occupy once extracted, and the ZIP itself downloads smaller. A pack over 2 GB comes in several parts (`Pack.part1of2.zip`, `Pack.part2of2.zip`), and every part is needed. Each part is an ordinary ZIP: extract them all into the same folder. Every release ships `SHA256SUMS.txt` and a detached signature of it, checkable against `allowed_signers` in this repository: [verifying a release](https://abdess.github.io/retrobios/wiki/release-process/#verifying-a-release). +The Files column counts what a pack holds once extracted, the figure a file manager shows for the folder. It differs from one platform to the next because a pack carries only what that platform's emulators load. The size is what the files occupy once extracted, and the ZIP itself downloads smaller. + | Platform | Files | Extracted size | Extract to | Download | |----------|------:|---------------:|-----------|----------| | Batocera | 12,810 | 7.5 GB | `/userdata/bios/` | Download: [part 1 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part3of3.zip) | diff --git a/scripts/generate_pack.py b/scripts/generate_pack.py index 5bb88852..f857a1dc 100644 --- a/scripts/generate_pack.py +++ b/scripts/generate_pack.py @@ -1252,16 +1252,10 @@ class DamagedArchiveError(OSError): def _normalize_zip_for_pack( source_zip: str, dest_path: str, target_zf: zipfile.ZipFile ) -> None: - """Add a MAME BIOS ZIP to the pack as a deterministic rebuild. + """Add a MAME BIOS ZIP to the pack, rebuilt from its ROMs. - Instead of copying the original ZIP (with non-deterministic metadata), - extracts the ROM atoms, rebuilds the ZIP deterministically, and writes - the normalized version into the pack. - - This ensures: - - Same ROMs -> same ZIP hash in every pack build - - No dependency on how the user built their MAME ROM set - - Bit-identical ZIPs across platforms and build times + The same ROMs give the same archive bytes in every build, whatever + metadata the source ZIP carried. """ import tempfile as _tmp diff --git a/scripts/generate_readme.py b/scripts/generate_readme.py index 1656270b..74af5c30 100644 --- a/scripts/generate_readme.py +++ b/scripts/generate_readme.py @@ -380,7 +380,7 @@ def generate_readme(db: dict, platforms_dir: str) -> str: "A pack holds the files its emulators read, as their source code names" " them, and leaves out what the emulators install themselves.", "", - "## Quick Install", + "## Quick install", "", "Copy one command into your terminal:", "", @@ -404,12 +404,6 @@ def generate_readme(db: dict, platforms_dir: str) -> str: "## Download BIOS packs", "", "One pack per platform, and it holds everything the platform runs: its own BIOS list plus every file its emulator cores load. Pick your platform, download the ZIP, extract to the BIOS path. The installer above does the same file by file. At a terminal it asks whether to install everything or to choose systems, cores and regions, and `--system`, `--core`, `--region` and `--target` make the same choice without questions. For a bare minimum, build your own pack below.", - "Files is how many the pack holds once extracted, the figure a file" - " manager shows for the folder. It differs from one platform to the" - " next because a pack carries only what that platform's emulators" - " load." - " The size is what the files occupy once extracted, and the ZIP itself" - " downloads smaller.", "", "A pack over 2 GB comes in several parts (`Pack.part1of2.zip`," " `Pack.part2of2.zip`), and every part is needed. Each part is an" @@ -421,6 +415,12 @@ def generate_readme(db: dict, platforms_dir: str) -> str: "(https://abdess.github.io/retrobios/wiki/release-process/" "#verifying-a-release).", "", + "The Files column counts what a pack holds once extracted, the figure a" + " file manager shows for the folder. It differs from one platform to" + " the next because a pack carries only what that platform's emulators" + " load. The size is what the files occupy once extracted, and the ZIP" + " itself downloads smaller.", + "", ] # Where the pack itself is extracted, which is not always the BIOS folder: diff --git a/scripts/truth.py b/scripts/truth.py index 10f3cc33..718387a6 100644 --- a/scripts/truth.py +++ b/scripts/truth.py @@ -39,27 +39,13 @@ def _serialize_source_ref(sr: object) -> str: def _enrich_hashes(entry: dict, db: dict) -> None: - """Fill missing sibling hashes from the database, ground-truth preserving. + """Fill the hashes a profile omits from the record its own hash names. - The profile's hashes come from the emulator source code (ground truth). - Any hash of a given file set of bytes is a projection of that same - ground truth. sha1, md5 and crc32 all identify the same bytes. If the - profile has ONE ground-truth hash, the DB can supply its siblings. - - Lookup order (all are hash-anchored, never name-based): - 1. SHA1 direct - 2. MD5 -> SHA1 via indexes.by_md5 - 3. CRC32 -> SHA1 via indexes.by_crc32 (weaker 32-bit anchor, - requires size match when profile has size) - - Name-based enrichment is NEVER used: a name alone has no ground-truth - anchor, the file in bios/ may not match what the source code expects. - - Multi-hash entries (lists of accepted variants) are left untouched to - preserve variant information. + The record is found by sha1, then md5, then crc32 with the size when the + profile gives one. Never by name: a file in bios/ under the same name may + not be the bytes the source code expects. An entry listing several + accepted values for a hash is left as is. """ - # Skip multi-hash entries. They express ground truth as "any of these N - # variants", enriching with a single sibling would lose that information. for h in ("sha1", "md5", "crc32"): if isinstance(entry.get(h), list): return @@ -105,7 +91,6 @@ def _enrich_hashes(entry: dict, db: dict) -> None: return # Copy sibling hashes and size from the anchored record. - # These are projections of the same ground-truth bytes. for field in ("sha1", "md5", "sha256", "crc32"): if not entry.get(field) and record.get(field): entry[field] = record[field] diff --git a/tests/test_audit_regressions.py b/tests/test_audit_regressions.py index 4124036f..cb392ce6 100644 --- a/tests/test_audit_regressions.py +++ b/tests/test_audit_regressions.py @@ -54,7 +54,7 @@ class ReadmeRegressions(unittest.TestCase): def _quick_install(self) -> str: readme = (ROOT / "README.md").read_text(encoding="utf-8") - return readme.split("## Quick Install", 1)[1].split( + return readme.split("## Quick install", 1)[1].split( "## Download BIOS packs", 1 )[0] diff --git a/tests/test_doc_pages.py b/tests/test_doc_pages.py new file mode 100644 index 00000000..153db369 --- /dev/null +++ b/tests/test_doc_pages.py @@ -0,0 +1,420 @@ +"""What the wiki and the generated README state holds against the code. + +Each class reads a page and the code it describes. A page that quotes a +function's return shape, a resolution order or a field list drifts when the +code moves, and nothing else notices. +""" + +from __future__ import annotations + +import ast +import hashlib +import inspect +import io +import json +import re +import sys +import unittest +import zipfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +WIKI = ROOT / "wiki" +sys.path.insert(0, str(ROOT / "scripts")) + +from tests.test_prose_style import _python_prose, _sentence_case # noqa: E402 + +FENCED = re.compile(r"^```.*?^```", re.S | re.M) +PUBLISHED = sorted(WIKI.glob("*.md")) + [ROOT / "README.md", ROOT / "CONTRIBUTING.md"] + + +def _page(name: str) -> str: + return (WIKI / name).read_text(encoding="utf-8") + + +def _section(text: str, heading: str) -> str: + """Body of a markdown section, up to the next heading of any level.""" + start = re.search(rf"(?m)^#+ {re.escape(heading)}\n", text) + if start is None: + raise AssertionError(f"no section {heading!r}") + rest = text[start.end():] + blanked = FENCED.sub(lambda m: re.sub(r"[^\n]", " ", m.group(0)), rest) + end = re.search(r"(?m)^#+ ", blanked) + return rest[: end.start()] if end else rest + + +def _table_rows(text: str) -> list[list[str]]: + rows = [] + for line in text.splitlines(): + if line.startswith("|") and not re.match(r"\|\s*-", line): + rows.append([cell.strip() for cell in line.strip().strip("|").split(" | ")]) + return rows[1:] + + +class TestSkeletonReadsTheResultShape(unittest.TestCase): + """The skeleton indexed verify_platform's result as a list, and a test + copied from it stopped on KeyError: 0 before any assertion.""" + + def test_every_key_it_reads_is_one_verify_platform_returns(self): + from verify import verify_platform + + result = verify_platform( + {"platform": "x", "verification_mode": "existence", "systems": {}}, + {"files": {}, "indexes": {}}, + emu_profiles={}, + ) + text = _page("testing-guide.md").split("Example skeleton:", 1)[1] + code = re.search(r"```python\n(.*?)```", text, re.S).group(1) + read = [ + node.slice + for node in ast.walk(ast.parse(code)) + if isinstance(node, ast.Subscript) + and isinstance(node.value, ast.Name) + and node.value.id == "result" + ] + self.assertTrue(read, "the skeleton no longer reads the result") + for key in read: + self.assertIsInstance(key, ast.Constant) + self.assertIn(key.value, result) + + +class InstallerOutputExample(unittest.TestCase): + """The page showed a run with no `Not installable:` line while the + RetroArch manifest left eleven entries out, and a count that the menu + example below it no longer shared.""" + + def test_run_and_menu_show_one_dated_run_of_the_manifest(self): + import install + + text = _page("installer.md") + intro, run = re.search(r"(?s)\n([^\n]*)\n\n```\n(Fetching file index.*?)```", text).groups() + self.assertRegex(intro, r"\d{4}-\d{2}-\d{2}", "an output example says when it was taken") + manifest = json.loads((ROOT / "install" / "retroarch.json").read_text(encoding="utf-8")) + omitted = re.search(r"Not installable: \d+ files? \(\d+ required, (.*)\)\.", run) + if manifest.get("omitted_files"): + self.assertIsNotNone(omitted, "the manifest omits files and the run shows no line for it") + if omitted: + for part in omitted.group(1).split(", "): + self.assertIn(part.rsplit(":", 1)[0], install.OMISSION_REASONS.values()) + count, size = re.search(r"\n (\d+) files \(([\d.]+ [KMG]B)\)", run).groups() + self.assertIn(f"0/{count} present", run) + menu = re.search(r"RetroArch: (\d+) files, ([\d.]+ [KMG]B)\.", text) + self.assertEqual(menu.groups(), (count, size)) + + +class OneResolutionChain(unittest.TestCase): + """architecture.md drew path_suffix as the first step, before SHA1, while + the code and verification-modes.md try it after every hash.""" + + ORDER = ("SHA1", "SHA256", "CRC32", "MD5", "Path suffix") + + def test_the_table_follows_the_code(self): + import common + + source = inspect.getsource(common.resolve_local_file) + steps = re.findall(r"# \d+[a-z]?\. (SHA1|SHA256|CRC32|MD5|Path suffix)", source) + self.assertEqual(tuple(dict.fromkeys(steps)), self.ORDER) + table = _table_rows(_section(_page("verification-modes.md"), "File resolution chain")) + methods = [row[1] for row in table[: len(self.ORDER)]] + self.assertEqual( + [next(m for m in self.ORDER if method.startswith(m)) for method in methods], + list(self.ORDER), + ) + + def test_architecture_links_instead_of_drawing_a_second_one(self): + section = _section(_page("architecture.md"), "File resolution chain") + self.assertNotIn("```mermaid", section) + self.assertIn("verification-modes.md#file-resolution-chain", section) + + +class ReleaseRefPromise(unittest.TestCase): + """RETROBIOS_REF was said to pin a reproducible file set, while every + release deletes the previous tags and large files come from one fixed + release whatever the ref.""" + + def test_the_page_says_what_a_tag_does_not_pin(self): + import install + + self.assertIn("/large-files/", install.RELEASE_URL) + self.assertIn("--cleanup-tag", _page("release-process.md")) + row = next( + line for line in _page("installer.md").splitlines() + if line.startswith("| `RETROBIOS_REF`") + ) + self.assertIn("`large-files`", row) + self.assertIn("next release", row) + for page in PUBLISHED: + for paragraph in page.read_text(encoding="utf-8").split("\n\n"): + if "RETROBIOS_REF" in paragraph: + self.assertNotIn("reproducible", paragraph, page.name) + + +class RequirementFieldsTable(unittest.TestCase): + """The table stopped at native_id: native_path, mirror_of, alt_md5 and + the native dict, the fields that make a platform YAML reversible, were + named nowhere a scraper author reads first.""" + + def test_every_field_and_native_key_is_listed(self): + from dataclasses import fields + + from scraper.base_scraper import BiosRequirement + + section = _section(_page("adding-a-platform.md"), "BiosRequirement fields") + listed = {row[0].strip("`") for row in _table_rows(section)} + self.assertEqual(listed, {f.name for f in fields(BiosRequirement)}) + native_row = next(row for row in _table_rows(section) if row[0] == "`native`")[2] + keys: set[str] = set() + for path in (ROOT / "scripts" / "scraper").glob("*_scraper.py"): + source = path.read_text(encoding="utf-8") + keys |= set(re.findall(r'native\["(\w+)"\]', source)) + keys |= set(re.findall(r'native(?:: dict\[str, object\])?\s*=\s*\{"(\w+)"', source)) + self.assertTrue(keys) + for key in keys: + self.assertIn(f"`{key}`", native_row) + + +class ReleaseCommitCarriesTheManifests(unittest.TestCase): + """The documented release commit left out install/*.json, which the + release pipeline regenerates: v2026.10.10 added three by hand.""" + + def test_the_commit_step_adds_what_the_pipeline_rewrites(self): + pipeline = (ROOT / "scripts" / "pipeline.py").read_text(encoding="utf-8") + self.assertIn('"--manifest"', pipeline) + block = _section(_page("release-process.md"), "Cutting a release") + add = next(line for line in block.splitlines() if line.startswith("git add ")) + self.assertIn("install/*.json", add.split()) + + +class PublishedHeadings(unittest.TestCase): + """README.md kept '## Quick Install', the one Title Case heading left, + because the casing test read wiki/ only.""" + + def test_readme_and_contributing_use_sentence_case(self): + offenders = [] + for page in (ROOT / "README.md", ROOT / "CONTRIBUTING.md"): + text = FENCED.sub("", page.read_text(encoding="utf-8")) + for title in re.findall(r"(?m)^#{1,4} (.+)$", text): + if _sentence_case(title) != title: + offenders.append(f"{page.name}: {title}") + self.assertEqual(offenders, [], "\n".join(offenders)) + + +class FilesColumnParagraph(unittest.TestCase): + """'Files is how many the pack holds' was glued to the paragraph above + it, two paragraphs before the table whose column it described.""" + + def test_it_names_the_column_in_its_own_paragraph_before_the_table(self): + import generate_readme + + source = inspect.getsource(generate_readme) + self.assertNotIn("Files is how many", source) + readme = (ROOT / "README.md").read_text(encoding="utf-8") + paragraph = next(p for p in readme.split("\n\n") if "The Files column" in p) + self.assertNotIn("\n", paragraph.strip()) + after = readme.split(paragraph, 1)[1].lstrip("\n") + self.assertTrue(after.startswith("| Platform | Files |")) + + +class PageOpenings(unittest.TestCase): + """Four pages opened by announcing what they cover.""" + + META = re.compile(r"(?i)\bthis (page|guide) (covers|documents|explains|describes|picks)\b|^Diagnosis guide") + + def test_no_page_announces_itself(self): + offenders = [ + f"{page.name}: {line[:80]}" + for page in PUBLISHED + for line in FENCED.sub("", page.read_text(encoding="utf-8")).splitlines() + if self.META.search(line) + ] + self.assertEqual(offenders, [], "\n".join(offenders)) + + +class NoHedgedRules(unittest.TestCase): + """The profiling guide and the verification discipline turned rules into + wishes ("it's worth", "Ideally", "something to look into").""" + + FILLER = re.compile( + r"(?i)\bit'?s worth\b|\bworth noting\b|\bit helps to\b|^Ideally\b" + r"|\bsomething to look into\b|\busually indicate\b" + ) + + def test_published_pages(self): + offenders = [ + f"{page.name}: {line[:80]}" + for page in PUBLISHED + for line in FENCED.sub("", page.read_text(encoding="utf-8")).splitlines() + if self.FILLER.search(line) + ] + self.assertEqual(offenders, [], "\n".join(offenders)) + + +class DocstringsWithoutInsistence(unittest.TestCase): + """truth.py shouted ONE and NEVER in an eighteen-line docstring, and + generate_pack.py listed three guarantees under 'This ensures:' of which + two said the same thing.""" + + TICS = re.compile(r"\b(NEVER|ALWAYS|ONE|MUST)\b|This ensures:") + + def test_scripts_and_installer(self): + offenders = [] + paths = sorted((ROOT / "scripts").rglob("*.py")) + [ROOT / "install.py"] + for path in paths: + for line in _python_prose(path.read_text(encoding="utf-8")).splitlines(): + if self.TICS.search(line): + offenders.append(f"{path.relative_to(ROOT)}: {line.strip()[:80]}") + self.assertEqual(offenders, [], "\n".join(offenders)) + + +class PeerToneTowardPlatforms(unittest.TestCase): + """Platform lists were called wrong, stale or corrected on public pages + that their maintainers read.""" + + WORDS = re.compile( + r"(?i)\bgets wrong\b|\bstale (hash|entries|entry)\b||\bpatch their\b" + r"|, corrected\b|\bcorrected in place\b|\bdeclare incorrectly\b|\blist error\b" + ) + + def test_published_pages(self): + offenders = [ + f"{page.name}: {line[:80]}" + for page in PUBLISHED + for line in page.read_text(encoding="utf-8").splitlines() + if self.WORDS.search(line) + ] + self.assertEqual(offenders, [], "\n".join(offenders)) + + +class RecalboxSeverity(unittest.TestCase): + """The table gave verify.py a mapping from hashMatchMandatory that the + code never reads, and called a missing optional file GREEN.""" + + def test_the_verify_column_is_what_compute_severity_returns(self): + from verify import Status, compute_severity + + section = _section(_page("verification-modes.md"), "Mandatory levels (Recalbox)") + rows = _table_rows(section) + self.assertTrue(rows) + status = {"hash matches": Status.OK, "missing": Status.MISSING, "hash differs": Status.UNTESTED} + for state, mandatory, _hash_match, _colour, expected in rows: + for required in ((True, False) if mandatory == "(any)" else (mandatory == "true",)): + self.assertEqual( + compute_severity(status[state], required, "md5").upper(), expected, + f"{state} mandatory={mandatory}", + ) + self.assertNotIn("severity levels", _page("tools.md")) + + +class Md5CompositeDefinition(unittest.TestCase): + """The page described the MD5 of the members' MD5s, the code hashes the + members' contents in name order.""" + + def test_page_and_code_agree(self): + from hashing import md5_composite_of + + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w") as zf: + zf.writestr("b.rom", b"BBBB") + zf.writestr("a.rom", b"AAAA") + buffer.seek(0) + self.assertEqual(md5_composite_of(buffer), hashlib.md5(b"AAAABBBB").hexdigest()) + section = _section(_page("verification-modes.md"), "md5_composite (Recalbox ZIP verification)") + self.assertIn("contents of every member", section) + self.assertNotIn("inner file MD5s", section) + + +class ValidationIndexScope(unittest.TestCase): + """The page said the index reads every profile. verify and the builder + pass it only the profiles the platform resolves.""" + + def test_page_names_the_scope_the_code_uses(self): + import verify + + source = inspect.getsource(verify.verify_platform) + self.assertRegex(source, r"plat_cores = resolve_platform_cores\(config, profiles\)") + self.assertIn("_build_validation_index(platform_profiles)", source) + section = _section(_page("verification-modes.md"), "Validation index") + self.assertIn("resolve_platform_cores", section) + self.assertNotIn("reads all emulator profiles", section) + + +class CoreResolutionWrittenOnce(unittest.TestCase): + """architecture.md and adding-a-platform.md each stated the core + resolution rule, and only one knew that a list naming retroarch takes + every libretro profile. Neither said built_by profiles stay out.""" + + def test_one_statement_matching_the_platforms_and_the_code(self): + import yaml + + import common + + self.assertNotIn("cores: [list]", _page("architecture.md")) + source = inspect.getsource(common.resolve_platform_cores) + self.assertIn('not p.get("built_by")', source) + self.assertIn('"retroarch" in core_set', source) + naming = set() + for path in (ROOT / "platforms").glob("*.yml"): + config = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + cores = config.get("cores") + if isinstance(cores, list) and "retroarch" in map(str, cores): + naming.add(str(config.get("platform") or path.stem).lower()) + section = _section(_page("adding-a-platform.md"), "Required fields") + bullet = next(p for p in section.split("\n- ") if p.startswith("**Explicit list**")) + used_by = re.search(r"Used by ([^.]+)\.", bullet.replace("\n", " ")).group(1) + names = [n.strip().lower() for n in used_by.split(",")] + self.assertIn("the last\n four", bullet) + self.assertEqual(set(names[-4:]), naming) + libretro = next(p for p in section.split("\n- ") if p.startswith("**all_libretro**")) + self.assertIn("built_by", libretro) + + +class ExitStatusTable(unittest.TestCase): + """Code 2 was documented as 'every file failed' only, while argparse + exits 2 on every refused option, and the 141 example named an option + that does not exist.""" + + def test_codes_and_options_are_the_installers(self): + source = (ROOT / "install.py").read_text(encoding="utf-8") + section = _section(_page("installer.md"), "Exit status") + documented = {int(row[0].strip("`")) for row in _table_rows(section)} + raised = {int(code) for code in re.findall(r"sys\.exit\((\d+)\)", source)} + self.assertEqual(documented, raised | {2}) + self.assertIn("install.py: error:", section) + for option in set(re.findall(r"(--[a-z][a-z-]+)", section)): + self.assertIn(f'"{option}"', source, option) + + +class ProfileFieldsRequired(unittest.TestCase): + """The table marked source, profiled_date, core_version and systems as + required and called the schema authoritative, which requires none.""" + + def test_yes_matches_the_schema(self): + schema = json.loads((ROOT / "schemas" / "emulator.schema.json").read_text(encoding="utf-8")) + required = set(schema["required"]) + for rule in schema.get("allOf", []): + for branch in ("then", "else"): + required |= set(rule.get(branch, {}).get("required", [])) + section = _section(_page("profiling.md"), "Profile fields") + marked = { + row[0].strip("`") for row in _table_rows(section) + if row[1].startswith(("yes", "for `type")) + } + self.assertEqual(marked, required) + + +class PythonFloorOnRetroPie(unittest.TestCase): + """The page offered `apt install python3` to a RetroPie 4.8 image, where + python3 is 3.7.3 and apt has nothing newer.""" + + def test_both_pages_name_the_case(self): + install_sh = (ROOT / "install.sh").read_text(encoding="utf-8") + self.assertIn("(3, 8)", install_sh) + first = _section(_page("first-install.md"), "Error: Python 3.8 or newer is required.") + self.assertIn("RetroPie 4.8", first) + self.assertIn("3.7.3", first) + changing = _section(_page("installer.md"), "Changing the installer") + self.assertIn("3.7.3", changing) + + +if __name__ == "__main__": + unittest.main() diff --git a/wiki/adding-a-platform.md b/wiki/adding-a-platform.md index b74b5379..f8966ab1 100644 --- a/wiki/adding-a-platform.md +++ b/wiki/adding-a-platform.md @@ -67,7 +67,21 @@ def validate_format(self, raw_data: str) -> bool: | `destination` | `str` | Relative path within the BIOS directory | | `required` | `bool` | Whether the platform considers this file mandatory | | `zipped_file` | `str \| None` | If set, the hash refers to a ROM inside a ZIP | -| `native_id` | `str \| None` | Original system name before normalization | +| `native_id` | `str \| None` | The platform's own system id, written as `native_system` | +| `sha256` | `str \| None` | SHA256 hash if available | +| `alt_md5` | `str \| None` | A second accepted MD5 (Batocera and ROCKNIX `altmd5`) | +| `native_path` | `str \| None` | The file reference as the platform writes it, kept when it differs from `name`. libretro writes `iplromco.dat` bare although the file lives under `keropi/`, and `ep128emu/roms/cpc464.rom` with its directory | +| `mirror_of` | `str \| None` | The native system whose entry this one repeats under another system. The platform's own file does not carry the copy, so the export skips it | +| `native` | `dict` | Fields the platform declares that the model has no place for, written back unchanged by the export: `native_name`, `core`, `note`, `description`, `hash_match_mandatory`, `mandatory_declared`, `alt_paths`, `component`, `required_label` | + +`requirement_entry()` in `base_scraper.py` is the single serializer from a +requirement to a platform YAML entry, and every scraper goes through it. +Anything an export writes back is recorded by the scraper, never rebuilt by the +exporter: several native systems can fall onto one slug (Recalbox files +`pcengine`, `pcenginecd` and `supergrafx` under `nec-pc-engine`), and a path +or a label the YAML does not hold cannot be derived. When an export needs a +value the YAML lacks, the scraper and `schemas/platform.schema.json` gain it, +then the platform is scraped again. ### System ID mapping @@ -147,16 +161,22 @@ none is edited by hand: RetroPie inherits RetroArch's file set and its scraper writes the packages it builds, read from the `rp_module_id` of each scriptmodule, and any manual change is overwritten on the next scrape. -The `cores` field determines which emulator profiles are resolved for this platform. -Three strategies exist: +The `cores` field determines which emulator profiles are resolved for this +platform (`resolve_platform_cores` in `common.py`). Alias profiles are never +resolved. Three strategies exist: - **Explicit list**: `cores: [beetle_psx, dolphin, ...]` - match by profile key name - or by any name in a profile's `cores:`. Used by Batocera, Recalbox, RetroBat, RomM, - RetroPie, EmuDeck, RetroDECK, ROCKNIX. A list that names `retroarch` (the last four) - also takes every libretro profile. -- **all_libretro**: `cores: all_libretro` - include every profile with `type: libretro` - or `type: standalone + libretro`. Used by RetroArch and Lakka. -- **Omitted**: fallback to system ID intersection. + or by any name in a profile's `cores:`. When a profile's `built_by` names this + platform, that profile answers for the name. Used by Batocera, Recalbox, RetroBat, + RomM, RetroPie, EmuDeck, RetroDECK, ROCKNIX. A list that names `retroarch` (the last + four) also takes every libretro profile, as `all_libretro` does. An empty list + (BizHawk, MiSTer FPGA) resolves no profile, and the pack holds the platform's own + files. +- **all_libretro**: `cores: all_libretro` - every profile whose `type` contains + `libretro`, except those that carry `built_by`: a build a platform compiles itself + is absent from the buildbot and is reached only by its name (Recalbox's `mame0258` + and `mame0278`). Used by RetroArch and Lakka. +- **Omitted**: fallback to system ID intersection, `built_by` profiles excluded. Two lists say how each resolved emulator is laid out. `standalone_cores` names the ones the platform launches as standalone apps: their files go to `standalone_path`. @@ -397,17 +417,13 @@ class Exporter(BaseExporter): ### Round-trip validation -```bash -python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml -python scripts/export_native.py --platform myplatform --fetch --output-dir tmp/native -diff .cache/upstream-native/myplatform/ \ - tmp/native/myplatform/ -``` - -The export must not lose anything the platform already declares. If your +The scraper and the exporter are checked against each other as +[round-trip testing](adding-a-scraper.md#round-trip-testing) describes. The +export must not lose anything the platform already declares. If your format carries code as well as data (a script, a C# database, a manifest whose BIOS list sits beside launch configuration), say so with -`needs_original()` and patch their file rather than writing a new one. An +`needs_original()` and patch the platform's original file rather than writing +a new one. An exporter that regenerates such a file from BIOS data alone hands the maintainer something that no longer runs. diff --git a/wiki/adding-a-scraper.md b/wiki/adding-a-scraper.md index 5f898f45..584b1b8b 100644 --- a/wiki/adding-a-scraper.md +++ b/wiki/adding-a-scraper.md @@ -425,19 +425,19 @@ be missing from the round trip. # Scrape upstream -> platform YAML python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml -# Rewrite the platform's own file, corrected +# Rewrite the platform's own file with the changes applied python scripts/export_native.py --platform myplatform --fetch \ --output-dir tmp/native # Compare with the file upstream publishes, cached by --fetch -diff .cache/upstream-native/myplatform/ \ - tmp/native/myplatform/ +diff .cache/upstream-native/myplatform/ \ + tmp/native/myplatform/ ``` -Every line in that diff should be a correction you can defend. A line that -merely reshapes their formatting means the scraper lost something the -exporter then had to invent: add it to the requirement instead, through -`native_path`, `native_system` or the `native` dict. +Every line of that diff is a change the emulator source supports. A line that +only reshapes the original formatting means the scraper lost something the +exporter then had to invent: record it in the requirement instead, through +the fields listed in [BiosRequirement fields](adding-a-platform.md#biosrequirement-fields). ### Common issues diff --git a/wiki/advanced-usage.md b/wiki/advanced-usage.md index 667c53de..0379e491 100644 --- a/wiki/advanced-usage.md +++ b/wiki/advanced-usage.md @@ -297,8 +297,8 @@ without `--target`. Truth data is ground truth generated from emulator profiles. It represents what each core actually needs based on source code analysis, independent of what platform -scrapers declare. The purpose is gap analysis: finding files that platforms miss -or declare incorrectly. +scrapers declare. The purpose is gap analysis: finding the files the code loads that a platform +list does not name, and the values that differ from the code. ### Generate truth diff --git a/wiki/architecture.md b/wiki/architecture.md index 14289f14..ab066c99 100644 --- a/wiki/architecture.md +++ b/wiki/architecture.md @@ -88,7 +88,7 @@ emulators/*.yml verify.py checks generate_pack.py resolves truth.py generates diff_truth.py export_native.py ground truth from compares truth vs rewrites each platform's - emulator profiles scraped platform own file, corrected + emulator profiles scraped platform own file, updated ``` Pipeline runs all steps in sequence: DB, provenance report, data dirs, @@ -127,14 +127,13 @@ graph LR The pack combines platform baseline (layer 1) with core requirements (layer 3). Neither too much (no files from unused cores) nor too few (no missing files for active cores). -A same-named local file that contradicts an explicit hash is recorded as an -unsafe omission on hash-verifying platforms, while existence platforms ship it and -report the divergence, because their code never reads the bytes. +A same-named local file that contradicts a declared hash is settled by the +platform's verification mode ([discrepancy detection](verification-modes.md#discrepancy-detection)). The emulator's source code serves as ground truth for what files are needed, what names they use, and what validation the emulator performs. Platform YAML -configs are scraped from upstream and are generally accurate, though they can -occasionally have gaps or stale entries. The emulator profiles complement the +configs are scraped from upstream and record what each platform declares. A +list kept by hand can lag behind an emulator release. The emulator profiles complement the platform data by documenting what the code actually loads. When the two disagree, the profile takes precedence for pack generation: files the code needs are included even if the platform does not declare them. Files the platform declares but no @@ -143,7 +142,8 @@ upstream may cover cases not yet profiled. A platform YAML is regenerated by its scraper and never edited by hand: an edit would be lost at the next scrape, and the file would stop describing the -platform. What the platform gets wrong is answered in the other two layers. +platform. Where the emulator source states something else, the profile and +`_shared.yml` carry it. Another name for the same file is an `aliases:` entry in the profile, never a second entry in a platform YAML or in `_shared.yml`. A name a platform's own launcher reads, such as the 22 names Batocera's DuckStation generator tries @@ -373,46 +373,10 @@ platform's own verification mode. ### File resolution chain -`resolve_local_file` in `common.py` tries each strategy in order, returning the -first match. Used by both `verify.py` and `generate_pack.py`. - -```mermaid -graph TD - START([resolve_local_file]) --> S0{path_suffix
exact match?} - S0 -- yes --> EXACT([exact]) - S0 -- no --> S1{SHA1
exact match?} - S1 -- yes --> EXACT - S1 -- no --> S1B{SHA256
exact match?} - S1B -- yes --> EXACT - S1B -- no --> S1C{CRC32 + size,
no stronger hash?} - S1C -- yes --> EXACT - S1C -- no --> S2{MD5 direct
or truncated?} - S2 -- yes --> MD5([md5_exact]) - S2 -- no --> S3{name + aliases
no MD5?} - S3 -- yes --> EXACT - S3 -- no --> S4{name + aliases
md5_composite /
direct MD5?} - S4 -- match --> EXACT - S4 -- name only --> HM([hash_mismatch]) - S4 -- no --> S5{zippedFile
inner ROM MD5?} - S5 -- yes --> ZE([zip_exact]) - S5 -- no --> S6{MAME clone
map lookup?} - S6 -- yes --> MC([mame_clone]) - S6 -- no --> S7{data_dir
cache scan?} - S7 -- yes --> DD([data_dir]) - S7 -- no --> S8{agnostic
fallback?} - S8 -- yes --> AG([agnostic_fallback]) - S8 -- no --> NF([not_found]) - - style START fill:#2d333b,stroke:#adbac7,color:#adbac7 - style EXACT fill:#2d333b,stroke:#adbac7,color:#adbac7 - style MD5 fill:#2d333b,stroke:#adbac7,color:#adbac7 - style HM fill:#2d333b,stroke:#adbac7,color:#adbac7 - style ZE fill:#2d333b,stroke:#adbac7,color:#adbac7 - style MC fill:#2d333b,stroke:#adbac7,color:#adbac7 - style DD fill:#2d333b,stroke:#adbac7,color:#adbac7 - style AG fill:#2d333b,stroke:#adbac7,color:#adbac7 - style NF fill:#2d333b,stroke:#adbac7,color:#adbac7 -``` +`resolve_local_file` in `common.py` turns a file entry into a local path, and +`verify.py` and `generate_pack.py` both call it. Every declared hash is tried +before a path or a name, so a path never wins over a hash. The steps, in order, +are listed under [file resolution chain](verification-modes.md#file-resolution-chain). ## Platform inheritance @@ -420,11 +384,8 @@ Platform configs support `inherits:` to share definitions. Lakka inherits from RetroArch, RetroPie inherits from RetroArch with `base_destination: BIOS`. `overrides:` allows child platforms to modify specific systems from the parent. -Core resolution (`resolve_platform_cores`) uses three strategies: - -- `cores: all_libretro` - include all profiles with `libretro` in their type -- `cores: [list]` - include only named profiles -- `cores:` absent - fallback to system ID intersection between platform and profiles +Which emulator profiles a platform resolves follows its `cores` field, as +[adding a platform](adding-a-platform.md#required-fields) describes. ## Hardware target filtering diff --git a/wiki/first-install.md b/wiki/first-install.md index f3781b7b..71f39e5a 100644 --- a/wiki/first-install.md +++ b/wiki/first-install.md @@ -1,8 +1,7 @@ # First install One path, from an empty BIOS folder to files the emulator can read. Every step -shows what the screen looks like when it worked, so there is never a moment of -wondering. +shows what the screen looks like when it worked. If something on this page does not match what happens, the last section covers the cases that come up. @@ -154,11 +153,17 @@ its BIOS folder, see [Getting started](getting-started.md). ### Error: Python 3.8 or newer is required. -Python is missing from the machine. The message names the command that installs -it, `sudo apt install python3` on Debian and Ubuntu. On a handheld running Batocera or Recalbox it is already there, so +No Python 3.8 or newer was found: Python is missing, or older. The message +names the command that installs it, `sudo apt install python3` on Debian and +Ubuntu. On a handheld running Batocera or Recalbox it is already there, so this message usually means the command ran on the computer rather than on the console. +A stock RetroPie 4.8 image is Debian Buster, where `python3` is 3.7.3 and apt +has nothing newer, so the same command changes nothing. The RetroPie pack from +the [releases page](https://github.com/Abdess/retrobios/releases/latest), +extracted into `~/RetroPie/BIOS/`, installs the same files. + ### Error: the downloaded installer does not match its expected fingerprint. Nothing was run and nothing was written. The download did not match the diff --git a/wiki/installer.md b/wiki/installer.md index d736822d..ec1a3c49 100644 --- a/wiki/installer.md +++ b/wiki/installer.md @@ -103,14 +103,15 @@ macOS. rest. 7. Copies into standalone emulator directories, only with `--standalone-copies`. -A run looks like this: +A RetroArch run into an empty folder printed this on 2026-10-21: ``` Fetching file index for retroarch... - 5865 files (5.7 GB) + 8069 files (6.5 GB) + Not installable: 11 files (1 required, not found: 11). Checking existing files... - 0/5865 present (0 verified, 0 wrong hash) + 0/8069 present (0 verified, 0 wrong hash) ``` When the manifest leaves entries out (a file the collection cannot supply, or @@ -126,9 +127,9 @@ By default the installer fetches everything the platform runs. At a terminal, after detection, it asks: ``` -RetroArch: 5865 files, 5.7 GB. +RetroArch: 8069 files, 6.5 GB. Enter) install everything - i) only what the installed cores read: 102 files, 44.5 MB + i) only what the installed cores read: 100 files, 44.2 MB c) choose systems, cores and regions q) quit ``` @@ -259,7 +260,7 @@ Files the list does not name are never read, moved or deleted. | Variable | Effect | |----------|--------| -| `RETROBIOS_REF` | Branch or tag the file list and the payloads are read from. Defaults to `main`, and a release tag pins a reproducible file set | +| `RETROBIOS_REF` | Branch or tag the file list and the files committed to the repository are read from. Defaults to `main`. Files the manifest marks `release_asset` always come from the `large-files` release, whatever the ref, and that release replaces an asset in place when it is rebuilt. Each release deletes the previous release tags ([release process](release-process.md#cutting-a-release)), so a tag holds a file list until the next release | | `RETROBIOS_BASE_URL` | Base URL for the manifest and the files it declares. HTTPS only, plain HTTP allowed to loopback for end-to-end tests. Whoever controls this base controls the expected hashes, hence the check. A file list that cannot be fetched names the address, and this variable when it is set | | `RETROBIOS_OS` | Names the host outright (`linux`, `wsl`, `windows`, `darwin`, `android`) instead of detecting it | | `EXTERNAL_STORAGE` | Android only, the shared storage root RetroArch builds its default directories from. Defaults to `/storage/emulated/0` | @@ -483,8 +484,12 @@ followed. ## Changing the installer -`install.py` runs on the user's machine, often an older handheld or a stock -RetroPie image, so it stays on Python 3.8 and the standard library alone. +`install.py` runs on the user's machine, often an older handheld, so it stays +on Python 3.8 and the standard library alone. The RetroPie 4.8 images +(`retropie-buster-4.8-*.img.gz`) are Debian Buster, whose `python3` is 3.7.3: +there the bootstrap stops before running anything, and the RetroPie pack from +the releases page is the way to install +([first install](first-install.md#error-python-38-or-newer-is-required)). Tests refuse any 3.9 syntax or API, and one runs the installer end to end under a real 3.8 interpreter, which CI installs beside 3.12 so the check cannot skip. The build scripts need 3.12 and are not bound by this. @@ -503,9 +508,9 @@ fields, the installer refuses to narrow rather than install everything. |------|---------| | `0` | everything asked for is installed or already up to date | | `1` | a failure: the one-liner could not download or verify `install.py`, or found no Python 3.8 (nothing was run), an unusable base URL, an unknown platform or target, `--installed-cores` with no core found, a manifest that fails validation, a fetch that never succeeded, no platform to install for in a non-interactive run, some files failed while others installed, or `--check` found files to fetch | -| `2` | nothing was installed: every file the run tried failed (a BIOS folder that cannot be written, for one) | +| `2` | nothing was installed: every file the run tried failed (a BIOS folder that cannot be written, for one). Also an option the installer refused before any download: an unknown option, `--jobs` outside 1 to 32, `--target` with `--every-board`. The message then starts with `install.py: error:` | | `130` | stopped with Ctrl+C, or with Ctrl+C or Ctrl+D at a question. Run the same command again to carry on | -| `141` | the output was piped to a reader that closed early (`--list \| head`) | +| `141` | the output was piped to a reader that closed early (`--list-platforms \| head`) | `install.ps1` passes the installer's code on unchanged: run as a file it exits with it, run through `irm \| iex` it sets `$LASTEXITCODE` (an `exit` there diff --git a/wiki/profiling.md b/wiki/profiling.md index d920f642..262ec16e 100644 --- a/wiki/profiling.md +++ b/wiki/profiling.md @@ -55,17 +55,15 @@ a system_dir lookup) are expected. What matters are file additions the port introduces, files the port dropped, or hash values that differ between the two codebases. -If the source is hosted outside GitHub, it's worth exploring further. Emulator -source on GitLab, Codeberg, SourceForge, Bitbucket, archive.org -snapshots, and community mirror tarballs. Inspecting copyright headers -or license strings in the libretro fork often points to the original -author's site. The upstream code exists somewhere, so it's worth continuing the search before concluding the source is unavailable. +Source hosted outside GitHub is searched on GitLab, Codeberg, SourceForge, +Bitbucket, archive.org snapshots and community mirror tarballs. Copyright +headers and license strings in the libretro fork often name the original +author's site. A source not found yet is a search not finished, and the +profile waits for it. -One thing worth noting: even when the same repository was analyzed for -a related profile (e.g., fbneo for arcade systems), it helps to do a -fresh pass for each new profile. When fbneo_neogeo was profiled, the -NeoGeo subset referenced BIOS files that the main arcade analysis -hadn't encountered. A fresh look avoids carrying over blind spots. +Each profile gets its own reading of the code, even when the same repository +was read for a related profile (`fbneo`, then `fbneo_neogeo`). The earlier +reading says what that profile loads, not what the new one loads. A declared upstream that answers 404 is not proof the project is gone. The account can differ from the Android package id (`aenu.aps3e` lives at @@ -470,19 +468,23 @@ even if documentation mentions it. ### Profile fields -The authoritative version of this table is `schemas/emulator.schema.json`, -which CI validates every profile against. +`schemas/emulator.schema.json` is the contract CI validates every profile +against. It requires `emulator` and `type`, then `files`, or `alias_of` for an +alias. A profile also carries `source_commit` or `profiled_date`: without +either, profile_sync has no revision to read the `source_ref` lines at and +reports the profile as uncheckable, and the CI job that runs +`profile_sync.py --strict` on each profile a pull request adds fails on it. | Field | Required | Description | |-------|----------|-------------| | `emulator` | yes | display name | | `type` | yes | `libretro`, `standalone`, `standalone + libretro`, `alias`, `launcher`, `game`, `utility`, `test` | | `core_classification` | no | `pure_libretro`, `official_port`, `community_fork`, `frozen_snapshot`, `enhanced_fork`, `game_engine`, `embedded_hle`, `launcher`, `alias`, `other` | -| `source` | yes | libretro core repository URL. A dict when the core ships under several repos | +| `source` | no | libretro core repository URL. A dict when the core ships under several repos | | `upstream` | no | original emulator repository URL | -| `profiled_date` | yes | date of source analysis. Quote it, or YAML parses it as a date object | -| `core_version` | yes | version analyzed | -| `source_commit` | no | commit SHA of `source` the code was read at. Anchors every `source_ref` line number | +| `profiled_date` | see above | date of source analysis. Quote it, or YAML parses it as a date object | +| `core_version` | no | version analyzed | +| `source_commit` | see above | commit SHA of `source` the code was read at. Anchors every `source_ref` line number | | `upstream_commit` | no | same, for `upstream` | | `source_branch` | no | branch `source` was read from when it is not the default one (a port living on a pull request branch). profile_sync follows it | | `built_by` | no | platforms that ship this build under its core name, compiled by them or installed from the project's own release (Recalbox's `mame0278`, the GSplus legacy build). On those platforms the name resolves to this profile, never through `all_libretro`: the buildbot does not publish it | @@ -490,7 +492,7 @@ which CI validates every profile against. | `bios_directory` | no | the folder an agnostic core scans, relative to the system directory (`pcsx2/bios/`). A scan never seeds from files outside it | | `display_name` | no | full display name (e.g. "Sega - Mega Drive (BlastEm)") | | `logo` | no | image URL used on the emulator page | -| `systems` | yes | list of system IDs this core handles | +| `systems` | no | list of system IDs this core handles | | `cores` | no | every upstream name the core is known by, for buildbot/target matching | | `alias_of` | for `type: alias` | profile key this one aliases. The alias carries no `files` | | `mode` | no | default mode: `standalone`, `libretro`, or `both` | diff --git a/wiki/release-process.md b/wiki/release-process.md index fe95fd93..5151da20 100644 --- a/wiki/release-process.md +++ b/wiki/release-process.md @@ -1,11 +1,11 @@ # Release process -This page documents the CI/CD pipeline: what each workflow does, how releases -are built, and how to run the process manually. +Two GitHub workflows check pull requests and publish the site. Releases are +built, checked and uploaded from the maintainer's machine. ## CI workflows overview -The project uses 2 GitHub Actions workflows. All use only official GitHub +Both workflows use only official GitHub actions (`actions/checkout`, `actions/setup-python`, `actions/upload-pages-artifact`, `actions/deploy-pages`). No third-party actions. @@ -239,8 +239,12 @@ python scripts/check_release.py --tag "v${DATE}" --sums dist/SHA256SUMS.txt \ && gh release edit "v${DATE}" --draft=false --latest # The record and the pages it feeds go to main with the release, so the -# table never describes a pack other than the one the link serves. -git add release.json README.md mkdocs.yml +# table never describes a pack other than the one the link serves. The +# install manifests go too: step 1 regenerated them, and their pack_files +# and pack_size describe the packs just built. git status names anything +# else the run rewrote. +git status --short +git add release.json README.md mkdocs.yml install/*.json install/targets/*.json git commit -m "chore: record release v${DATE}" && git push # 6. Keep only the new release plus large-files: an older pack carries hashes diff --git a/wiki/testing-guide.md b/wiki/testing-guide.md index 10078c1a..495b35db 100644 --- a/wiki/testing-guide.md +++ b/wiki/testing-guide.md @@ -1,7 +1,5 @@ # Testing guide -This page covers how to run, understand, and extend the test suite. - The suite is discovered dynamically across focused modules, so its exact count is reported by the test runner instead of being hand-maintained here. Tests do not require network access. Most build synthetic fixtures under the repository's @@ -46,7 +44,7 @@ Every module in `tests/`, by the area it guards. | Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_selection` | | Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_diff`, `test_truth_layout` | | Scrapers, upstream and provenance | inline sources, synthetic | `test_check_freshness`, `test_dedup`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_profile_sync`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_scraper_contract`, `test_upstream`, `test_upstream_index` | -| Site, docs and repository contracts | rendered pages, sources | `test_contributing_text`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_doc_surfaces`, `test_optional_dependencies`, `test_prose_style`, `test_site_availability`, `test_site_cores`, `test_site_exports`, `test_site_platform_hashes`, `test_site_system_links`, `test_site_validation`, `test_site_write`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` | +| Site, docs and repository contracts | rendered pages, sources | `test_contributing_text`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_doc_pages`, `test_doc_surfaces`, `test_optional_dependencies`, `test_prose_style`, `test_site_availability`, `test_site_cores`, `test_site_exports`, `test_site_platform_hashes`, `test_site_system_links`, `test_site_validation`, `test_site_write`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` | ## Test architecture @@ -193,17 +191,19 @@ script could have established, the script gains that check in the same change. Example skeleton: ```python -def test_42_my_new_behavior(self): - # Write minimal fixtures to self.root - profile = {"emulator": "test_core", "files": [...]} +def test_my_new_behavior(self): + # Minimal fixtures under self.root + profile = {"emulator": "test_core", "type": "libretro", "files": [...]} with open(os.path.join(self.emulators_dir, "test_core.yml"), "w") as f: yaml.dump(profile, f) - # Call production code - result = verify_platform(self.config, self.db, ...) + # The entry point the CLI uses + config = load_platform_config("test_md5", self.platforms_dir) + result = verify_platform(config, self.db, self.emulators_dir) - # Assert specific outcomes - self.assertEqual(result[0]["status"], Status.OK) + # One outcome, read from the per-file details + by_name = {d["name"]: d for d in result["details"]} + self.assertEqual(by_name["my_file.bin"]["status"], Status.OK) ``` ### test_pack_integrity.py @@ -242,9 +242,13 @@ The test suite is one layer of verification. The full quality gate is: 5. Pack integrity passes (every declared file extractable with correct hash) 6. `mkdocs build --strict` and `python scripts/validate_site.py` pass -If a change passes tests but breaks the pipeline, it's worth investigating before merging. Similarly, new CRITICAL entries in the verify output after a change usually indicate something to look into. The pipeline is designed so that all steps agree: if verify reports N files for a platform, the pack should contain exactly N files. +A change that passes the tests but breaks the pipeline is not merged. A new +CRITICAL entry in the verify output after a change is explained before the +change lands. The steps agree with each other: when verify reports N files for +a platform, the pack holds exactly N files. -Ideally, tests, code, and documentation ship together. When profiles and platform configs are involved, updating them in the same change helps keep everything in sync. +Tests, code, profiles, platform configs and documentation ship in the same +change. A run that dies with a segmentation fault or an exception the code cannot raise is run again once, alone, before anything is investigated. Only a diff --git a/wiki/tools.md b/wiki/tools.md index 5b6bc627..e3698000 100644 --- a/wiki/tools.md +++ b/wiki/tools.md @@ -129,7 +129,7 @@ Verification modes per platform: |----------|------|-------| | RetroArch, Lakka, RetroPie | existence | file present = OK | | Batocera, RetroBat | md5 | MD5 hash match, plus `checkInsideZip` for `zippedFile` entries | -| Recalbox | md5 | MD5 multi-hash, `md5_composite` for ZIPs, 3 severity levels | +| Recalbox | md5 | MD5 multi-hash, `md5_composite` for ZIPs | | EmuDeck | md5 | MD5 whitelist per system | | RetroDECK | md5 | MD5 per file via component manifests | | RomM | md5 | size + any hash (MD5/SHA1/CRC32), no ZIP inspection | @@ -203,12 +203,8 @@ Packs include platform baseline files plus files required by the platform's core When a file passes platform verification but fails emulator validation, the tool searches for a variant that satisfies both. If none exists, the platform version is kept and the discrepancy is reported. -What happens when a declared hash and the local dump disagree follows the -platform's own verification mode. A hash platform would reject the file, so it -is omitted and recorded in the pack report and the installer manifest. An -existence platform reads no bytes, so the file is packed and the divergence is -reported instead: an upstream list error must not remove a file the frontend -would have loaded. +A declared hash that the local dump contradicts is settled by the platform's +own verification mode ([discrepancy detection](verification-modes.md#discrepancy-detection)). ### Granular options @@ -292,7 +288,7 @@ python scripts/export_native.py --all --refresh-cache A list is written in the order the code looks: `priority:` first (lowest wins), then the profile's own declaration order. What a format cannot state is reported rather than written: EmuDeck's arrays are shared -between emulators its file does not name, so a hash is corrected in place +between emulators its file does not name, so a hash is updated in place and never added. `exports/.json` lists every change under `changes`, each with diff --git a/wiki/troubleshooting.md b/wiki/troubleshooting.md index 6438a92f..4b2ced03 100644 --- a/wiki/troubleshooting.md +++ b/wiki/troubleshooting.md @@ -1,7 +1,5 @@ # Troubleshooting -Diagnosis guide organized by symptom. Each section describes what to check and how to fix it. - ## Game won't start / black screen Most launch failures are caused by a missing or incorrect BIOS file. @@ -308,9 +306,9 @@ The one-line bootstrap verifies the downloaded installer before running it, and the installer reads its manifests from the same revision. A platform name is checked before anything is downloaded, so a "No file list" message comes from the revision: `RETROBIOS_REF` naming a tag that does not exist, or one older -than the platform. Unset it, or pick a tag from the releases page. -`RETROBIOS_REF` pins an installation to a published tag when a reproducible -file set matters more than the newest one. +than the platform. Unset it, or pick a tag from the releases page. What a tag +holds and for how long is under +[environment overrides](installer.md#environment-overrides). ### Manual pack not found in release diff --git a/wiki/verification-modes.md b/wiki/verification-modes.md index 0900d44a..5658005e 100644 --- a/wiki/verification-modes.md +++ b/wiki/verification-modes.md @@ -57,10 +57,11 @@ the actual hash is compared against only its first N characters. ### md5_composite (Recalbox ZIP verification) -Recalbox computes `Zip::Md5Composite` for ZIP files: the MD5 of the concatenation of all -inner file MD5s (sorted by filename). `verify.py` replicates this with `md5_composite()` -from `common.py`. When a ZIP file's direct MD5 does not match, the composite is tried -before reporting a mismatch. +Recalbox hashes a ZIP with `Zip::Md5Composite` (`es-core/src/utils/Zip.cpp:71-96` in +recalbox-emulationstation at `f4cd0ae4`): one MD5 over the contents of every member, +read in filename order. The members' own MD5s are not involved. `verify.py` computes +the same value with `md5_composite()` (`scripts/hashing.py`). When a ZIP file's direct +MD5 does not match, the composite is tried before reporting a mismatch. ### Multi-hash (Recalbox) @@ -70,14 +71,24 @@ listed hash is OK. ### Mandatory levels (Recalbox) -Recalbox uses three severity levels derived from two YAML fields (`mandatory` and -`hashMatchMandatory`): +Recalbox colours each entry of `es_bios.xml` from two attributes, `mandatory` and +`hashMatchMandatory`, both true unless written `"false"` +(`es-app/src/bios/Bios.cpp:162-163` in recalbox-emulationstation at `f4cd0ae4`, +colours set by `Bios::Scan` at lines 84-108). The platform YAML carries them as +`required` and, when the attribute is written, `hash_match_mandatory`. -| mandatory | hashMatchMandatory | Color | verify.py mapping | -|-----------|--------------------|--------|-------------------| -| true | true | RED | CRITICAL | -| true | false | YELLOW | WARNING | -| false | (any) | GREEN | WARNING | +| File | mandatory | hashMatchMandatory | Recalbox | verify.py | +|------|-----------|--------------------|----------|-----------| +| hash matches | (any) | (any) | GREEN | OK | +| missing | true | (any) | RED | CRITICAL | +| missing | false | (any) | YELLOW | WARNING | +| hash differs | true | true | RED | WARNING | +| hash differs | true | false | YELLOW | WARNING | +| hash differs | false | (any) | YELLOW | WARNING | + +`verify.py` applies the MD5-mode table above and does not read +`hash_match_mandatory`, so a present file whose hash differs is a WARNING where +Recalbox shows RED for a mandatory entry with `hashMatchMandatory`. ### checkInsideZip (Batocera zippedFile) @@ -154,8 +165,15 @@ from `validation.py`. This layer uses data from emulator profiles (YAML files in ### Validation index -`_build_validation_index()` reads all emulator profiles and builds a per-filename -index of validation rules. When multiple emulators reference the same file, checks +`_build_validation_index()` (`validation.py`) builds a per-filename index of +validation rules from the profiles it is given. A platform report, its pack and +its install manifest pass only the profiles `resolve_platform_cores` resolves for +that platform (`verify.py`, `verify_platform`). A profile that does not run on the +platform has nothing to say about its pack: standalone ZEsarUX reads a 48 KB +`cpc6128.rom` of its own, and its size check reported cap32's 32 KB file in the +RetroArch pack as a mismatch while every profile was indexed. The same holds for +`hle_fallback`, which is read from the same set of profiles. A report or pack for `--emulator` or `--system` indexes the +profiles it selected. When multiple emulators reference the same file, checks are merged (union of all check types). Conflicting expected values are kept as sets (e.g., multiple accepted CRC32 values for different ROM versions). @@ -256,12 +274,8 @@ A path or filename is never allowed to mask a declared strong hash: the identity steps run first, and a path or name is accepted only when no content hash was declared or when the content agrees with the one that was. -What the pack does with a `hash_mismatch` follows the platform's own mode. An -MD5 or SHA1 platform would reject those bytes, so the file is left out and -counted as an unsafe exclusion, distinct from a file the collection does not -have. An existence platform reads no bytes at all, so the file is packed and -the divergence is printed as a discrepancy: a value in an upstream BIOS list that -differs from the emulator source must not remove a file the frontend would load. +What the pack does with a `hash_mismatch` is set by the platform's mode, as +[discrepancy detection](#discrepancy-detection) describes. ## Discrepancy detection @@ -272,10 +286,11 @@ accepted. A local file that contradicts a declared hash is handled by the platform's own mode, never by one global rule. An existence frontend never reads the -bytes, so the file is packed and the disagreement reported: a stale hash in -System.dat must not remove a file RetroArch would have loaded. An md5 or sha1 -frontend would reject it, so it is left out of the pack and counted as -`unsafe excluded`, which is distinct from `missing`. The pack builder and the +bytes, so the file is packed and the difference reported: a hash in System.dat +that differs from the local dump must not remove a file RetroArch would have +loaded. An md5 or sha1 frontend would reject it, so it is left out of the pack +and counted as `unsafe excluded`, which is distinct from `missing`. The pack +report and the installer manifest both record it. The pack builder and the pack verification apply the same rule. ### Variant search