mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-10-23 12:19:13 -05:00
chore: merge round 2 fixes (docs)
# Conflicts: # CONTRIBUTING.md # scripts/check_release_assets.py # scripts/generate_readme.py
This commit is contained in:
Abdessamad Derraz committed
commit
6409757c39
22 files changed
+654
-115
No files matched your search
@@ -0,0 +1,5 @@
|
||||
blank_issues_enabled: true
|
||||
contact_links:
|
||||
- name: Add a BIOS file
|
||||
url: https://github.com/Abdess/retrobios/upload/main/bios
|
||||
about: Upload the file into bios/ from the browser. GitHub opens the pull request, and CI checks the file and names the folder of its system.
|
||||
+21
-8
@@ -2,12 +2,21 @@
|
||||
|
||||
## Add a BIOS file
|
||||
|
||||
1. Fork this repository
|
||||
2. Place the file in `bios/Manufacturer/Console/filename`.
|
||||
`python scripts/validate_pr.py path/to/file` names the folder its system
|
||||
is kept in, and whether a platform or an emulator profile declares it
|
||||
3. Variants (alternate hashes for the same file): `bios/Manufacturer/Console/.variants/`
|
||||
4. Open a pull request. Its hashes are checked automatically and reported in a comment
|
||||
One file needs no clone and no local setup:
|
||||
|
||||
1. [Upload it into `bios/`](https://github.com/Abdess/retrobios/upload/main/bios)
|
||||
from the browser, 25 MB at most per file. GitHub writes it to a fork and
|
||||
opens the pull request.
|
||||
2. CI hashes it and looks it up in every platform list and emulator profile.
|
||||
For a file it recognises, it names the folder holding the other files of its
|
||||
system. The report is in the summary of the pull request's Validation run.
|
||||
3. The file can be moved to that folder in the same pull request. A file whose
|
||||
hash differs from the one already held under its name goes to that folder's
|
||||
`.variants/`.
|
||||
|
||||
From a clone, the file goes to `bios/Manufacturer/Console/filename`, and
|
||||
`python scripts/validate_pr.py path/to/file` names that folder for a file still
|
||||
outside `bios/`. A file over the browser's 25 MB limit goes in that way.
|
||||
|
||||
The [dump provenance](https://abdess.github.io/retrobios/provenance/) page lists catalogued dumps still
|
||||
missing from the collection, with their hashes. A file matching one of those is
|
||||
@@ -45,7 +54,9 @@ Full walkthrough: [profiling guide](https://abdess.github.io/retrobios/wiki/prof
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
The checks CI runs on a pull request (Python 3.12 or newer):
|
||||
A pull request that only adds files under `bios/` needs none of this: the
|
||||
validation above is its check. For code, profiles or platform files, these are
|
||||
the checks CI runs on a pull request (Python 3.12 or newer):
|
||||
|
||||
```bash
|
||||
pip install pyyaml "jsonschema[format-nongpl]" markdown \
|
||||
@@ -64,7 +75,9 @@ in `dist/`, which is why maintainers run it before a release.
|
||||
|
||||
CI computes SHA1/MD5/CRC32 for every new file, checks them against the platform
|
||||
configs, validates the YAML against the schemas, confronts each added profile
|
||||
with its upstream source, runs the test suite, and posts a report on the PR.
|
||||
with its upstream source and runs the test suite. The BIOS report goes to the
|
||||
summary of the Validation run, and also to a comment when the branch lives in
|
||||
this repository: a fork's pull request runs with a read-only token.
|
||||
|
||||
Contributors who add platform support are credited in the README,
|
||||
on the documentation site, and in the BIOS packs.
|
||||
@@ -27,9 +27,13 @@ curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh |
|
||||
# SD card mounted on a PC: --dest is the BIOS folder on the card
|
||||
# (RetroArch on Switch: retroarch/cores/system, on Android: RetroArch/system)
|
||||
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform retroarch --dest /path/to/sdcard/retroarch/cores/system
|
||||
|
||||
# Options in PowerShell: anything after | iex goes to iex, so the script runs as a block
|
||||
&([scriptblock]::Create((irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1))) --check
|
||||
```
|
||||
|
||||
The script auto-detects your platform, downloads only missing files, and verifies checksums.
|
||||
Every option, and what the installer detects and writes, is on the [installer page](https://abdess.github.io/retrobios/wiki/installer/#passing-options).
|
||||
|
||||
## Download BIOS packs
|
||||
|
||||
@@ -65,7 +69,7 @@ The RetroDECK pack already carries its own `bios/` folder, so it extracts into `
|
||||
BIOS, firmware, and system files for consoles from Atari to PlayStation 3.
|
||||
These are the files an emulator loads from disk instead of carrying inside itself. Some are required to boot a system, others improve accuracy or unlock a feature, and the packs carry both.
|
||||
|
||||
Each file is checked the way your platform checks it. Most compare a checksum, the fingerprint of a file's contents, which catches a corrupt or unexpected copy. RetroArch, Lakka and RetroPie only look for the filename, because that is all their code does: the Coverage table says which applies to you. Independently of that, the collection records five fingerprints per file, and wherever an emulator's code states an expected size or hash, that value is read from its source and rechecked here.
|
||||
Each file is checked the way your platform checks it. Most compare a checksum, the fingerprint of a file's contents, which catches a corrupt or unexpected copy. RetroArch, Lakka and RetroPie check that the file is present under its name without reading its contents (`path_is_valid` in RetroArch's `core_info.c`): the Coverage table says which applies to you. Independently of that, the collection records five fingerprints per file, and wherever an emulator's code states an expected size or hash, that value is read from its source and rechecked here.
|
||||
|
||||
- **936 files** the platforms' emulators load are still to be found, and 136 more cannot be sourced at all (per-user keys, user-filled slots, dumps nobody has made). Both are named in the [gap analysis](https://abdess.github.io/retrobios/gaps/)
|
||||
- **12 platforms** supported with platform-specific verification
|
||||
@@ -73,7 +77,7 @@ Each file is checked the way your platform checks it. Most compare a checksum, t
|
||||
- **533 systems** handled by those emulators (NES, SNES, PlayStation, Saturn, Dreamcast, ...)
|
||||
- **19,526 files in the collection**, each with its SHA1, MD5, SHA256, CRC32 and Adler-32 fingerprints: 5,743 system files, 2,851 arcade ROM sets, 10,932 game and engine data files. That is every platform and emulator together, so no pack holds them all: each carries what its own emulators load, counted in the download table
|
||||
- **555 of 5,743 system files** matched to dump-preservation catalogs (No-Intro, Redump, TOSEC). Arcade sets and engine data fall outside what those catalogs index
|
||||
- **16011 MB** total collection size
|
||||
- **15.6 GB** for the whole collection
|
||||
|
||||
## Supported systems
|
||||
|
||||
@@ -85,16 +89,16 @@ Full list with per-file details: **[https://abdess.github.io/retrobios/](https:/
|
||||
|
||||
| Platform | On its BIOS list | Files its emulators load | Checked by |
|
||||
|----------|-----------------:|-------------------------:|------------|
|
||||
| Batocera | 353/353 | 13,921/15,009 | MD5 hash |
|
||||
| Batocera | 353/353 | 13,918/15,006 | MD5 hash |
|
||||
| BizHawk | 118/118 | - | SHA1 hash |
|
||||
| EmuDeck | 168/168 | 8,037/8,111 | MD5 hash |
|
||||
| Lakka | 532/532 | 7,473/7,531 | file presence |
|
||||
| EmuDeck | 168/168 | 8,034/8,108 | MD5 hash |
|
||||
| Lakka | 532/532 | 7,471/7,529 | file presence |
|
||||
| MiSTer FPGA | 81/81 | - | MD5 hash |
|
||||
| ROCKNIX | 38/38 | 6,963/8,232 | MD5 hash |
|
||||
| Recalbox | 351/351 | 2,748/2,911 | MD5 hash |
|
||||
| RetroArch | 526/526 | 7,481/7,522 | file presence |
|
||||
| RetroBat | 348/348 | 4,732/9,196 | MD5 hash |
|
||||
| RetroDECK | 2,007/2,008 | 6,240/6,490 | MD5 hash |
|
||||
| ROCKNIX | 38/38 | 6,959/8,228 | MD5 hash |
|
||||
| Recalbox | 351/351 | 2,746/2,909 | MD5 hash |
|
||||
| RetroArch | 526/526 | 7,479/7,520 | file presence |
|
||||
| RetroBat | 348/348 | 4,735/9,199 | MD5 hash |
|
||||
| RetroDECK | 2,007/2,008 | 6,237/6,487 | MD5 hash |
|
||||
| RetroPie * | 532/532 | 7,748/12,264 | file presence |
|
||||
| RomM | 383/383 | 83/843 | MD5 hash |
|
||||
|
||||
@@ -146,8 +150,7 @@ The [documentation site](https://abdess.github.io/retrobios/) provides:
|
||||
|
||||
## How it works
|
||||
|
||||
Documentation and metadata can drift from what emulators actually load.
|
||||
To keep packs accurate, platform lists are checked against emulator source code, file by file where a profile exists. When the two disagree, the code wins.
|
||||
A platform's BIOS list and an emulator's code are maintained separately. Each list is compared with the emulator source code, file by file where a profile exists, and where the two differ the code decides.
|
||||
|
||||
Hashes document what emulator code loads and accepts, not dump provenance. That boundary, and how it relates to preservation catalogs such as No-Intro, is drawn in the [FAQ](https://abdess.github.io/retrobios/wiki/faq/#are-these-files-verified-against-original-hardware-dumps).
|
||||
|
||||
@@ -180,4 +183,4 @@ The scripts and tooling are released under the [MIT License](LICENSE).
|
||||
The BIOS and firmware files are not covered by that license: they are third-party system software, preserved and provided for personal backup, archival, and interoperability with emulation software. [NOTICE](NOTICE) sets out their status and how to ask for a file to be removed.
|
||||
The reasoning, and where it is weakest, is in the [FAQ](https://abdess.github.io/retrobios/wiki/faq/#is-this-legal).
|
||||
|
||||
*Auto-generated on 2026-10-21T05:58:35Z*
|
||||
*Auto-generated on 2026-10-21T08:29:32Z*
|
||||
+38
-18
@@ -250,6 +250,9 @@ def format_size(size: int) -> str:
|
||||
SITE_URL = "https://abdess.github.io/retrobios/"
|
||||
REPO_PATH = "../.."
|
||||
REPO = "Abdess/retrobios"
|
||||
# GitHub's "Add file > Upload files" page for bios/: a browser upload by
|
||||
# someone without write access lands on a fork and opens a pull request.
|
||||
UPLOAD_URL = f"https://github.com/{REPO}/upload/main/bios"
|
||||
|
||||
|
||||
def _existing_contributor_block(readme_path: str = "README.md") -> list[str]:
|
||||
@@ -329,7 +332,6 @@ def _catalog_matched_line(db: dict) -> list[str]:
|
||||
def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
total_files = db.get("total_files", 0)
|
||||
total_size = db.get("total_size", 0)
|
||||
size_mb = total_size / (1024 * 1024)
|
||||
ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
platform_names = list_registered_platforms(platforms_dir, include_archived=True)
|
||||
@@ -397,9 +399,13 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
"# SD card mounted on a PC: --dest is the BIOS folder on the card",
|
||||
"# (RetroArch on Switch: retroarch/cores/system, on Android: RetroArch/system)",
|
||||
"curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform retroarch --dest /path/to/sdcard/retroarch/cores/system",
|
||||
"",
|
||||
"# Options in PowerShell: anything after | iex goes to iex, so the script runs as a block",
|
||||
"&([scriptblock]::Create((irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1))) --check",
|
||||
"```",
|
||||
"",
|
||||
"The script auto-detects your platform, downloads only missing files, and verifies checksums.",
|
||||
f"Every option, and what the installer detects and writes, is on the [installer page]({SITE_URL}wiki/installer/#passing-options).",
|
||||
"",
|
||||
"## Download BIOS packs",
|
||||
"",
|
||||
@@ -462,16 +468,17 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
"",
|
||||
"Each file is checked the way your platform checks it. Most compare"
|
||||
" a checksum, the fingerprint of a file's contents, which catches a"
|
||||
" corrupt or unexpected copy. RetroArch, Lakka and RetroPie only"
|
||||
" look for the filename, because that is all their code does: the"
|
||||
" corrupt or unexpected copy. RetroArch, Lakka and RetroPie check"
|
||||
" that the file is present under its name without reading its"
|
||||
" contents (`path_is_valid` in RetroArch's `core_info.c`): the"
|
||||
" Coverage table says which applies to you. Independently of that,"
|
||||
" the collection records five fingerprints per file, and wherever"
|
||||
" an emulator's code states an expected size or hash, that value is"
|
||||
" read from its source and rechecked here.",
|
||||
"",
|
||||
(
|
||||
f"- **{acquirable_total} files** the platforms' emulators load"
|
||||
f" are still to be found, and {unsourceable_total} more cannot"
|
||||
f"- **{acquirable_total:,} files** the platforms' emulators load"
|
||||
f" are still to be found, and {unsourceable_total:,} more cannot"
|
||||
f" be sourced at all (per-user keys, user-filled slots, dumps"
|
||||
f" nobody has made). Both are named in the"
|
||||
f" [gap analysis]({SITE_URL}gaps/)"
|
||||
@@ -484,7 +491,7 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
f"- **{len(system_ids)} systems** handled by those emulators (NES, SNES, PlayStation, Saturn, Dreamcast, ...)",
|
||||
collection_line(total_files, comp),
|
||||
*_catalog_matched_line(db),
|
||||
f"- **{size_mb:.0f} MB** total collection size",
|
||||
f"- **{format_size(total_size)}** for the whole collection",
|
||||
"",
|
||||
"## Supported systems",
|
||||
"",
|
||||
@@ -621,10 +628,10 @@ def generate_readme(db: dict, platforms_dir: str) -> str:
|
||||
"",
|
||||
"## How it works",
|
||||
"",
|
||||
"Documentation and metadata can drift from what emulators actually load.",
|
||||
"To keep packs accurate, platform lists are checked against emulator"
|
||||
" source code, file by file where a profile exists. When the two"
|
||||
" disagree, the code wins.",
|
||||
"A platform's BIOS list and an emulator's code are maintained"
|
||||
" separately. Each list is compared with the emulator source code,"
|
||||
" file by file where a profile exists, and where the two differ the"
|
||||
" code decides.",
|
||||
"",
|
||||
"Hashes document what emulator code loads and accepts, not dump"
|
||||
" provenance. That boundary, and how it relates to preservation"
|
||||
@@ -710,12 +717,21 @@ def contributing_text(
|
||||
|
||||
## Add a BIOS file
|
||||
|
||||
1. Fork this repository
|
||||
2. Place the file in `bios/Manufacturer/Console/filename`.
|
||||
`python scripts/validate_pr.py path/to/file` names the folder its system
|
||||
is kept in, and whether a platform or an emulator profile declares it
|
||||
3. Variants (alternate hashes for the same file): `bios/Manufacturer/Console/.variants/`
|
||||
4. Open a pull request. Its hashes are checked automatically and reported in a comment
|
||||
One file needs no clone and no local setup:
|
||||
|
||||
1. [Upload it into `bios/`]({UPLOAD_URL})
|
||||
from the browser, 25 MB at most per file. GitHub writes it to a fork and
|
||||
opens the pull request.
|
||||
2. CI hashes it and looks it up in every platform list and emulator profile.
|
||||
For a file it recognises, it names the folder holding the other files of its
|
||||
system. The report is in the summary of the pull request's Validation run.
|
||||
3. The file can be moved to that folder in the same pull request. A file whose
|
||||
hash differs from the one already held under its name goes to that folder's
|
||||
`.variants/`.
|
||||
|
||||
From a clone, the file goes to `bios/Manufacturer/Console/filename`, and
|
||||
`python scripts/validate_pr.py path/to/file` names that folder for a file still
|
||||
outside `bios/`. A file over the browser's 25 MB limit goes in that way.
|
||||
|
||||
The [dump provenance]({link("provenance")}) page lists catalogued dumps still
|
||||
missing from the collection, with their hashes. A file matching one of those is
|
||||
@@ -753,7 +769,9 @@ Full walkthrough: [profiling guide]({link("wiki/profiling")}).
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
The checks CI runs on a pull request (Python 3.12 or newer):
|
||||
A pull request that only adds files under `bios/` needs none of this: the
|
||||
validation above is its check. For code, profiles or platform files, these are
|
||||
the checks CI runs on a pull request (Python 3.12 or newer):
|
||||
|
||||
```bash
|
||||
pip install pyyaml "jsonschema[format-nongpl]" markdown \\
|
||||
@@ -772,7 +790,9 @@ in `dist/`, which is why maintainers run it before a release.
|
||||
|
||||
CI computes SHA1/MD5/CRC32 for every new file, checks them against the platform
|
||||
configs, validates the YAML against the schemas, confronts each added profile
|
||||
with its upstream source, runs the test suite, and posts a report on the PR.
|
||||
with its upstream source and runs the test suite. The BIOS report goes to the
|
||||
summary of the Validation run, and also to a comment when the branch lives in
|
||||
this repository: a fork's pull request runs with a read-only token.
|
||||
|
||||
Contributors who add platform support are credited in the README,
|
||||
{site_words}, and in the BIOS packs.
|
||||
|
||||
@@ -2697,7 +2697,7 @@ def generate_emulators_index(profiles: dict) -> str:
|
||||
lines = [
|
||||
"# Emulators",
|
||||
"",
|
||||
f"**{len(unique)}** emulator profiles, **{total_files}** files total, **{len(aliases)}** aliases.",
|
||||
f"**{len(unique):,}** emulator profiles, **{total_files:,}** files total, **{len(aliases):,}** aliases.",
|
||||
"",
|
||||
"| Classification | Count | Description |",
|
||||
"|---------------|-------|-------------|",
|
||||
|
||||
@@ -314,8 +314,8 @@ def _hash_matches(declared: str, actual: str) -> bool:
|
||||
"""Compare a declared hash value against an actual hex digest.
|
||||
|
||||
Handles comma-separated multi-hash lists and uppercase (Recalbox)
|
||||
and truncated MD5s (Batocera 29-char): a declared value shorter
|
||||
than the digest matches by prefix.
|
||||
and MD5s shorter than 32 characters: a declared value shorter than
|
||||
the digest matches by prefix.
|
||||
"""
|
||||
actual = actual.lower()
|
||||
for cand in declared.split(","):
|
||||
|
||||
@@ -793,7 +793,12 @@ def digest_verdict(
|
||||
actual = hashes()[algorithm].lower()
|
||||
if algorithm != "md5":
|
||||
return actual in declared
|
||||
if any(actual == d or (len(d) < 32 and actual.startswith(d)) for d in declared):
|
||||
# Recalbox hashes a .zip by its members only (Bios.cpp:64-68), so the
|
||||
# container's own md5 answers nothing there.
|
||||
composite_only = file_entry.get("zip_digest") == "composite"
|
||||
if not composite_only and any(
|
||||
actual == d or (len(d) < 32 and actual.startswith(d)) for d in declared
|
||||
):
|
||||
return True
|
||||
member_digest = composite()
|
||||
return bool(member_digest) and member_digest.lower() in declared
|
||||
|
||||
+1
-1
@@ -1476,7 +1476,7 @@ def _format_ground_truth_aggregate(ground_truth: list[dict]) -> str:
|
||||
def _format_ground_truth_verbose(ground_truth: list[dict]) -> list[str]:
|
||||
"""Format ground truth as one line per core with expected values and source ref.
|
||||
|
||||
Example: handy validates size=512,crc32=0d973c9d [rom.h:48-49]
|
||||
Example: handy validates crc32=0d973c9d,size=512 [lynx/rom.h:48-49]
|
||||
"""
|
||||
lines = []
|
||||
for gt in ground_truth:
|
||||
|
||||
@@ -1251,6 +1251,30 @@ class OneBaselineVerdict(unittest.TestCase):
|
||||
("frontend_digest_exact", None),
|
||||
)
|
||||
|
||||
def test_a_composite_only_archive_ignores_its_raw_md5(self):
|
||||
"""Recalbox hashes a .zip by its members (Bios.cpp:64-68), and the
|
||||
pack check accepted the container's own md5 before the composite:
|
||||
neocd.bin shipped as neocdz.zip passed while Recalbox marks it red."""
|
||||
import hashlib
|
||||
|
||||
from scripts.common import md5_composite
|
||||
from scripts.validation import digest_verdict
|
||||
|
||||
with tempfile.TemporaryDirectory(dir=TMP_ROOT) as directory:
|
||||
archive = self._archive(Path(directory), "rom.bin")
|
||||
raw = hashlib.md5(Path(archive).read_bytes()).hexdigest()
|
||||
members = md5_composite(archive)
|
||||
for declared, accepted in ((raw, False), (members, True)):
|
||||
entry = {"name": "dragon32.zip", "md5": declared, "zip_digest": "composite"}
|
||||
with self.subTest(declared=declared):
|
||||
self.assertIs(
|
||||
digest_verdict(entry, 0, lambda: {"md5": raw}, "md5", "",
|
||||
lambda: members),
|
||||
accepted,
|
||||
)
|
||||
plain = {"name": "dragon32.zip", "md5": raw}
|
||||
self.assertTrue(digest_verdict(plain, 0, lambda: {"md5": raw}, "md5"))
|
||||
|
||||
def test_a_release_asset_keeps_its_answer(self):
|
||||
from scripts.slots import apply_override
|
||||
|
||||
@@ -4537,11 +4561,9 @@ class ToolReferenceCountsMatchTheirArtifacts(unittest.TestCase):
|
||||
stated = [int(group.replace(",", "")) for group in match.groups()]
|
||||
self.assertEqual(stated, [len(versions), listed, len(store["entries"])])
|
||||
|
||||
def test_derivable_count(self):
|
||||
derivable = json.loads((ROOT / "derivable.json").read_text(encoding="utf-8"))
|
||||
match = re.search(r"(\d+) entries on the current snapshots", self.text)
|
||||
self.assertIsNotNone(match)
|
||||
self.assertEqual(int(match.group(1)), len(derivable))
|
||||
def test_no_derivable_count(self):
|
||||
"""The count now lives in derivable.json alone, which step 1b rewrites."""
|
||||
self.assertNotRegex(self.text, r"\d+ entries on the current snapshots")
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -83,5 +83,58 @@ class BeforeOpeningAPullRequest(unittest.TestCase):
|
||||
self.assertIn(f'"{spec}"', text)
|
||||
|
||||
|
||||
class OneFileThroughTheBrowser(unittest.TestCase):
|
||||
"""Adding one BIOS asked for a fork, a 5.5 GB clone, five pip packages,
|
||||
the data directories and the whole suite, and promised a PR comment that
|
||||
a fork's read-only token cannot post. The browser upload into bios/ and
|
||||
the CI report already did the work."""
|
||||
|
||||
def test_the_guide_and_the_issue_chooser_offer_the_upload(self):
|
||||
import yaml
|
||||
|
||||
text = generate_readme.generate_contributing()
|
||||
section = text.split("## Add a BIOS file", 1)[1].split("\n## ", 1)[0]
|
||||
self.assertIn(generate_readme.UPLOAD_URL, section)
|
||||
self.assertNotIn("Fork this repository", section)
|
||||
self.assertNotIn("reported in a comment", text)
|
||||
chooser = yaml.safe_load(
|
||||
(REPO_ROOT / ".github" / "ISSUE_TEMPLATE" / "config.yml").read_text(encoding="utf-8")
|
||||
)
|
||||
self.assertIn(generate_readme.UPLOAD_URL, [link["url"] for link in chooser["contact_links"]])
|
||||
|
||||
def test_ci_comments_only_on_branches_of_this_repository(self):
|
||||
workflow = (REPO_ROOT / ".github" / "workflows" / "validate.yml").read_text(encoding="utf-8")
|
||||
self.assertIn("github.event.pull_request.head.repo.full_name == github.repository", workflow)
|
||||
self.assertIn('>> "$GITHUB_STEP_SUMMARY"', workflow)
|
||||
self.assertIn("summary of the Validation run", generate_readme.generate_contributing())
|
||||
|
||||
def test_an_upload_at_the_bios_root_is_told_its_folder(self):
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
import validate_pr
|
||||
|
||||
scratch = REPO_ROOT / "tmp" / "tests"
|
||||
scratch.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(dir=scratch) as directory:
|
||||
(Path(directory) / "bios").mkdir()
|
||||
(Path(directory) / "bios" / "boot.rom").write_bytes(b"boot")
|
||||
from common import compute_hashes
|
||||
|
||||
sha1 = compute_hashes(str(Path(directory) / "bios" / "boot.rom"))["sha1"]
|
||||
hashes = {"sha1": {sha1}, "folders": {sha1: {"bios/Maker/Console/"}}}
|
||||
cwd = os.getcwd()
|
||||
os.chdir(directory)
|
||||
try:
|
||||
result = validate_pr.validate_file("bios/boot.rom", None, hashes)
|
||||
finally:
|
||||
os.chdir(cwd)
|
||||
self.assertTrue(result.passed)
|
||||
self.assertTrue(any(
|
||||
"bios/ root" in message and "`bios/Maker/Console/`" in message
|
||||
for _status, message in result.checks
|
||||
))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -273,6 +273,8 @@ class PeerToneTowardPlatforms(unittest.TestCase):
|
||||
WORDS = re.compile(
|
||||
r"(?i)\bgets wrong\b|\bstale (hash|entries|entry)\b|<their file>|\bpatch their\b"
|
||||
r"|, corrected\b|\bcorrected in place\b|\bdeclare incorrectly\b|\blist error\b"
|
||||
r"|\btheir syntax\b|\bthe platform lacks\b|\bthe odd one\b|\bmay not match reality\b"
|
||||
r"|\ball their code does\b|\bcan drift from\b"
|
||||
)
|
||||
|
||||
def test_published_pages(self):
|
||||
@@ -322,6 +324,18 @@ class Md5CompositeDefinition(unittest.TestCase):
|
||||
self.assertIn("contents of every member", section)
|
||||
self.assertNotIn("inner file MD5s", section)
|
||||
|
||||
def test_composite_only_is_read_by_both_checks(self):
|
||||
"""The page said the direct MD5 is tried before the composite, which
|
||||
Recalbox never does for a .zip, and the pack check did the same."""
|
||||
import common
|
||||
import validation
|
||||
|
||||
section = _section(_page("verification-modes.md"), "md5_composite (Recalbox ZIP verification)")
|
||||
self.assertIn("zip_digest: composite", section)
|
||||
self.assertNotIn("is tried before reporting a mismatch", section)
|
||||
for function in (common.resolve_local_file, validation.digest_verdict):
|
||||
self.assertIn('get("zip_digest") == "composite"', inspect.getsource(function))
|
||||
|
||||
|
||||
class ValidationIndexScope(unittest.TestCase):
|
||||
"""The page said the index reads every profile. verify and the builder
|
||||
@@ -416,5 +430,318 @@ class PythonFloorOnRetroPie(unittest.TestCase):
|
||||
self.assertIn("3.7.3", changing)
|
||||
|
||||
|
||||
class ToolsRunOnTheStatedFloor(unittest.TestCase):
|
||||
"""tools.md and testing-guide.md state Python 3.9 for every script, and
|
||||
check_release_assets.py built `int | None` at import: a TypeError on 3.9
|
||||
before any argument was read, in check_release.py too. CI runs 3.12 only,
|
||||
so this test reads the source for what 3.9 cannot evaluate."""
|
||||
|
||||
TYPES = {"int", "str", "float", "bytes", "bool", "dict", "list", "tuple", "set", "type"}
|
||||
|
||||
def _type_like(self, node: ast.AST) -> bool:
|
||||
if isinstance(node, ast.Constant) and node.value is None:
|
||||
return True
|
||||
if isinstance(node, ast.Subscript):
|
||||
node = node.value
|
||||
if isinstance(node, ast.BinOp) and isinstance(node.op, ast.BitOr):
|
||||
return self._type_like(node.left) or self._type_like(node.right)
|
||||
return isinstance(node, ast.Name) and node.id in self.TYPES
|
||||
|
||||
def _runtime_unions(self, tree: ast.AST) -> list[int]:
|
||||
annotations = set()
|
||||
for node in ast.walk(tree):
|
||||
for field in ("annotation", "returns"):
|
||||
value = getattr(node, field, None)
|
||||
if value is not None:
|
||||
annotations.update(id(n) for n in ast.walk(value))
|
||||
return [
|
||||
node.lineno for node in ast.walk(tree)
|
||||
if id(node) not in annotations
|
||||
and (isinstance(node, ast.Match)
|
||||
or isinstance(node, ast.BinOp) and isinstance(node.op, ast.BitOr)
|
||||
and (self._type_like(node.left) or self._type_like(node.right)))
|
||||
]
|
||||
|
||||
def test_pages_state_the_floor(self):
|
||||
self.assertIn("Python 3.9 or newer", _page("tools.md"))
|
||||
self.assertIn("Python 3.9 or newer", _page("testing-guide.md"))
|
||||
|
||||
def test_no_script_evaluates_310_syntax(self):
|
||||
offenders = [
|
||||
f"{path.relative_to(ROOT)}:{line}"
|
||||
for path in sorted((ROOT / "scripts").rglob("*.py"))
|
||||
for line in self._runtime_unions(ast.parse(path.read_text(encoding="utf-8")))
|
||||
]
|
||||
self.assertEqual(offenders, [], "\n".join(offenders))
|
||||
|
||||
def test_the_check_catches_the_alias(self):
|
||||
self.assertTrue(self._runtime_unions(ast.parse("F = tuple[str, int | None]")))
|
||||
self.assertFalse(self._runtime_unions(ast.parse("def f(x: int | None) -> str | None: ...")))
|
||||
self.assertFalse(self._runtime_unions(ast.parse("mask = flags | other")))
|
||||
|
||||
|
||||
class DeriveFiguresLiveWithTheTool(unittest.TestCase):
|
||||
"""tools.md gave --derive about 90 s and 67 entries while its --help said
|
||||
about 35 s: two hand-kept figures for one run, measured at 33 s."""
|
||||
|
||||
def test_the_page_carries_no_duration_or_count(self):
|
||||
paragraph = next(
|
||||
p for p in _page("tools.md").split("\n\n") if "`--derive` reads" in p
|
||||
)
|
||||
self.assertNotRegex(paragraph, r"\d+ ?s\b|\d+ entries")
|
||||
self.assertIn("derivable.json", paragraph)
|
||||
|
||||
|
||||
class EmulatorVerboseSample(unittest.TestCase):
|
||||
"""advanced-usage.md showed `handy validates size=512 crc32=0x0d973c9d
|
||||
[src/handy/system.h:45]`, a format verify.py does not print and a
|
||||
reference handy.yml does not cite."""
|
||||
|
||||
def test_every_sample_line_is_printed(self):
|
||||
import subprocess
|
||||
|
||||
section = _section(_page("advanced-usage.md"), "Per-emulator checks")
|
||||
sample = [body for lang, body in re.findall(r"(?ms)^```(\w*)\n(.*?)^```", section) if not lang][-1]
|
||||
printed = subprocess.run(
|
||||
[sys.executable, "scripts/verify.py", "--emulator", "handy", "--verbose"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True,
|
||||
).stdout.splitlines()
|
||||
for line in sample.splitlines():
|
||||
with self.subTest(line=line):
|
||||
self.assertIn(line, printed)
|
||||
|
||||
|
||||
class ShortMd5Handling(unittest.TestCase):
|
||||
"""Three pages said some Batocera entries carry 29-character MD5s and show
|
||||
as untested. No platform YAML holds one, since the Batocera scraper
|
||||
completes them, and adding-a-scraper.md said to store them as-is."""
|
||||
|
||||
def test_no_page_claims_short_md5_entries(self):
|
||||
offenders = [
|
||||
page.name for page in PUBLISHED
|
||||
if re.search(r"(?i)29[- ]char|truncated md5 hashes in some entries|store the hash as-is",
|
||||
page.read_text(encoding="utf-8"))
|
||||
]
|
||||
self.assertEqual(offenders, [])
|
||||
|
||||
def test_the_page_names_what_the_code_does(self):
|
||||
import common
|
||||
from scraper import batocera_scraper
|
||||
|
||||
scraper = inspect.getsource(batocera_scraper)
|
||||
self.assertIn("md5 = _resolve_truncated_md5(", scraper)
|
||||
resolver = inspect.getsource(common.resolve_local_file)
|
||||
self.assertIn("if len(md5_candidate) < 32:", resolver)
|
||||
self.assertIn("_md5_name_ok(path)", resolver)
|
||||
section = _section(_page("verification-modes.md"), "MD5 shorter than 32 characters (Batocera)")
|
||||
self.assertIn("_resolve_truncated_md5", section)
|
||||
|
||||
|
||||
class RegistryKeysDocumented(unittest.TestCase):
|
||||
"""zip_digest, firmware_folder_flat and extract_to were read from the
|
||||
registry by the resolver, the pack and the README, and the page that
|
||||
adds a platform named none of them while saying runtime behaviour lives
|
||||
in the platform YAML only."""
|
||||
|
||||
def test_every_registry_key_is_on_the_page(self):
|
||||
import yaml
|
||||
|
||||
registry = yaml.safe_load((ROOT / "platforms" / "_registry.yml").read_text(encoding="utf-8"))
|
||||
keys = {key for entry in registry["platforms"].values() for key in entry}
|
||||
page = _page("adding-a-platform.md")
|
||||
self.assertEqual(sorted(k for k in keys if not re.search(rf"\b{k}\b", page)), [])
|
||||
|
||||
def test_the_page_says_the_registry_reaches_the_config(self):
|
||||
import common
|
||||
|
||||
source = inspect.getsource(common.load_platform_config)
|
||||
self.assertIn("config[key] = val", source)
|
||||
section = _section(_page("adding-a-platform.md"), "Step 2: Register the platform")
|
||||
self.assertIn("copies every registry key", section)
|
||||
self.assertNotIn("belongs to the platform\nYAML", section)
|
||||
|
||||
|
||||
class ResolutionSkipsNamesForUnheldFiles(unittest.TestCase):
|
||||
"""The resolution table promised name_exact to any entry without a hash,
|
||||
while an unsourceable entry, or one whose indexed path is absent from
|
||||
disk, skips every step that answers by name and returns not_found."""
|
||||
|
||||
def test_the_page_names_both_cases_the_code_guards(self):
|
||||
import common
|
||||
|
||||
source = inspect.getsource(common.resolve_local_file)
|
||||
self.assertIn('unsourceable = bool(file_entry.get("unsourceable")) or own_file_absent', source)
|
||||
for guard in (
|
||||
"if not has_strong_hash and not unsourceable:",
|
||||
"elif not unsourceable:",
|
||||
"if _depth < 3 and not has_strong_hash and not unsourceable:",
|
||||
"if data_dir_registry and (has_strong_hash or not unsourceable):",
|
||||
):
|
||||
self.assertIn(guard, source)
|
||||
section = _section(_page("verification-modes.md"), "File resolution chain")
|
||||
self.assertIn("`unsourceable:`", section)
|
||||
self.assertIn("restore_large_files.py", section)
|
||||
|
||||
|
||||
class CiTriggersPointToTheirLists(unittest.TestCase):
|
||||
"""The CI table listed six directories for validate.yml, which also runs
|
||||
on the installer, wiki/ and every other site input: a contributor
|
||||
touching only wiki/ read that no check would run."""
|
||||
|
||||
def test_the_table_does_not_copy_a_path_list(self):
|
||||
import yaml
|
||||
|
||||
section = _section(_page("architecture.md"), "CI workflows")
|
||||
self.assertNotRegex(section, r"on bios/, platforms/|\(platforms, emulators, wiki, scripts\)")
|
||||
self.assertIn("`paths:`", section)
|
||||
self.assertIn("test_workflow_paths.py", section)
|
||||
workflow = yaml.safe_load((ROOT / ".github/workflows/validate.yml").read_text(encoding="utf-8"))
|
||||
paths = workflow.get("on", workflow.get(True))["pull_request"]["paths"]
|
||||
for named in ("wiki/**", "install.sh"):
|
||||
self.assertIn(named, paths)
|
||||
self.assertIn(f"`{named.rstrip('*')}`", section)
|
||||
|
||||
|
||||
class ScratchFilesGoToRepoTmp(unittest.TestCase):
|
||||
"""adding-a-scraper.md wrote its comparison to /tmp while two other
|
||||
pages say /tmp is a RAM disk and send scratch files to the repo's tmp/."""
|
||||
|
||||
@staticmethod
|
||||
def _code_lines(text: str) -> list[str]:
|
||||
lines, inside = [], False
|
||||
for line in text.splitlines():
|
||||
if line.strip().startswith("```"):
|
||||
inside = not inside
|
||||
elif inside:
|
||||
lines.append(line)
|
||||
return lines
|
||||
|
||||
def test_no_command_writes_to_slash_tmp(self):
|
||||
offenders = [
|
||||
f"{page.name}: {line.strip()}"
|
||||
for page in PUBLISHED
|
||||
for line in self._code_lines(page.read_text(encoding="utf-8"))
|
||||
if re.search(r"(?<![\w.])/tmp/", line)
|
||||
]
|
||||
self.assertEqual(offenders, [])
|
||||
old = " ```bash\n python x.py --output /tmp/test.yml\n ```\n"
|
||||
self.assertEqual(self._code_lines(old), [" python x.py --output /tmp/test.yml"])
|
||||
self.assertIn("tmp/", (ROOT / ".gitignore").read_text(encoding="utf-8").splitlines())
|
||||
|
||||
|
||||
class ReadmeInstallerOptions(unittest.TestCase):
|
||||
"""The README named --system, --region and --check but showed options
|
||||
only after `sh -s --`. Appended to `irm ... | iex`, PowerShell rejects
|
||||
them, and the form that works was on the installer page alone, which the
|
||||
README never linked."""
|
||||
|
||||
def test_quick_install_gives_the_powershell_form_and_the_page(self):
|
||||
import generate_readme
|
||||
|
||||
source = inspect.getsource(generate_readme.generate_readme)
|
||||
self.assertIn("&([scriptblock]::Create((irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1)))", source)
|
||||
self.assertIn("wiki/installer/#passing-options", source)
|
||||
self.assertIn("## Passing options", _page("installer.md"))
|
||||
readme = (ROOT / "README.md").read_text(encoding="utf-8")
|
||||
quick = _section(readme, "Quick install")
|
||||
self.assertIn("[scriptblock]::Create", quick)
|
||||
self.assertIn("wiki/installer/#passing-options", quick)
|
||||
self.assertNotRegex(readme, r"(?m)^irm \S+ \| iex \S")
|
||||
|
||||
|
||||
class PublishedNumbersShareOneFormat(unittest.TestCase):
|
||||
"""The README gave the collection as `16011 MB` beside `19,526 files`,
|
||||
the site home as 15.6 GB, and the emulators page `10095 files`."""
|
||||
|
||||
def test_generators_format_counts_and_sizes_alike(self):
|
||||
import generate_readme
|
||||
import generate_site
|
||||
|
||||
readme_source = inspect.getsource(generate_readme.generate_readme)
|
||||
self.assertNotIn("size_mb", readme_source)
|
||||
self.assertIn("format_size(total_size)", readme_source)
|
||||
self.assertIn("{total_files:,}", inspect.getsource(generate_site))
|
||||
self.assertEqual(generate_readme.format_size(16011 * 1024 * 1024), "15.6 GB")
|
||||
|
||||
def test_the_readme_has_no_bare_large_count(self):
|
||||
readme = (ROOT / "README.md").read_text(encoding="utf-8")
|
||||
self.assertEqual(re.findall(r"\*\*\d{4,}\b[^*]*\*\*", readme), [])
|
||||
|
||||
|
||||
class FirstInstallWalkthrough(unittest.TestCase):
|
||||
"""First install showed a Batocera run of 12893 files with no omission
|
||||
line and said any other ending meant work left undone, while every
|
||||
Batocera run printed 399 files not installed. Its disk sizes (Batocera
|
||||
4.0 GB) contradicted its own example (8.6 GB), and its no-platform case
|
||||
said a piped command cannot ask, where install.sh hands it the terminal
|
||||
and the installer shows a menu."""
|
||||
|
||||
def setUp(self):
|
||||
self.text = _page("first-install.md")
|
||||
|
||||
def test_no_hand_kept_size_before_the_run(self):
|
||||
import install
|
||||
|
||||
section = _section(self.text, "Before starting")
|
||||
# The confirmation threshold is a constant of the installer, not a pack size.
|
||||
self.assertEqual(install.CONFIRM_FROM_BYTES, 1024 ** 3)
|
||||
self.assertIn("reaches 1 GB", section)
|
||||
self.assertNotRegex(section.replace("reaches 1 GB", ""), r"\d+(\.\d+)? ?[KMG]B\b")
|
||||
|
||||
def test_the_run_is_dated_and_consistent_with_the_manifest(self):
|
||||
import install
|
||||
|
||||
self.assertRegex(self.text, r"published on \d{4}-\d{2}-\d{2}")
|
||||
step3 = _section(self.text, "Step 3. Let it work")
|
||||
step4 = _section(self.text, "Step 4. Read the end of the run")
|
||||
step5 = _section(self.text, "Step 5. Confirm it later")
|
||||
count, size = re.search(r"\n (\d+) files \(([\d.]+ [KMG]B)\)", step3).groups()
|
||||
self.assertIn(f"0/{count} present", step3)
|
||||
self.assertIn(f"Downloading {count} files ({size})", step3)
|
||||
self.assertIn(f"Done. {count} files installed", step4)
|
||||
self.assertIn(f"{count}/{count} present ({count} verified", step5)
|
||||
manifest = json.loads((ROOT / "install" / "batocera.json").read_text(encoding="utf-8"))
|
||||
omitted = re.search(r"Not installable: (\d+) files \((\d+) required", step3)
|
||||
if manifest.get("omitted_files"):
|
||||
self.assertIsNotNone(omitted, "the manifest omits files and the run shows no line for it")
|
||||
if omitted:
|
||||
total, required = map(int, omitted.groups())
|
||||
first, needed = install._omission_lines(
|
||||
total, [{"required": True, "dest": str(i)} for i in range(required)]
|
||||
)[:2]
|
||||
self.assertIn(first, step4)
|
||||
self.assertIn(needed.split(":")[0], step4)
|
||||
self.assertIn(f"all {required} belong", step4)
|
||||
self.assertIn(f"other {total - required} are optional", step4)
|
||||
|
||||
def test_no_platform_detected_reads_like_the_installer(self):
|
||||
source = (ROOT / "install.py").read_text(encoding="utf-8")
|
||||
branch = source.split('print(" No supported platform detected.")', 1)[1][:1600]
|
||||
self.assertIn("if not interactive or listing:", branch)
|
||||
self.assertIn("_prompt_manual_platform(os_type)", branch)
|
||||
self.assertIn('"$INSTALLER" "$@" </dev/tty', (ROOT / "install.sh").read_text(encoding="utf-8"))
|
||||
section = _section(self.text, "No supported platform detected.")
|
||||
self.assertIn("a menu follows", section)
|
||||
self.assertIn("--no-input", section)
|
||||
self.assertNotIn("cannot stop and ask", section)
|
||||
|
||||
|
||||
class RetroArchWindowsSystemFolder(unittest.TestCase):
|
||||
"""Getting started gave %APPDATA%\\RetroArch\\system for the Windows
|
||||
installer. RetroArch's default is `:\\system`, the folder of
|
||||
retroarch.exe (platform_win32.c). %APPDATA% is where a retroarch.cfg
|
||||
can live, and install.py reads it there."""
|
||||
|
||||
def test_page_and_installer_agree(self):
|
||||
import install
|
||||
|
||||
source = inspect.getsource(install)
|
||||
self.assertIn('ra_dir or exe_dir / "system"', source)
|
||||
self.assertTrue(str(install._os_default_dests("windows")["retroarch"]).replace("\\", "/").endswith("/system"))
|
||||
section = _section(_page("getting-started.md"), "RetroArch")
|
||||
self.assertNotIn("%APPDATA%\\RetroArch\\system", section)
|
||||
self.assertIn("`system\\` next to `retroarch.exe`", section)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+2
-2
@@ -265,7 +265,7 @@ class TestE2E(unittest.TestCase):
|
||||
).hexdigest() # sorted: a.rom, b.rom
|
||||
multi_wrong = "0000000000000000000000000000000"
|
||||
multi_right = f["multi.zip"]["inner_md5s"]["rom.bin"]
|
||||
truncated_md5 = f["truncated.bin"]["md5"][:29] # Batocera 29-char
|
||||
truncated_md5 = f["truncated.bin"]["md5"][:29] # shorter than 32 characters
|
||||
|
||||
config = {
|
||||
"platform": "TestMD5",
|
||||
@@ -347,7 +347,7 @@ class TestE2E(unittest.TestCase):
|
||||
"zipped_file": "rom.bin",
|
||||
"required": True,
|
||||
},
|
||||
# Truncated MD5 (Batocera 29 chars)
|
||||
# MD5 shorter than 32 characters
|
||||
{
|
||||
"name": "truncated.bin",
|
||||
"destination": "truncated.bin",
|
||||
|
||||
@@ -86,7 +86,7 @@ class DeclaredHashVerdict(unittest.TestCase):
|
||||
)
|
||||
|
||||
def test_a_truncated_md5_still_decides(self):
|
||||
"""Batocera publishes 29-character prefixes, which are still evidence."""
|
||||
"""An MD5 shorter than 32 characters is still evidence, by prefix."""
|
||||
self.assertEqual(
|
||||
self.verdict(self.right, md5_list=[self.d["md5"][:29]]),
|
||||
"data_dir_hash_exact",
|
||||
|
||||
@@ -118,7 +118,7 @@ class ShippedFilesSatisfyTheirDeclaration(unittest.TestCase):
|
||||
digests["md5"].append(common.md5_composite(str(blob_path)))
|
||||
except (zipfile.BadZipFile, OSError):
|
||||
pass
|
||||
# Batocera publishes 29-character prefixes, so compare by prefix.
|
||||
# A declared MD5 can be shorter than 32 characters: compare by prefix.
|
||||
satisfied = any(
|
||||
all(
|
||||
any(
|
||||
|
||||
@@ -131,10 +131,13 @@ This fetches from upstream and prints a summary without writing anything.
|
||||
|
||||
Add an entry to `platforms/_registry.yml` under the `platforms:` key.
|
||||
|
||||
The registry describes *where the data comes from*. How the platform behaves
|
||||
at runtime (`verification_mode`, `base_destination`) belongs to the platform
|
||||
YAML, because that is what `verify.py` and `generate_pack.py` load. Keeping the
|
||||
two apart means a re-scrape can rewrite the YAML without touching the registry.
|
||||
The registry describes *where the data comes from*, and the platform YAML
|
||||
holds what the upstream file declares, `verification_mode` and
|
||||
`base_destination` included. A re-scrape rewrites the YAML without touching the
|
||||
registry. `load_platform_config` copies every registry key the YAML does not
|
||||
carry into the config that `verify.py` and `generate_pack.py` read, so a runtime
|
||||
rule that no upstream file states (`zip_digest`, `firmware_folder_flat`) is
|
||||
written in the registry, where the next scrape leaves it in place.
|
||||
|
||||
### Required fields
|
||||
|
||||
@@ -190,6 +193,11 @@ its libretro layout whatever the platform calls it.
|
||||
logo: https://... # SVG or PNG for UI/docs
|
||||
inherits_from: retroarch # inherit systems/cores from another platform
|
||||
case_insensitive_fs: true # if the platform runs on case-insensitive filesystems
|
||||
extract_to: /userdata/bios/ # folder the pack extracts into, shown in the README,
|
||||
# the site and the release notes
|
||||
zip_digest: composite # the frontend hashes a .zip by its members only
|
||||
firmware_folder_flat: true # the frontend lists each system folder without
|
||||
# descending into it
|
||||
source_wiki: https://... # human-readable upstream reference
|
||||
target_scraper: myplatform_targets # hardware target scraper name, null if static
|
||||
target_source: https://... # target data source URL
|
||||
@@ -205,6 +213,19 @@ its libretro layout whatever the platform calls it.
|
||||
parse_key: bios_directory
|
||||
```
|
||||
|
||||
`zip_digest: composite` is set where the frontend's own check takes
|
||||
`Zip::Md5Composite` for every `.zip` and never the container's MD5 (Recalbox,
|
||||
`Bios.cpp:64-68`). Each `.zip` entry is then checked against its composite alone,
|
||||
by the resolver and by the pack check, as
|
||||
[verification modes](verification-modes.md#md5_composite-recalbox-zip-verification)
|
||||
describes.
|
||||
|
||||
`firmware_folder_flat: true` is set where the frontend reads a system's firmware
|
||||
folder without descending into it (RomM 5.4.0,
|
||||
`backend/handler/filesystem/base_handler.py:1002`, `recursive=False`). On such a platform, a core file whose destination has a
|
||||
subfolder is left out of the pack (`nested` in `scripts/packextras.py`), since no
|
||||
core would receive it.
|
||||
|
||||
### Platform YAML header
|
||||
|
||||
The runtime behavior lives at the top of `platforms/myplatform.yml`:
|
||||
|
||||
@@ -206,10 +206,12 @@ System ID consistency matters for cross-platform operations. The same console mu
|
||||
- Handle missing hashes gracefully (set to `None`, not empty string).
|
||||
- Some platforms provide multiple hash types per entry. Populate whichever fields
|
||||
are available.
|
||||
- Batocera uses 29-character truncated MD5 hashes in some entries. The resolution
|
||||
layer handles prefix matching, but the scraper should store the hash as-is.
|
||||
- An MD5 shorter than 32 characters is completed when exactly one MD5 in
|
||||
`database.json` starts with it, as the Batocera scraper does
|
||||
(`_resolve_truncated_md5`), and kept as written otherwise. The resolution layer
|
||||
matches what remains by prefix.
|
||||
|
||||
Scraped data reflects what the upstream declares, which may not match reality. The scraper's job is faithful transcription of upstream data, not correction. Corrections happen in the emulator profiles (source-verified) and in `_shared.yml` (curated). If a scraper meets a hash or filename that differs from the emulator source, log a warning and still include the upstream value. The divergence will surface during truth diffing.
|
||||
Scraped data reflects what the upstream declares, which can differ from what the emulator source reads. The scraper's job is faithful transcription of upstream data, not correction. Corrections happen in the emulator profiles (source-verified) and in `_shared.yml` (curated). If a scraper meets a hash or filename that differs from the emulator source, log a warning and still include the upstream value. The divergence will surface during truth diffing.
|
||||
|
||||
A few rules keep that transcription honest:
|
||||
|
||||
@@ -394,11 +396,14 @@ manipulation rather than load-dump to maintain human-readable formatting.
|
||||
python -m scripts.scraper.myplatform_scraper --json | python -m json.tool | head -50
|
||||
```
|
||||
|
||||
3. **Compare with existing YAML** if updating a scraper:
|
||||
3. **Compare with existing YAML** if updating a scraper. The output goes to the
|
||||
repository's `tmp/`, listed in `.gitignore`, because `/tmp` is a RAM disk on
|
||||
many systems:
|
||||
|
||||
```bash
|
||||
python -m scripts.scraper.myplatform_scraper --output /tmp/test.yml
|
||||
diff platforms/myplatform.yml /tmp/test.yml
|
||||
mkdir -p tmp
|
||||
python -m scripts.scraper.myplatform_scraper --output tmp/test.yml
|
||||
diff platforms/myplatform.yml tmp/test.yml
|
||||
```
|
||||
|
||||
4. **Run verification** after generating:
|
||||
|
||||
+17
-13
@@ -331,10 +331,11 @@ The diff reports:
|
||||
|
||||
### Export to native formats
|
||||
|
||||
Write a platform's own BIOS file, updated from the emulators' code. The output is the file its
|
||||
maintainers keep, not our data in their syntax: hashes the truth can prove
|
||||
are applied, entries the truth says nothing about are left alone, and files
|
||||
the truth knows and the platform lacks are added.
|
||||
Write a platform's own BIOS file with the updates its emulators' source code
|
||||
supports. The output is the platform's original file with those updates applied:
|
||||
a hash the emulator source states is updated, an entry the source says nothing
|
||||
about is kept as written, and a file the platform's emulators read that the list
|
||||
does not name yet is added.
|
||||
|
||||
```bash
|
||||
python scripts/export_native.py --platform batocera --fetch # batocera-systems
|
||||
@@ -355,10 +356,10 @@ patched from the platform's own file, which `--fetch` downloads once into
|
||||
`.cache/upstream-native/`. Without it the export fails rather than
|
||||
publishing a fragment.
|
||||
|
||||
RetroPie is the odd one: it publishes no BIOS list at all, and the only
|
||||
declaration is the sentence in each package's `rp_module_help`. The export
|
||||
adds a missing file name to a list a maintainer already wrote, never
|
||||
removes one, and never drafts a sentence where there was none.
|
||||
RetroPie keeps its BIOS information in the `rp_module_help` sentence of each
|
||||
package rather than in a list. The export adds a file name to a sentence a
|
||||
maintainer already wrote, never removes one, and never drafts a sentence where
|
||||
there was none.
|
||||
|
||||
The run reports what it changed, and what it had to leave out: an entry a
|
||||
format cannot express is named, never written half-formed. `es_bios.xsd`
|
||||
@@ -377,13 +378,16 @@ python scripts/verify.py --emulator handy
|
||||
python scripts/verify.py --emulator handy --verbose
|
||||
```
|
||||
|
||||
Default output shows aggregate results per file: the core name and which checks apply.
|
||||
With `--verbose`, each file expands to one line per core with the exact validation
|
||||
parameters and source code reference:
|
||||
The first line counts the files the profile declares and names the checks they
|
||||
carry. A file that is missing or fails a check gets a line of its own, followed by
|
||||
the cores that check it. `--verbose` also lists the files that pass, each followed
|
||||
by one line per core with the values it expects and the source reference the
|
||||
profile cites:
|
||||
|
||||
```
|
||||
lynxboot.img
|
||||
handy validates size=512 crc32=0x0d973c9d [src/handy/system.h:45]
|
||||
handy: 2/2 OK [crc32+existence+size]
|
||||
OK (optional): lynxboot.img
|
||||
handy validates crc32=0d973c9d,size=512 [lynx/rom.h:48-49 (ROM_SIZE=0x200, ROM_CRC32=0x0D973C9D), lynx/rom.cpp:76-129 (CRom constructor), libretro/libretro.cpp:1231-1258 (bios path + CSystem init)]
|
||||
```
|
||||
|
||||
### Per-system checks
|
||||
|
||||
@@ -372,7 +372,7 @@ platform's own verification mode.
|
||||
|
||||
| Case | Handling |
|
||||
|------|---------|
|
||||
| Batocera truncated MD5 (29 chars) | prefix match in resolution |
|
||||
| MD5 shorter than 32 characters | completed by the Batocera scraper when one MD5 in the collection starts with it, matched by prefix in resolution |
|
||||
| `zippedFile` entries | MD5 is of the ROM inside the ZIP, not the ZIP itself |
|
||||
| Regional variants (same filename) | `by_path_suffix` index disambiguates |
|
||||
| MAME BIOS ZIPs | `contents` field documents inner structure |
|
||||
@@ -450,8 +450,14 @@ pattern and how to add a test.
|
||||
|
||||
| Workflow | File | Trigger | Role |
|
||||
|----------|------|---------|------|
|
||||
| Deploy Site | `deploy-site.yml` | push to main (platforms, emulators, wiki, scripts) + manual | validate contracts, generate site, build with MkDocs, validate rendered HTML, deploy to Pages |
|
||||
| Validation | `validate.yml` | pull request and push to main on bios/, platforms/, emulators/, schemas/, scripts/, tests/ | schema check and full test suite on both, BIOS hash validation and auto-label on pull requests |
|
||||
| Deploy Site | `deploy-site.yml` | push to main on any input of the site build, and manual | validate contracts, generate site, build with MkDocs, validate rendered HTML, deploy to Pages |
|
||||
| Validation | `validate.yml` | pull request and push to main on the collection, profiles, schemas, scripts, tests, the installer and every input of the site build | schema check and full test suite on both, BIOS hash validation and auto-label on pull requests |
|
||||
|
||||
Each workflow's `paths:` list is the exact set of files that starts it.
|
||||
`tests/test_workflow_paths.py` holds the Deploy Site list to the scripts the build
|
||||
imports, and the Validation list to every input of the site build, so a change that
|
||||
would fail the deploy fails the pull request first. A change to `wiki/`, to
|
||||
`install.sh` or to any other path in that list runs the full suite.
|
||||
|
||||
Releases are not built in CI: the packs are generated and checked on the
|
||||
maintainer's machine and uploaded with `gh`, see the
|
||||
|
||||
+50
-17
@@ -15,9 +15,10 @@ Three things, and a few minutes:
|
||||
computer.
|
||||
- `curl` or `wget`, and Python 3.8 or newer. The command below checks for both
|
||||
and stops with a message if either is absent.
|
||||
- Room on the drive. A full RetroArch set is 5.5 GB, Batocera 4.0 GB, MiSTer
|
||||
FPGA 24 MB. The installer measures the free space and refuses before writing
|
||||
anything if there is not enough.
|
||||
- Room on the drive. The size depends on the platform and changes with each
|
||||
release: the run prints it before downloading (Step 3), and asks first when it
|
||||
reaches 1 GB. The installer also measures the free space and refuses before
|
||||
writing anything if there is not enough.
|
||||
|
||||
Nothing is written outside the BIOS folder, and games and saves are never
|
||||
touched.
|
||||
@@ -76,46 +77,73 @@ keeps its BIOS files, stop with Ctrl+C and see the last section.
|
||||
|
||||
## Step 3. Let it work
|
||||
|
||||
The examples on this page follow the Batocera list as published on 2026-10-21.
|
||||
The figures change with each release.
|
||||
|
||||
Two phases follow. The first reads what is already in the folder, which takes a
|
||||
few minutes on a memory card because every file is hashed:
|
||||
|
||||
```
|
||||
Fetching file index for batocera...
|
||||
12893 files (8.6 GB)
|
||||
14481 files (9.1 GB)
|
||||
Not installable: 399 files (79 required, not found: 399).
|
||||
|
||||
Checking existing files...
|
||||
0/12893 present (0 verified, 0 wrong hash)
|
||||
0/14481 present (0 verified, 0 wrong hash)
|
||||
```
|
||||
|
||||
`Not installable` counts the files of the list that the collection does not
|
||||
hold. They are named at the end of the run (Step 4).
|
||||
|
||||
On a first install the present count is zero, which is expected, not a fault.
|
||||
On a later run it counts what is already there, and only the rest is
|
||||
downloaded. A download of 1 GB or more is confirmed first
|
||||
(`Download 8.6 GB to /userdata/bios? [Y/n]`).
|
||||
(`Download 9.1 GB to /userdata/bios? [Y/n]`).
|
||||
|
||||
The second phase downloads what is missing, one line per file:
|
||||
|
||||
```
|
||||
Downloading 12893 files (8.6 GB)...
|
||||
Downloading 14481 files (9.1 GB)...
|
||||
Press Ctrl+C to stop. Running the same command again carries on from here.
|
||||
[1/12893] 3do_arcade_saot.bin ok
|
||||
[2/12893] 3dobios.zip ok
|
||||
[1/14481] 3do_arcade_saot.bin ok
|
||||
[2/14481] 3dobios.zip ok
|
||||
```
|
||||
|
||||
Ctrl+C is safe at any point. Files already installed stay, and running the
|
||||
command again picks up where it stopped.
|
||||
|
||||
## Step 4. Read the last three lines
|
||||
## Step 4. Read the end of the run
|
||||
|
||||
This is what a finished run looks like:
|
||||
|
||||
```
|
||||
Done. 12893 files installed, 0 files already up to date.
|
||||
Done. 14481 files installed, 0 files already up to date.
|
||||
Location: /userdata/bios
|
||||
399 files were not installed: the collection does not hold them, or its copy fails their check.
|
||||
Needed to start an emulator (79 files):
|
||||
dark-forces (theforceengine): DARK.GOB, SOUNDS.GOB, SPRITES.GOB, TEXTURES.GOB
|
||||
dark-forces-ii (openjkdf2): openjkdf2/episode/JK1.gob, openjkdf2/episode/JK1CTF.gob, openjkdf2/episode/JK1MP.gob, openjkdf2/resource/Res1hi.gob, openjkdf2/resource/Res2.gob
|
||||
diablo (devilutionx): devilutionx/diabdat.mpq
|
||||
doom3 (dhewm3): base/pak000.pk4, base/pak002.pk4, base/pak003.pk4, base/pak004.pk4
|
||||
droidworks (openjkdf2): dwCD.GOB, dwHD.GOB, dwMin.GOB, dwStream.GOB
|
||||
fallout (fallout1-ce): fallout1-ce/critter.dat, fallout1-ce/master.dat
|
||||
fallout2 (fallout2-ce): fallout2-ce/critter.dat, fallout2-ce/master.dat
|
||||
fury (fury): fury.grp
|
||||
... and 9 more systems (--check --verbose lists them all)
|
||||
Each emulator's page says what these files are: https://abdess.github.io/retrobios/emulators/
|
||||
To check this later, run the same command with --check.
|
||||
```
|
||||
|
||||
The word `Done` and a location are the confirmation. Any other ending means
|
||||
something was left undone, and the run says which case it was.
|
||||
The word `Done` and a location are the confirmation. Any other ending (`Stopped`,
|
||||
`Finished with errors`, `Nothing was installed`) means something was left undone,
|
||||
and the run says which case it was.
|
||||
|
||||
The lines between `Location:` and the `--check` hint name the files of the list
|
||||
that the collection does not hold. `Needed to start an emulator` gives, for each
|
||||
system, the emulators that read it and the files they cannot start without. On
|
||||
the run above, all 79 belong to games (Dark Forces, Diablo, Doom 3, Tomb Raider
|
||||
and the others): the engine reads them from a copy of the game itself. The
|
||||
other 320 are optional, and the emulators reading them start without them.
|
||||
|
||||
## Step 5. Confirm it later
|
||||
|
||||
@@ -127,12 +155,13 @@ curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh |
|
||||
|
||||
```
|
||||
Checking existing files...
|
||||
72/72 present (72 verified, 0 wrong hash)
|
||||
14481/14481 present (14481 verified, 0 wrong hash)
|
||||
|
||||
All files up to date.
|
||||
```
|
||||
|
||||
`0 wrong hash` is the line that matters. A number there means a file is present
|
||||
The files the collection does not hold are listed after it again, as long as
|
||||
they are absent from the folder. `0 wrong hash` is the line that matters. A number there means a file is present
|
||||
whose contents the platform will reject, and running the install again replaces
|
||||
it.
|
||||
|
||||
@@ -140,8 +169,12 @@ it.
|
||||
|
||||
### No supported platform detected.
|
||||
|
||||
The command was piped from the internet, so it cannot stop and ask a question.
|
||||
Name the platform and the folder instead:
|
||||
At a terminal, a menu follows that line. The platform is picked by its number,
|
||||
then the run asks for the BIOS folder, offering the platform's usual folder on
|
||||
this system as the default when it has one.
|
||||
|
||||
Without a terminal, or with `--no-input`, the run stops there and prints the
|
||||
command to use instead, with the platform and the folder named:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh \
|
||||
|
||||
@@ -108,18 +108,19 @@ RetroArch uses the `system_directory` setting in `retroarch.cfg`. Default locati
|
||||
|
||||
| OS | Default path |
|
||||
|----|-------------|
|
||||
| Windows (installer) | `%APPDATA%\RetroArch\system\` |
|
||||
| Windows (portable .7z) | `system\` next to `retroarch.exe` |
|
||||
| Windows (installer or portable .7z) | `system\` next to `retroarch.exe` |
|
||||
| Windows (UWP/MSIX package) | `%LOCALAPPDATA%\Packages\1e4cf179-f3c2-404f-b9f3-cb2070a5aad8_*\LocalState\system\` |
|
||||
| Linux | `~/.config/retroarch/system/` |
|
||||
| Linux (Flatpak) | `~/.var/app/org.libretro.RetroArch/config/retroarch/system/` |
|
||||
| macOS | `~/Library/Application Support/RetroArch/system/` |
|
||||
| Steam Deck (Flatpak) | `~/.var/app/org.libretro.RetroArch/config/retroarch/system/` |
|
||||
| Android | `/storage/emulated/0/RetroArch/system/` |
|
||||
|
||||
Windows has two layouts because the installer stores its data under `%APPDATA%`
|
||||
while the portable archive keeps everything inside the folder you extracted it
|
||||
to. If both exist on the machine, the one RetroArch actually reads is the one
|
||||
shown in the UI.
|
||||
On Windows, `retroarch.cfg` is read beside `retroarch.exe` first, then from
|
||||
`%APPDATA%\RetroArch\`. Either way, a `system_directory` left at its default is
|
||||
the `system` folder beside `retroarch.exe` (`:\system` in RetroArch's
|
||||
`frontend/drivers/platform_win32.c`, where `:` stands for the folder of the
|
||||
executable).
|
||||
|
||||
To check your actual path: open RetroArch, go to **Settings > Directory > System/BIOS**, or look for `system_directory` in `retroarch.cfg`.
|
||||
|
||||
|
||||
+3
-4
@@ -893,13 +893,12 @@ python scripts/provenance_report.py --json # full report
|
||||
```
|
||||
|
||||
A catalog lists chips as they come off the board, the collection often holds
|
||||
the image an emulator reads. `--derive` reads the collection (about 90 s) and
|
||||
the image an emulator reads. `--derive` reads every file of the collection and
|
||||
marks a missing entry DERIVABLE when a held file produces its sha1 by halving,
|
||||
quartering, splitting into even and odd bytes or 16-bit words (byte-swapped
|
||||
or not), swapping bytes, mirroring, or joining two files of one directory.
|
||||
The Amiga Kickstart chip pairs and TOSEC's split C64 and VES ROMs are found
|
||||
this way: 67 entries on the current snapshots, recorded in `derivable.json`
|
||||
by pipeline step 1b.
|
||||
this way. Pipeline step 1b records what it finds in `derivable.json`.
|
||||
|
||||
A missing entry that is a file an emulator profile reads and the collection
|
||||
lacks is listed first, with `needed by <emulator> as <name>` and whether the
|
||||
@@ -959,4 +958,4 @@ Located in `scripts/exporter/`. Each inherits `BaseExporter`, defined in
|
||||
| `romm_exporter` | `known_bios_files.json`, every entry with a size and a hash |
|
||||
| `bizhawk_exporter` | `FirmwareDatabase.cs`, BizHawk's own C# |
|
||||
| `misterfpga_exporter` | hash and size in MiSTer's BiosDB (`bios_db.json`) |
|
||||
| `retropie_exporter` | the `rp_module_help` prose of each scriptmodule, by adding a name it lacks |
|
||||
| `retropie_exporter` | the `rp_module_help` prose of each scriptmodule, by adding a file name the package's emulator reads |
|
||||
@@ -47,21 +47,29 @@ hashed with MD5, and compared case-insensitively against the expected value.
|
||||
If the `resolve_local_file` step already confirmed the MD5 match (status `md5_exact`),
|
||||
`verify.py` skips re-hashing and returns OK directly.
|
||||
|
||||
### Truncated MD5 (Batocera)
|
||||
### MD5 shorter than 32 characters (Batocera)
|
||||
|
||||
Some entries in Batocera's system data carry 29-character MD5 strings instead of
|
||||
the usual 32. Batocera's script compares the full string (`batocera-systems:1088`), so
|
||||
those entries report the file as untested. `verify.py` compares the prefix instead, to
|
||||
find the file the entry designates: if the expected hash is shorter than 32 characters,
|
||||
the actual hash is compared against only its first N characters.
|
||||
Batocera's script compares the full string (`batocera-systems:1088`), so an MD5
|
||||
shorter than 32 characters matches no file there. The Batocera scraper completes such
|
||||
a value when exactly one MD5 in `database.json` starts with it
|
||||
(`_resolve_truncated_md5` in `scripts/scraper/batocera_scraper.py`) and keeps it as
|
||||
written otherwise. `resolve_local_file` matches a value still shorter than 32
|
||||
characters by prefix, and only for a file whose name the entry also names.
|
||||
|
||||
### md5_composite (Recalbox ZIP verification)
|
||||
|
||||
Recalbox hashes a ZIP with `Zip::Md5Composite` (`es-core/src/utils/Zip.cpp:71-96` in
|
||||
recalbox-emulationstation at `f4cd0ae4`): one MD5 over the contents of every member,
|
||||
read in filename order. The members' own MD5s are not involved. `verify.py` computes
|
||||
the same value with `md5_composite()` (`scripts/hashing.py`). When a ZIP file's direct
|
||||
MD5 does not match, the composite is tried before reporting a mismatch.
|
||||
read in filename order. The members' own MD5s are not involved, and neither is the
|
||||
MD5 of the ZIP file itself: `Bios::Scan` takes the composite for every `.zip`
|
||||
(`es-app/src/bios/Bios.cpp:64-68`). `verify.py` computes the same value with
|
||||
`md5_composite()` (`scripts/hashing.py`).
|
||||
|
||||
The registry gives Recalbox `zip_digest: composite`. For a platform so marked, a
|
||||
`.zip` entry is checked against its composite alone, by the resolver
|
||||
(`resolve_local_file`) and by the pack check (`digest_verdict` in
|
||||
`scripts/validation.py`). On other platforms the ZIP file's direct MD5 is tried first
|
||||
and the composite after it.
|
||||
|
||||
### Multi-hash (Recalbox)
|
||||
|
||||
@@ -266,6 +274,20 @@ evidence-compatible match:
|
||||
|
||||
If no step matches, the result is `(None, "not_found")`.
|
||||
|
||||
Two kinds of entry skip every answer given by a name alone: steps 6 and 9, the
|
||||
`hash_mismatch` of step 7, and the name lookup of step 10 when no hash is declared.
|
||||
|
||||
- An entry whose profile carries `unsourceable:`. The collection cannot hold that
|
||||
file, so a file answering to its name belongs to something else: `kanji.rom`
|
||||
names an FPseNG font, an openMSX font and a 3DO ROM.
|
||||
- An entry whose full destination path is in the `by_path_suffix` index while no
|
||||
indexed copy is on disk. That file is a release asset the checkout has not
|
||||
restored (`scripts/restore_large_files.py`), and another file of the same name
|
||||
is not it.
|
||||
|
||||
The content steps still apply to both, so collecting the real bytes, or restoring
|
||||
the asset, makes the entry resolve.
|
||||
|
||||
The `hash_mismatch` status means a file with the right name or path exists but its hash
|
||||
does not match. This still resolves to a local path (the file is present), but verification
|
||||
will report it as UNTESTED with a reason string showing the expected vs actual hash prefix.
|
||||
|
||||
Reference in new issue
Block a user