mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-11-06 01:53:56 -06:00
chore: merge site and docs fixes, third pass
This commit is contained in:
Abdessamad Derraz committed
commit
19898cb6ea
46 files changed
+38582
-163
No files matched your search
@@ -53,6 +53,7 @@ on:
|
||||
- "scripts/sitemeasurements.py"
|
||||
- "scripts/siteexports.py"
|
||||
- "scripts/siterender.py"
|
||||
- "scripts/retired_files.py"
|
||||
- "scripts/sitesources.py"
|
||||
- "scripts/sitestarlight.py"
|
||||
- "scripts/sitewrite.py"
|
||||
@@ -79,6 +80,7 @@ on:
|
||||
- "database.json"
|
||||
- "release.json"
|
||||
- "derivable.json"
|
||||
- "retired.json"
|
||||
- "measurements/**"
|
||||
# The Proof lines of the emulator pages read what an issue's runs showed.
|
||||
- "tests/campaign/observations.yml"
|
||||
|
||||
@@ -29,6 +29,7 @@ on:
|
||||
- "database.json"
|
||||
- "release.json"
|
||||
- "derivable.json"
|
||||
- "retired.json"
|
||||
- "measurements/**"
|
||||
- "website/**"
|
||||
# Spelled out twice because the workflow parser reads no YAML anchor. A test
|
||||
@@ -54,6 +55,7 @@ on:
|
||||
- "database.json"
|
||||
- "release.json"
|
||||
- "derivable.json"
|
||||
- "retired.json"
|
||||
- "measurements/**"
|
||||
- "website/**"
|
||||
|
||||
|
||||
+4
-5
@@ -71,7 +71,7 @@ the checks CI runs on a pull request (Python 3.12 or newer):
|
||||
pip install pyyaml "jsonschema[format-nongpl]" markdown pyte hypothesis vermin setuptools
|
||||
python scripts/validate_pr.py --changed # the BIOS files the branch adds
|
||||
python scripts/validate_schemas.py --source-only # profiles and platform files
|
||||
python scripts/profile_sync.py --added --base upstream/main --strict # each profile the branch adds
|
||||
python scripts/profile_sync.py --added --strict # each profile the branch adds
|
||||
python scripts/refresh_data_dirs.py # data directories the pack-building tests read
|
||||
python scripts/generate_site.py # the pages the site tests read
|
||||
npm ci --prefix website && npm run build --prefix website # the built site, Node 24
|
||||
@@ -80,10 +80,9 @@ python scripts/validate_site.py --site-dir website/dist # links and page budget
|
||||
vermin --no-tips -t=3.8- --violations install.py # install.py stays on Python 3.8
|
||||
```
|
||||
|
||||
`upstream` is this repository as a remote of your fork
|
||||
(`git remote add upstream https://github.com/Abdess/retrobios`, then
|
||||
`git fetch upstream`). A profile you only edit is not held to `--strict`: its
|
||||
older refs are reviewed by hand.
|
||||
`profile_sync.py` compares the branch with this repository's `main`, and
|
||||
fetches it in a fork's clone. A profile you only edit is not held to
|
||||
`--strict`: its older refs are reviewed by hand.
|
||||
|
||||
The full pipeline is not needed for a contribution: it rebuilds every pack
|
||||
in `dist/`, which is why maintainers run it before a release.
|
||||
|
||||
@@ -27,7 +27,7 @@ irm https://abdess.github.io/retrobios/i.ps1 | iex
|
||||
# Android (Termux: it installs Python and asks for storage access)
|
||||
curl -fsSL https://abdess.github.io/retrobios/i | sh
|
||||
|
||||
# SD card mounted on a PC: a muOS, KNULLI, ROCKNIX, Onion, spruceOS or MinUI card
|
||||
# SD card mounted on a PC: a muOS, KNULLI, ROCKNIX, Lakka, MiSTer, Onion, spruceOS or MinUI card
|
||||
# is found by itself. For another card, --dest is its BIOS folder
|
||||
# (RetroArch on Switch: retroarch/cores/system, on Android: RetroArch/system)
|
||||
curl -fsSL https://abdess.github.io/retrobios/i | sh -s -- --platform retroarch --dest /path/to/sdcard/retroarch/cores/system
|
||||
@@ -39,6 +39,8 @@ curl -fsSL https://abdess.github.io/retrobios/i | sh -s -- --platform retroarch
|
||||
The script opens a full-screen installer, driven with the arrow keys and Enter, or with a controller on a handheld. It finds your setup and its BIOS folder, downloads only the missing files and checks each one. It can also copy the BIOS files from your old SD card.
|
||||
Every option, and what the installer detects and writes, is on the [installer page](https://abdess.github.io/retrobios/wiki/installer/#passing-options).
|
||||
|
||||
First time? Follow the [first install, step by step](https://abdess.github.io/retrobios/wiki/first-install/). A game does not start: [what to check](https://abdess.github.io/retrobios/wiki/troubleshooting/). Other questions: the [FAQ](https://abdess.github.io/retrobios/wiki/faq/).
|
||||
|
||||
## Download BIOS packs
|
||||
|
||||
A pack holds everything its platform runs: its own BIOS list plus every file its emulator cores load. Pick your platform, download the ZIP, extract to the BIOS path. Each Download link is a page that keeps its address from one release to the next. The installer above does the same file by file: its screen installs everything, or fewer under **Choose systems**, and `--system`, `--core`, `--region` and `--target` make the same choice without questions. For a bare minimum, build your own pack below.
|
||||
@@ -72,7 +74,7 @@ The Files column counts what a pack holds once extracted, the figure a file mana
|
||||
|
||||
ZIPs are made for the setups most people download. The others read "Use the command": the installer above installs every file for them and checks each one, and `--dry-run` shows what it would install ([why no ZIP](https://abdess.github.io/retrobios/wiki/faq/#why-is-there-no-zip-for-my-setup)).
|
||||
|
||||
\* Archived: the configuration is kept and the installer serves it, but upstream is no longer scraped on a schedule.
|
||||
\* Archived: the installer serves it, and checks that cover every setup include it only on request.
|
||||
|
||||
## What's included
|
||||
|
||||
@@ -206,4 +208,4 @@ The scripts and tooling are released under the [MIT License](LICENSE).
|
||||
The BIOS and firmware files are not covered by that license: they are third-party system software, preserved and provided for personal backup, archival, and interoperability with emulation software. [NOTICE](NOTICE) sets out their status and how to ask for a file to be removed.
|
||||
The reasoning, and where it is weakest, is in [the project's legal position](https://abdess.github.io/retrobios/wiki/legal/).
|
||||
|
||||
*Auto-generated on 2026-11-06T04:40:27Z*
|
||||
*Auto-generated on 2026-11-06T05:38:30Z*
|
||||
+3
-2
@@ -87,8 +87,9 @@ jobs:
|
||||
body: >-
|
||||
Regenerated by the offline pipeline, database.json and the
|
||||
manifests computed twice with the same result.
|
||||
paths: [database.json, install/*.json, install/targets/*.json, install/games/*.json,
|
||||
install/games/*/*.json, README.md, CONTRIBUTING.md, website/src/data/*.json]
|
||||
paths: [database.json, derivable.json, retired.json, install/*.json, install/targets/*.json,
|
||||
install/games/*.json, install/games/*/*.json, README.md, CONTRIBUTING.md,
|
||||
wiki/installer.md, website/src/data/*.json]
|
||||
scratch: [dist, tmp/autorun]
|
||||
|
||||
bios-integrity:
|
||||
|
||||
@@ -452,6 +452,9 @@ guides:
|
||||
Turn on **SSH**, a terminal over the network, in Lakka's **Settings > Services**. From
|
||||
a computer, connect with `ssh root@lakka`, password `root`. Paste the line. Press
|
||||
Enter.
|
||||
# /storage is the card's ext4 partition, the other one FAT32 (Lakka-LibreELEC
|
||||
# 4696b48f, scripts/mkimage:71-72 and 95): from a computer, only Linux opens it.
|
||||
card_run: [linux]
|
||||
reference_notes:
|
||||
- The folder is reachable over SSH and over the network share.
|
||||
pack_manual: |
|
||||
@@ -648,8 +651,7 @@ guides:
|
||||
RetroPie also installs standalone emulators (openMSX, PCSX2, XRoar, Amiberry and
|
||||
others) whose files only the RetroPie file list carries.
|
||||
- >-
|
||||
RetroPie is archived in this project: its configuration is kept and the command
|
||||
serves it, but its upstream list is no longer read again on a schedule.
|
||||
RetroPie is archived here: checks of every setup include it only on request.
|
||||
pack_manual: |
|
||||
Option B: Via network share
|
||||
--------------------------
|
||||
|
||||
+37427
File diff suppressed because it is too large.
Load diff
@@ -389,11 +389,27 @@
|
||||
"layer": {
|
||||
"enum": [
|
||||
"collection",
|
||||
"archive-member",
|
||||
"retired",
|
||||
"data-directory",
|
||||
"platform",
|
||||
"emulator"
|
||||
]
|
||||
},
|
||||
"left": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$",
|
||||
"description": "On a retired record: the day the file, whose SHA1 is `sha1`, left the collection (retired.json)."
|
||||
},
|
||||
"successor": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9a-f]{40}$",
|
||||
"description": "On a retired record: SHA1 of the collection file now holding its place, whose card `url` leads to."
|
||||
},
|
||||
"archive": {
|
||||
"type": "string",
|
||||
"description": "On an archive-member record: the name of the collection archive holding the member, whose SHA1 is `sha1`. Its value is the member's CRC32, read from the archive's central directory."
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
|
||||
@@ -80,6 +80,9 @@ TOSEC_DOWNLOADS = "https://www.tosecdev.org/downloads"
|
||||
MAME_REPO = "https://github.com/mamedev/mame"
|
||||
FBNEO_REPO = "https://github.com/libretro/FBNeo"
|
||||
PYPI_URL = "https://pypi.org/pypi/{name}/json"
|
||||
NPM_URL = "https://registry.npmjs.org/{name}/latest"
|
||||
# The site's npm dependencies, pinned to exact versions and built by Deploy Site.
|
||||
SITE_PACKAGE = REPO_ROOT / "website" / "package.json"
|
||||
INSTALLERS = ("install.sh", "install.ps1")
|
||||
|
||||
SHOWN_ITEMS = 4 # items named in a diff summary before ", ..."
|
||||
@@ -1356,6 +1359,53 @@ def pypi_latest(name: str) -> str | None:
|
||||
return str(info.get("version") or "") or None
|
||||
|
||||
|
||||
def parse_npm_pins(text: str) -> dict[str, str]:
|
||||
"""Package -> the version a package.json names, dependencies of both kinds."""
|
||||
data = json.loads(text)
|
||||
pins: dict[str, str] = {}
|
||||
for key in ("dependencies", "devDependencies"):
|
||||
pins.update({str(k): str(v) for k, v in (data.get(key) or {}).items()})
|
||||
return pins
|
||||
|
||||
|
||||
def npm_latest(name: str) -> str | None:
|
||||
payload = upstream._http_json(NPM_URL.format(name=name)) # noqa: SLF001
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
return str(payload.get("version") or "") or None
|
||||
|
||||
|
||||
def _npm_findings() -> list[Finding]:
|
||||
"""One row per site dependency: an exact pin is stale once npm's latest
|
||||
release is newer, since nothing else asks npm."""
|
||||
findings = []
|
||||
for name, pinned in sorted(parse_npm_pins(SITE_PACKAGE.read_text(encoding="utf-8")).items()):
|
||||
subject = f"npm:{name}"
|
||||
if not re.fullmatch(r"\d+(\.\d+)*", pinned):
|
||||
findings.append(Finding("ci", subject, UNKNOWN, local=pinned, detail="a range, not an exact pin"))
|
||||
continue
|
||||
try:
|
||||
latest = npm_latest(name)
|
||||
except upstream.UpstreamError as exc:
|
||||
findings.append(Finding("ci", subject, UNKNOWN, local=pinned, detail=str(exc)))
|
||||
continue
|
||||
if latest is None:
|
||||
findings.append(Finding("ci", subject, UNKNOWN, local=pinned, detail="not on npm"))
|
||||
continue
|
||||
newer = _version_tuple(latest) > _version_tuple(pinned)
|
||||
findings.append(
|
||||
Finding(
|
||||
"ci",
|
||||
subject,
|
||||
STALE if newer else OK,
|
||||
local=pinned,
|
||||
remote=latest,
|
||||
detail="a newer release is out" if newer else "",
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
|
||||
def check_ci(cache_dir: str, offline: bool) -> list[Finding]:
|
||||
findings = [_check_installer_pins()]
|
||||
workflows = sorted((REPO_ROOT / ".github" / "workflows").glob("*.yml"))
|
||||
@@ -1367,8 +1417,9 @@ def check_ci(cache_dir: str, offline: bool) -> list[Finding]:
|
||||
pins.setdefault(name, spec)
|
||||
actions.update(parse_action_pins(text))
|
||||
if offline:
|
||||
findings.append(Finding("ci", "workflow pins", SKIPPED, detail="offline"))
|
||||
findings.append(Finding("ci", "workflow and site pins", SKIPPED, detail="offline"))
|
||||
return findings
|
||||
findings.extend(_npm_findings())
|
||||
for name, spec in sorted(pins.items()):
|
||||
try:
|
||||
latest = pypi_latest(name)
|
||||
|
||||
@@ -35,6 +35,7 @@ from common import (
|
||||
write_text_atomic,
|
||||
yaml_load,
|
||||
)
|
||||
import retired_files
|
||||
|
||||
# Platform and profile aliases are read from YAML: without pyyaml the
|
||||
# database lost every one of them and the run still exited 0.
|
||||
@@ -489,6 +490,19 @@ def _generate(args: argparse.Namespace) -> int:
|
||||
"indexes": indexes,
|
||||
}
|
||||
|
||||
# A sha1 the rebuild drops keeps its published address: the record
|
||||
# beside the database says when it left and what holds its place.
|
||||
output = Path(args.output)
|
||||
if output.is_file():
|
||||
previous = json.loads(output.read_text(encoding="utf-8"))
|
||||
left = retired_files.update(
|
||||
retired_files.retired_path(output), previous, database,
|
||||
database["generated_at"][:10],
|
||||
)
|
||||
if left:
|
||||
print(f" Left the collection: {left} files, recorded in "
|
||||
f"{retired_files.retired_path(output)}")
|
||||
|
||||
new_content = json.dumps(database, indent=2)
|
||||
written = write_if_changed(args.output, new_content)
|
||||
|
||||
|
||||
@@ -3672,9 +3672,9 @@ def _main(held: contextlib.ExitStack) -> None:
|
||||
# Platform mode
|
||||
if args.all and args.manifest:
|
||||
# Install manifests cover every registered platform: the installer
|
||||
# serves each one, with or without a release pack. "Archived" means
|
||||
# upstream is no longer scraped on a schedule, not that the platform
|
||||
# lost its users.
|
||||
# serves each one, with or without a release pack. "Archived" keeps a
|
||||
# platform out of the runs that cover every platform, not out of the
|
||||
# installer: it has not lost its users.
|
||||
platforms = list_registered_platforms(args.platforms_dir, include_archived=True)
|
||||
elif args.all:
|
||||
# A release publishes the packs the registry marks release_pack and
|
||||
|
||||
@@ -588,7 +588,7 @@ def generate_readme(db: dict, platforms_dir: str, db_path: str = "database.json"
|
||||
"# Android (Termux: it installs Python and asks for storage access)",
|
||||
install_command("sh"),
|
||||
"",
|
||||
"# SD card mounted on a PC: a muOS, KNULLI, ROCKNIX, Onion, spruceOS or MinUI card",
|
||||
"# SD card mounted on a PC: a muOS, KNULLI, ROCKNIX, Lakka, MiSTer, Onion, spruceOS or MinUI card",
|
||||
"# is found by itself. For another card, --dest is its BIOS folder",
|
||||
"# (RetroArch on Switch: retroarch/cores/system, on Android: RetroArch/system)",
|
||||
install_command("sh", "--platform retroarch --dest /path/to/sdcard/retroarch/cores/system"),
|
||||
@@ -602,6 +602,10 @@ def generate_readme(db: dict, platforms_dir: str, db_path: str = "database.json"
|
||||
" files and checks each one. It can also copy the BIOS files from your old SD card.",
|
||||
f"Every option, and what the installer detects and writes, is on the [installer page]({SITE_URL}wiki/installer/#passing-options).",
|
||||
"",
|
||||
f"First time? Follow the [first install, step by step]({SITE_URL}wiki/first-install/)."
|
||||
f" A game does not start: [what to check]({SITE_URL}wiki/troubleshooting/)."
|
||||
f" Other questions: the [FAQ]({SITE_URL}wiki/faq/).",
|
||||
"",
|
||||
"## Download BIOS packs",
|
||||
"",
|
||||
"A pack holds everything its platform runs: its own BIOS list plus every file its emulator cores load. Pick your platform, download the ZIP, extract to the BIOS path. Each Download link is a page that keeps its address from one release to the next. The installer above does the same file by file: its screen installs everything, or fewer under **" + _installer_words().TUI_SYSTEMS_ROW + "**, and `--system`, `--core`, `--region` and `--target` make the same choice without questions. For a bare minimum, build your own pack below.",
|
||||
@@ -647,8 +651,8 @@ def generate_readme(db: dict, platforms_dir: str, db_path: str = "database.json"
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"\\* Archived: the configuration is kept and the installer serves it,"
|
||||
" but upstream is no longer scraped on a schedule.",
|
||||
"\\* Archived: the installer serves it, and checks that cover every"
|
||||
" setup include it only on request.",
|
||||
]
|
||||
)
|
||||
|
||||
@@ -986,7 +990,7 @@ the checks CI runs on a pull request (Python 3.12 or newer):
|
||||
pip install pyyaml "jsonschema[format-nongpl]" markdown pyte hypothesis vermin setuptools
|
||||
python scripts/validate_pr.py --changed # the BIOS files the branch adds
|
||||
python scripts/validate_schemas.py --source-only # profiles and platform files
|
||||
python scripts/profile_sync.py --added --base upstream/main --strict # each profile the branch adds
|
||||
python scripts/profile_sync.py --added --strict # each profile the branch adds
|
||||
python scripts/refresh_data_dirs.py # data directories the pack-building tests read
|
||||
python scripts/generate_site.py # the pages the site tests read
|
||||
npm ci --prefix website && npm run build --prefix website # the built site, Node 24
|
||||
@@ -995,10 +999,9 @@ python scripts/validate_site.py --site-dir website/dist # links and page budget
|
||||
vermin --no-tips -t=3.8- --violations install.py # install.py stays on Python 3.8
|
||||
```
|
||||
|
||||
`upstream` is this repository as a remote of your fork
|
||||
(`git remote add upstream https://github.com/Abdess/retrobios`, then
|
||||
`git fetch upstream`). A profile you only edit is not held to `--strict`: its
|
||||
older refs are reviewed by hand.
|
||||
`profile_sync.py` compares the branch with this repository's `main`, and
|
||||
fetches it in a fork's clone. A profile you only edit is not held to
|
||||
`--strict`: its older refs are reviewed by hand.
|
||||
|
||||
The full pipeline is not needed for a contribution: it rebuilds every pack
|
||||
in `dist/`, which is why maintainers run it before a release.
|
||||
|
||||
+199
-40
@@ -47,6 +47,7 @@ from common import (
|
||||
GAME_DATA_TOPS,
|
||||
compute_composition,
|
||||
count_catalog_matched,
|
||||
entry_system_ids,
|
||||
list_registered_platforms,
|
||||
list_release_pack_platforms,
|
||||
load_platform_registry,
|
||||
@@ -88,6 +89,7 @@ from generate_readme import (
|
||||
from provenance_report import build_report, distinct_missing
|
||||
import sitemarkup as markup
|
||||
import siteentries
|
||||
import retired_files
|
||||
import sitestarlight
|
||||
from siteexports import generate_exports_page, load_export_reports
|
||||
from siteguides import (
|
||||
@@ -1622,7 +1624,8 @@ def generate_data_exports(
|
||||
"file-locations.json": ("file-locations", locations),
|
||||
}
|
||||
write_lookup_shards(
|
||||
api, file_lookup_records(db, coverages, profiles, locations, members, consumers),
|
||||
api, file_lookup_records(db, coverages, profiles, locations, members, consumers,
|
||||
retired_files.load(_REPO_ROOT / "retired.json")),
|
||||
generated_at, commit,
|
||||
)
|
||||
for filename, (kind, items) in envelopes.items():
|
||||
@@ -1808,7 +1811,9 @@ def generate_data_exports(
|
||||
# entry point is a document of its own, naming where to start.
|
||||
entry_points = [
|
||||
{"title": "Catalog", "url": "catalog.json",
|
||||
"description": "Every published file with its size, SHA256 and schema."},
|
||||
"description": "Each dataset, manifest, export and download with its size, "
|
||||
"SHA256 and schema. The one-item documents of emulators/, "
|
||||
"platforms/, systems/ and lookup/ are not listed."},
|
||||
{"title": "System ids", "url": "systems.json",
|
||||
"description": "Each system id with its name and the section holding its files."},
|
||||
{"title": "Core names", "url": "cores.json",
|
||||
@@ -2203,11 +2208,15 @@ def generate_data_page(exports: list[dict], measurements_sqlite: Path | None = N
|
||||
"One hash without the whole list: `api/v1/lookup/<xx>.json`, where `<xx>` "
|
||||
"is the first two characters of the hash in lowercase, holds every SHA1, "
|
||||
"MD5, CRC32 and SHA256 starting with them. Each record names its `layer` "
|
||||
"(the collection, a data directory, a platform's list or an emulator "
|
||||
"profile), the file `name`, the `url` of the page that shows it and, "
|
||||
"(the collection, a member of a collection archive, a file that left the "
|
||||
"collection, a data directory, a platform's list or an emulator profile), "
|
||||
"the file `name`, the `url` of the page that shows it and, "
|
||||
"when one is known, the `system_name` the site gives its system. "
|
||||
"[Find or check a file](files/index.md) reads it.",
|
||||
"",
|
||||
"`catalog.json` lists the datasets, manifests, exports and downloads. It "
|
||||
"does not list these one-item documents, which hold no SHA256 of their own.",
|
||||
"",
|
||||
"In a browser, the SHA1 of a file the reader picks finds its records when "
|
||||
"the collection holds that file. A list naming a file the collection lacks "
|
||||
"is filed under its MD5 or CRC32 alone, and Web Crypto has no MD5: "
|
||||
@@ -2225,7 +2234,7 @@ def generate_data_page(exports: list[dict], measurements_sqlite: Path | None = N
|
||||
" const response = await fetch(`${base}${sha1.slice(0, 2)}.json`);",
|
||||
" if (!response.ok) return [];",
|
||||
" const shard = await response.json();",
|
||||
" // layer: collection, data-directory, platform or emulator. owner: which one.",
|
||||
" // layer: collection, archive-member, retired, data-directory, platform, emulator.",
|
||||
" return shard.items.filter((item) => item.value === sha1);",
|
||||
"}",
|
||||
"```",
|
||||
@@ -3350,8 +3359,8 @@ def setup_table(coverages: dict, registry: dict | None = None) -> list[str]:
|
||||
lines += ["No ZIP: the setup is served by the command alone. "
|
||||
f"[Why?](../{NO_ZIP_FAQ})", ""]
|
||||
if archived:
|
||||
lines += ["An archived setup keeps its page and the command. Its list is no longer "
|
||||
"read again on a schedule.", ""]
|
||||
lines += ["An archived setup keeps its page and the command. Checks that cover "
|
||||
"every setup include it only on request.", ""]
|
||||
return lines
|
||||
|
||||
|
||||
@@ -4187,7 +4196,11 @@ def _card_route(name: str, display: str, guide: Guide) -> list[str]:
|
||||
lead = ("No network here? Run the command on a computer that has one, with a USB "
|
||||
f"drive's folder after `--dest`. Then copy it into {_folder_words(display, extract_to)}.")
|
||||
return [lead, "", *terminal_tabs(tabs, args, steps=False)]
|
||||
lead = ("No network? Put the device's card in your computer. After `--dest`, "
|
||||
# A folder on the card's Linux partition: Windows and macOS show the
|
||||
# small boot partition alone, and no folder there fits --dest.
|
||||
put = ("Put the device's card in your computer." if "windows" in tabs
|
||||
else "Only a Linux computer opens the card.")
|
||||
lead = (f"No network? {put} After `--dest`, "
|
||||
f"name the folder the device calls `{extract_to}`.")
|
||||
return [lead, "", *terminal_tabs(tabs, args, steps=False),
|
||||
f"[More on SD cards](../{CARD_ROUTE_FAQ})", ""]
|
||||
@@ -4729,7 +4742,9 @@ def generate_files_page() -> str:
|
||||
"A file's card lives on its manufacturer's page, and a file refiled "
|
||||
"between manufacturers moves with it. `files/?sha1=<sha1>` is the address "
|
||||
"that does not move: it opens the card wherever it is now. The link icon "
|
||||
"beside each card's name is that address.",
|
||||
"beside each card's name is that address. A file that left the "
|
||||
"collection keeps it: the page says when, and opens the card of the "
|
||||
"file in its place.",
|
||||
"",
|
||||
"## The index behind it",
|
||||
"",
|
||||
@@ -4763,6 +4778,33 @@ def data_directory_revisions(data_dir: Path = _REPO_ROOT / "data") -> dict[str,
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
_GIT_COMMIT = re.compile(r"[0-9a-f]{40}")
|
||||
_GITHUB_REPO = re.compile(r"https://github\.com/([^/]+/[^/]+)/")
|
||||
|
||||
|
||||
def data_directory_origin(entry: dict, fetched: dict) -> list[str]:
|
||||
"""Where a data directory is fetched from, and the commit the packs hold.
|
||||
|
||||
The URL is written with the `{version}` refresh_data_dirs substitutes:
|
||||
printed raw it led to a 404. A zip source records the server's ETag,
|
||||
which names no revision anyone can fetch, so only a git commit is shown.
|
||||
"""
|
||||
lines = []
|
||||
url = str(entry.get("source_url") or "")
|
||||
if url:
|
||||
url = url.replace("{version}", str(entry.get("version", "master")))
|
||||
lines.append(f"Fetched from <{url}>")
|
||||
sha = str(fetched.get("sha", "")).strip('"')
|
||||
if _GIT_COMMIT.fullmatch(sha):
|
||||
repo = _GITHUB_REPO.match(url)
|
||||
commit = f"[{_code(sha[:12])}](https://github.com/{repo.group(1)}/commit/{sha})" \
|
||||
if repo else _code(sha)
|
||||
branch = str(fetched.get("version", ""))
|
||||
lines.append(f"Revision: {commit}" + (
|
||||
f", the head of `{branch}` when fetched" if branch and branch != sha else ""))
|
||||
return lines
|
||||
|
||||
|
||||
def page_data_directories(name: str, profiles: dict) -> list[dict]:
|
||||
"""The data directories an emulator page lists.
|
||||
|
||||
@@ -4892,6 +4934,64 @@ def _data_member_digests(
|
||||
return found
|
||||
|
||||
|
||||
def retired_records(retired: dict[str, dict], files: dict, locations: list[dict]) -> list[dict]:
|
||||
"""A record for each hash of a file that left the collection.
|
||||
|
||||
Its address (files/?sha1=, #sha1-<12>) was published and answered that
|
||||
no file the site knows has the hash, inviting a contribution of a file
|
||||
the collection held. The record says when it left, and leads to the
|
||||
card of the file holding its place, followed to the one held today.
|
||||
"""
|
||||
cards = {item["sha1"]: item["url"] for item in locations}
|
||||
records = []
|
||||
for sha1, record in sorted(retired.items()):
|
||||
if sha1 in files:
|
||||
continue
|
||||
successor, seen = record.get("successor"), {sha1}
|
||||
while successor and successor not in files and successor not in seen:
|
||||
seen.add(successor)
|
||||
successor = (retired.get(successor) or {}).get("successor")
|
||||
url = cards.get(successor) if successor in files else None
|
||||
extra = {"left": str(record["left"])}
|
||||
if url:
|
||||
extra["successor"] = successor
|
||||
else:
|
||||
# The page its card was on: the reader lands among its neighbours.
|
||||
url = _site_path(f"systems/{card_page(str(record.get('path', '')), sha1)}.md")
|
||||
for algorithm in HASH_ALGORITHMS:
|
||||
value = sha1 if algorithm == "sha1" else record.get(algorithm)
|
||||
for one in _hash_values(value):
|
||||
records.append({"value": one, "algorithm": algorithm, "layer": "retired",
|
||||
"name": str(record.get("name", "")), "sha1": sha1,
|
||||
"url": url, **extra})
|
||||
return records
|
||||
|
||||
|
||||
def _archive_member_records(entry: dict, sha1: str, url: str,
|
||||
root: Path = _REPO_ROOT) -> list[dict]:
|
||||
"""The CRC32 of each member of a collection archive, read from its
|
||||
central directory.
|
||||
|
||||
A pasted CRC of a ROM the collection holds only inside a set (logica2.u2
|
||||
in a500.zip, 8484f426) found nothing: 26,014 of 33,403 members were in
|
||||
no record. The record leads to the archive's card and names it.
|
||||
"""
|
||||
path = root / str(entry.get("path", ""))
|
||||
if path.suffix.lower() != ".zip" or not path.is_file():
|
||||
return []
|
||||
try:
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
members = [(info.filename, info.CRC) for info in archive.infolist()
|
||||
if not info.is_dir()]
|
||||
except (OSError, zipfile.BadZipFile) as exc:
|
||||
print(f" WARNING: {path}: members unreadable ({exc})", file=sys.stderr)
|
||||
return []
|
||||
return [{"value": f"{crc:08x}", "algorithm": "crc32", "layer": "archive-member",
|
||||
"name": member.rsplit("/", 1)[-1], "archive": str(entry.get("name", "")),
|
||||
"sha1": sha1, "url": url}
|
||||
for member, crc in members]
|
||||
|
||||
|
||||
def file_lookup_records(
|
||||
db: dict,
|
||||
coverages: dict,
|
||||
@@ -4899,6 +4999,7 @@ def file_lookup_records(
|
||||
locations: list[dict],
|
||||
members: dict | None = None,
|
||||
consumers: FileConsumers | None = None,
|
||||
retired: dict | None = None,
|
||||
) -> list[dict]:
|
||||
"""Every hash the site knows, one record per value, for files/?hash=.
|
||||
|
||||
@@ -4936,6 +5037,7 @@ def file_lookup_records(
|
||||
"name": str(entry.get("name", "")), "sha1": sha1, "url": item["url"],
|
||||
**({"system_name": held_by} if held_by else {}),
|
||||
})
|
||||
records += _archive_member_records(entry, sha1, item["url"])
|
||||
for key, coverage in sorted(coverages.items()):
|
||||
for system_id, system in sorted((coverage["config"].get("systems") or {}).items()):
|
||||
url = _site_path(f"platforms/{key}.md#{system_id.replace(' ', '-')}")
|
||||
@@ -4975,6 +5077,7 @@ def file_lookup_records(
|
||||
"owner": key, "name": name, "url": url,
|
||||
**({"system_name": entry_label} if entry_label else {}),
|
||||
})
|
||||
records += retired_records(retired or {}, files, locations)
|
||||
records += _held_under_every_hash(records, files, db.get("indexes") or {})
|
||||
for record in records:
|
||||
if record["layer"] in ("platform", "emulator"):
|
||||
@@ -5999,7 +6102,9 @@ def _system_file_card(
|
||||
name = f["name"]
|
||||
sha1_full = f.get("sha1", "unknown")
|
||||
md5_full = f.get("md5", "unknown")
|
||||
size = _fmt_size(f.get("size", 0))
|
||||
# The size a card names is the file's own, exact: rounded, 44,348,015
|
||||
# and 44,356,548 bytes were one 42.3 MB.
|
||||
size = _card_size(int(f.get("size", 0)))
|
||||
# Who reads this file, not who reads a file of this name
|
||||
plats = sorted(consumers.platforms.get(sha1_full, ()))
|
||||
emus = sorted(consumers.emulators.get(sha1_full, ()))
|
||||
@@ -6254,7 +6359,7 @@ def search_excluded(lines: list[str]) -> list[str]:
|
||||
SEARCH_SKIPPED = (
|
||||
"- SHA1:", "- MD5:", "- CRC32:", "- SHA256:", "- Adler32", "- Size:",
|
||||
"- Validation", "- Priority:", "- Contents (",
|
||||
"- Platforms:", "- Also installed by:", "- Source:", "- Asked for by:",
|
||||
"- Platforms:", "- Source:", "- Asked for by:",
|
||||
"- Collection path:", "- Collection copy:", "- Collection copies:",
|
||||
"- Installed by the platform's own package on:",
|
||||
"- Not in the platform's package", "- Never opened on these platforms",
|
||||
@@ -6505,9 +6610,10 @@ def _profiles_on_disk() -> dict[str, dict]:
|
||||
|
||||
|
||||
def _entry_systems(emulator: str, entry: dict) -> list[str]:
|
||||
"""The system an entry is read for: its own, or its profile's only one."""
|
||||
if entry.get("system"):
|
||||
return [str(entry["system"])]
|
||||
"""The systems an entry is read for: its own `system:` or `systems:`
|
||||
(a MAME device set several systems load), else its profile's only one."""
|
||||
if entry.get("system") or entry.get("systems"):
|
||||
return entry_system_ids(entry, {})
|
||||
systems = (_profiles_on_disk().get(emulator) or {}).get("systems") or []
|
||||
return [str(systems[0])] if len(systems) == 1 else []
|
||||
|
||||
@@ -7719,20 +7825,17 @@ def _distinct_spellings(names: list) -> list[str]:
|
||||
return list(seen.values())
|
||||
|
||||
|
||||
# Region slugs whose words are not the slug title-cased.
|
||||
_REGION_WORDS = {
|
||||
"uk": "UK",
|
||||
"asia-ntsc": "Asia (NTSC)",
|
||||
"asia-pal": "Asia (PAL)",
|
||||
"world": "every region",
|
||||
}
|
||||
# Region slugs whose words are not the slug title-cased, beyond region.LABELS.
|
||||
_REGION_WORDS = {"world": "every region"}
|
||||
|
||||
|
||||
def _region_words(value: object, most: int = 0) -> str:
|
||||
"""A region: value in words. ``most`` caps the regions named, the rest
|
||||
counted: one Saturn BIOS serves seven."""
|
||||
raw = value if isinstance(value, list) else [value] if value else []
|
||||
words = [_REGION_WORDS.get(str(r), str(r).replace("-", " ").title()) for r in raw]
|
||||
from region import label # noqa: PLC0415
|
||||
|
||||
words = [_REGION_WORDS.get(str(r)) or label(str(r)) for r in raw]
|
||||
if most and len(words) > most:
|
||||
words = words[:most] + [f"{len(words) - most} more"]
|
||||
return _said_as_list(words)
|
||||
@@ -8015,6 +8118,38 @@ def _on_mismatch_lines(value: object) -> list[str]:
|
||||
return []
|
||||
|
||||
|
||||
def emulator_setups(coverages: dict, profiles: dict) -> dict[str, list[str]]:
|
||||
"""Each emulator with the platforms that run it, by display name."""
|
||||
setups: dict[str, list[str]] = {}
|
||||
for platform, cov in coverages.items():
|
||||
for emulator in resolve_platform_cores(cov["config"], profiles):
|
||||
setups.setdefault(emulator, []).append(platform)
|
||||
return {
|
||||
emulator: sorted(names, key=lambda p: _platform_display(p).casefold())
|
||||
for emulator, names in setups.items()
|
||||
}
|
||||
|
||||
|
||||
def _validation_names(validation: object) -> set[str]:
|
||||
"""The checks a `validation:` names, a list or one list per build."""
|
||||
if isinstance(validation, dict):
|
||||
return {str(v) for checks in validation.values() for v in checks or []}
|
||||
return {str(v) for v in validation or []}
|
||||
|
||||
|
||||
def _card_size(size: int) -> str:
|
||||
"""A file's size, exact: in whole MB or KB when it is one, else in B."""
|
||||
for unit, scale in (("MB", 1024 * 1024), ("KB", 1024)):
|
||||
if size >= scale and size % scale == 0:
|
||||
return f"{size // scale:,} {unit}"
|
||||
return f"{size:,} B"
|
||||
|
||||
|
||||
def _exact_size(size: int) -> str:
|
||||
"""A rounded size with its bytes, where the bytes are not the same figure."""
|
||||
return _fmt_size(size) if size < 1024 else f"{_fmt_size(size)} ({size:,} bytes)"
|
||||
|
||||
|
||||
def _build_rows(profile: dict) -> list[str]:
|
||||
"""Meta card rows for who ships the build, how its files are found, why
|
||||
its revision is held and where its releases are announced."""
|
||||
@@ -8148,25 +8283,28 @@ def _render_emulator_file(
|
||||
details.append(f"Found by: {found}")
|
||||
if fsystem:
|
||||
details.append(f"System: {_named_system_link(fsystem, '../')}")
|
||||
# Where the code compares the size, the exact bytes: raze tells three
|
||||
# DUKE3D.GRP apart by size, all three shown as 42.3 MB.
|
||||
sized = _exact_size if "size" in _validation_names(validation) else _fmt_size
|
||||
if size:
|
||||
if isinstance(size, list):
|
||||
size_str = " / ".join(_fmt_size(s) for s in size)
|
||||
size_str = " / ".join(sized(s) for s in size)
|
||||
else:
|
||||
size_str = _fmt_size(size)
|
||||
size_str = sized(size)
|
||||
if fmin or fmax:
|
||||
bounds = []
|
||||
if fmin:
|
||||
bounds.append(f"min {_fmt_size(fmin)}")
|
||||
bounds.append(f"min {sized(fmin)}")
|
||||
if fmax:
|
||||
bounds.append(f"max {_fmt_size(fmax)}")
|
||||
bounds.append(f"max {sized(fmax)}")
|
||||
size_str += f" ({', '.join(bounds)})"
|
||||
details.append(f"Size: {size_str}")
|
||||
elif fmin or fmax:
|
||||
bounds = []
|
||||
if fmin:
|
||||
bounds.append(f"min {_fmt_size(fmin)}")
|
||||
bounds.append(f"min {sized(fmin)}")
|
||||
if fmax:
|
||||
bounds.append(f"max {_fmt_size(fmax)}")
|
||||
bounds.append(f"max {sized(fmax)}")
|
||||
details.append(f"Size: {', '.join(bounds)}")
|
||||
# Starlight folds the hashes and gives each a copy button.
|
||||
copyable = markup.starlight()
|
||||
@@ -9040,6 +9178,21 @@ def _install_args(platform: str, system: str, shell: str) -> str:
|
||||
return f"--platform {platform} --system {quoted}"
|
||||
|
||||
|
||||
def _default_setup(declared: list[tuple[str, str]]) -> tuple[str, str]:
|
||||
"""The setup an install box shows before the reader picks one.
|
||||
|
||||
The first in alphabetical order was AmberELEC on 17 system pages, a
|
||||
handheld system under a Windows tab. The box shows the setup whose
|
||||
command runs under the most of its tabs (the guide's `run`), then on the
|
||||
most systems overall, then one the release publishes a ZIP for.
|
||||
"""
|
||||
def reach(pair: tuple[str, str]) -> tuple[int, int, bool]:
|
||||
runs = GUIDES[pair[0]].run if pair[0] in GUIDES else ()
|
||||
return (len(set(runs) & set(INSTALL_PICKER_OSES)), len(runs), has_release_pack(pair[0]))
|
||||
|
||||
return max(declared, key=reach)
|
||||
|
||||
|
||||
def _install_picker(declared: list[tuple[str, str]]) -> list[str]:
|
||||
"""One command, for the setup the reader picks, under the OS tabs.
|
||||
|
||||
@@ -9054,15 +9207,16 @@ def _install_picker(declared: list[tuple[str, str]]) -> list[str]:
|
||||
return html.escape(text, quote=False).replace('"', """)
|
||||
|
||||
names = Counter(platform for platform, _system in declared)
|
||||
lead = _default_setup(declared)
|
||||
options = "".join(
|
||||
f'<option value="{attribute(platform)}" '
|
||||
f'data-sh="{attribute(_install_args(platform, system, "sh"))}" '
|
||||
f'<option value="{attribute(platform)}"'
|
||||
+ (" selected" if (platform, system) == lead else "")
|
||||
+ f' data-sh="{attribute(_install_args(platform, system, "sh"))}" '
|
||||
f'data-ps="{attribute(_install_args(platform, system, "powershell"))}">'
|
||||
+ html.escape(_platform_display(platform) + (f" ({system})" if names[platform] > 1 else ""),
|
||||
quote=False)
|
||||
+ "</option>"
|
||||
for platform, system in declared)
|
||||
lead = declared[0]
|
||||
args = {key: _install_args(*lead, TERMINALS[key][2]) for key in INSTALL_PICKER_OSES}
|
||||
return [
|
||||
"Each platform names the system its own way.",
|
||||
@@ -9379,6 +9533,8 @@ def _said_as_list(items: list[str]) -> str:
|
||||
# Data directory members and revisions, read once per build.
|
||||
_DATA_MEMBERS: dict[str, dict[str, dict]] = {}
|
||||
_DATA_REVISIONS: dict[str, dict] = {}
|
||||
# Emulator -> the platforms that run it, as resolve_platform_cores answers.
|
||||
_EMULATOR_SETUPS: dict[str, list[str]] = {}
|
||||
# The measurement index items of each emulator, for the proof lines of its page.
|
||||
_MEASUREMENT_ITEMS: dict[str, list[dict]] = {}
|
||||
# The emulator pages that list each data directory, by ref.
|
||||
@@ -9974,6 +10130,15 @@ def emulator_pages(
|
||||
if revision:
|
||||
lines.append(f"| Revision | {revision} |")
|
||||
lines.extend(_build_rows(profile))
|
||||
setups = _EMULATOR_SETUPS.get(name)
|
||||
if setups:
|
||||
# The file cards name the setups whose own list holds a file: the
|
||||
# ones that run the emulator and pack its files were said nowhere.
|
||||
# Said once here: a line on each card put MAME's parts and four
|
||||
# other pages over their element budget. Out of the search: setup
|
||||
# names on 530 pages ranked the data page first for "batocera folder".
|
||||
lines.append(f"| Run by | <span {markup.search_exclude()}>" + ", ".join(
|
||||
f"[{_platform_display(p)}](../platforms/{p}.md)" for p in setups) + "</span> |")
|
||||
lines.append(f"| Profiled | {profiled} |")
|
||||
# The YAML is the reusable document this site renders, and the JSON is the
|
||||
# same profile for a program.
|
||||
@@ -10078,15 +10243,8 @@ def emulator_pages(
|
||||
if entry.get("description"):
|
||||
line += f": {entry['description']}"
|
||||
lines.append(line)
|
||||
if entry.get("source_url"):
|
||||
lines.append(f" - Fetched from <{entry['source_url']}>")
|
||||
fetched = _DATA_REVISIONS.get(ref) or {}
|
||||
if fetched.get("version") or fetched.get("sha"):
|
||||
# The URL serves whatever is current: the revision says
|
||||
# which content the packs were built from.
|
||||
revision = str(fetched.get("sha", "")).strip('"')
|
||||
lines.append(f" - Revision: {fetched.get('version', '')}"
|
||||
+ (f" ({_code(revision)})" if revision else ""))
|
||||
lines.extend(f" - {line}" for line in data_directory_origin(
|
||||
entry, _DATA_REVISIONS.get(ref) or {}))
|
||||
if dd.get("mode"):
|
||||
lines.append(f" - Read by the {dd['mode']} build only")
|
||||
if dd.get("bundled"):
|
||||
@@ -12311,6 +12469,7 @@ def _generate(args: argparse.Namespace) -> None:
|
||||
PAGE_DATA[f"platforms/{name}.md"] = platform_structured_data(name, cov)
|
||||
|
||||
print("Generating emulator pages...")
|
||||
_EMULATOR_SETUPS.update(emulator_setups(coverages, profiles))
|
||||
_DATA_MEMBERS.update(data_directory_members())
|
||||
_DATA_REVISIONS.update(data_directory_revisions())
|
||||
for key in sorted(profiles):
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
#!/usr/bin/env python3
|
||||
"""List available platforms for CI matrix strategy.
|
||||
"""List the registered platforms.
|
||||
|
||||
Respects the `status` field in _registry.yml:
|
||||
- active: included in CI releases and automated scraping
|
||||
- archived: excluded from CI, user can generate manually
|
||||
- active: listed by default
|
||||
- archived: listed with --include-archived only
|
||||
|
||||
Usage:
|
||||
python scripts/list_platforms.py # Active platforms (for CI)
|
||||
python scripts/list_platforms.py # active platforms
|
||||
python scripts/list_platforms.py --include-archived # archived ones too
|
||||
python scripts/list_platforms.py >> "$GITHUB_OUTPUT"
|
||||
"""
|
||||
|
||||
@@ -5,6 +5,7 @@ goes, and which ones the user has to rename."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from region import label as region_label
|
||||
from siteguides import INSTALL_PS1, INSTALL_SH, pack_manual
|
||||
|
||||
def _detect_systems(detect: list[dict]) -> list[str]:
|
||||
@@ -290,9 +291,7 @@ def _build_readme(
|
||||
|
||||
region_info = ""
|
||||
if regions:
|
||||
pretty = ", ".join(
|
||||
" ".join(w.title() for w in slug.split("-")) for slug in regions
|
||||
)
|
||||
pretty = ", ".join(region_label(slug) for slug in regions)
|
||||
region_info = (
|
||||
"PACK TYPE: Region Filtered\n\n"
|
||||
f" Region priority: {pretty}\n\n"
|
||||
|
||||
@@ -107,6 +107,19 @@ ALIASES: dict[str, str] = {
|
||||
}
|
||||
|
||||
|
||||
# Slugs a person does not write title-cased: "Asia Ntsc" read as machine text.
|
||||
LABELS: dict[str, str] = {
|
||||
"uk": "UK",
|
||||
"asia-ntsc": "Asia (NTSC)",
|
||||
"asia-pal": "Asia (PAL)",
|
||||
}
|
||||
|
||||
|
||||
def label(slug: str) -> str:
|
||||
"""A region slug in words: north-america as North America."""
|
||||
return LABELS.get(slug) or " ".join(word.title() for word in slug.split("-"))
|
||||
|
||||
|
||||
def unknown_region_message(raw: str, aliases: dict[str, str], regions: frozenset[str]) -> str:
|
||||
"""What to type instead: the closest spellings, then every short form.
|
||||
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Files that left the collection, and the file that took their place.
|
||||
|
||||
A file's content address (files/?sha1=, #sha1-<12> on its card) is
|
||||
published, and a published address never stops answering. When a rebuild
|
||||
of database.json drops a sha1, the file it named is recorded here: its
|
||||
name, its last path, its hashes, the day it left and the sha1 the
|
||||
collection now holds at that path, or under that name in that folder.
|
||||
The site's lookup reads the record and answers with it.
|
||||
|
||||
The record is append-only, written by generate_db.py beside the database.
|
||||
A sha1 that comes back leaves it. `--backfill` rebuilds it from the
|
||||
history of database.json since content addresses were published.
|
||||
|
||||
Usage:
|
||||
python scripts/retired_files.py --backfill
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from common import run_cli, write_if_changed # noqa: E402
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
RETIRED_FILE = REPO_ROOT / "retired.json"
|
||||
# The commit that published content addresses: 0d586ef6b, 2026-10-19.
|
||||
FIRST_ADDRESSED = "0d586ef6b"
|
||||
KEPT_FIELDS = ("name", "path", "size", "md5", "crc32", "sha256")
|
||||
|
||||
|
||||
def retired_path(database: str | Path) -> Path:
|
||||
"""The record kept beside a database file."""
|
||||
return Path(database).with_name("retired.json")
|
||||
|
||||
|
||||
def _successor(entry: dict, by_path: dict[str, str],
|
||||
by_name: dict[str, list[tuple[str, str]]]) -> str | None:
|
||||
"""The sha1 the new database holds at the entry's path, else under its
|
||||
name in the same folder, else the one file of that name: a2diskiing.zip
|
||||
was rebuilt from its recipe and filed under Apple II the same day."""
|
||||
path = str(entry.get("path", ""))
|
||||
if path in by_path:
|
||||
return by_path[path]
|
||||
named = by_name.get(str(entry.get("name", "")), [])
|
||||
folder = path.rsplit("/", 1)[0]
|
||||
beside = [sha1 for sha1, other in named if other.rsplit("/", 1)[0] == folder]
|
||||
for found in (beside, [sha1 for sha1, _other in named]):
|
||||
if len(found) == 1:
|
||||
return found[0]
|
||||
return None
|
||||
|
||||
|
||||
def retired_between(old: dict, new: dict, day: str) -> dict[str, dict]:
|
||||
"""sha1 -> record, for each file of *old* that *new* no longer holds."""
|
||||
old_files = old.get("files") or {}
|
||||
new_files = new.get("files") or {}
|
||||
by_path = {str(e.get("path", "")): sha1 for sha1, e in new_files.items()}
|
||||
by_name: dict[str, list[tuple[str, str]]] = {}
|
||||
for sha1, e in new_files.items():
|
||||
by_name.setdefault(str(e.get("name", "")), []).append((sha1, str(e.get("path", ""))))
|
||||
records = {}
|
||||
for sha1 in sorted(set(old_files) - set(new_files)):
|
||||
entry = old_files[sha1]
|
||||
record = {key: entry[key] for key in KEPT_FIELDS if entry.get(key) not in (None, "")}
|
||||
record["left"] = day
|
||||
successor = _successor(entry, by_path, by_name)
|
||||
if successor:
|
||||
record["successor"] = successor
|
||||
records[sha1] = record
|
||||
return records
|
||||
|
||||
|
||||
def merge(record: dict[str, dict], left: dict[str, dict], new: dict) -> dict[str, dict]:
|
||||
"""The record with *left* added, first departure kept, and every sha1
|
||||
the new database holds again taken out."""
|
||||
held = set(new.get("files") or {})
|
||||
merged = {sha1: entry for sha1, entry in record.items() if sha1 not in held}
|
||||
for sha1, entry in left.items():
|
||||
merged.setdefault(sha1, entry)
|
||||
return dict(sorted(merged.items()))
|
||||
|
||||
|
||||
def load(path: Path) -> dict[str, dict]:
|
||||
if not path.is_file():
|
||||
return {}
|
||||
return json.loads(path.read_text(encoding="utf-8")).get("files") or {}
|
||||
|
||||
|
||||
def dump(records: dict[str, dict]) -> str:
|
||||
return json.dumps({"schema_version": 1, "files": records}, indent=1, sort_keys=True) + "\n"
|
||||
|
||||
|
||||
def update(path: Path, old: dict, new: dict, day: str) -> int:
|
||||
"""Records what *new* dropped from *old*. Returns how many left now."""
|
||||
left = retired_between(old, new, day)
|
||||
record = load(path)
|
||||
merged = merge(record, left, new)
|
||||
if merged != record:
|
||||
write_if_changed(str(path), dump(merged))
|
||||
return len(set(left) - set(record))
|
||||
|
||||
|
||||
def _git(*args: str) -> str:
|
||||
return subprocess.run(["git", *args], cwd=REPO_ROOT, capture_output=True, text=True,
|
||||
check=True).stdout
|
||||
|
||||
|
||||
def backfill() -> dict[str, dict]:
|
||||
"""Every sha1 that left database.json in the commits since content
|
||||
addresses were published, with the day of the commit that dropped it."""
|
||||
revisions = [line.split() for line in _git(
|
||||
"log", "--reverse", "--format=%H %cs", f"{FIRST_ADDRESSED}^..HEAD", "--", "database.json",
|
||||
).splitlines() if line.strip()]
|
||||
records: dict[str, dict] = {}
|
||||
previous: dict | None = None
|
||||
for commit, day in revisions:
|
||||
current = json.loads(_git("show", f"{commit}:database.json"))
|
||||
if previous is not None:
|
||||
records = merge(records, retired_between(previous, current, day), current)
|
||||
previous = current
|
||||
return records
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(allow_abbrev=False, description=__doc__.split("\n\n")[0])
|
||||
parser.add_argument("--backfill", action="store_true",
|
||||
help="rebuild the record from the history of database.json")
|
||||
parser.add_argument("--output", default=str(RETIRED_FILE),
|
||||
help="the record to write (default: retired.json)")
|
||||
args = parser.parse_args()
|
||||
if not args.backfill:
|
||||
parser.print_help()
|
||||
return 2
|
||||
records = backfill()
|
||||
current = json.loads((REPO_ROOT / "database.json").read_text(encoding="utf-8"))
|
||||
records = merge(records, {}, current)
|
||||
write_if_changed(args.output, dump(records))
|
||||
print(f"{len(records)} files left the collection since {FIRST_ADDRESSED}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_cli(main)
|
||||
@@ -372,19 +372,21 @@ def _in_arcade(sha1: str, db: dict) -> bool:
|
||||
return sha1 in _arcade_suffix_sha1s(db)
|
||||
|
||||
|
||||
_suffix_cache: list[tuple[int, frozenset[str]] | None] = [None]
|
||||
_suffix_cache: list[tuple[dict, frozenset[str]] | None] = [None]
|
||||
|
||||
|
||||
def _arcade_suffix_sha1s(db: dict) -> frozenset[str]:
|
||||
# Holding the database itself, never id(db): a database freed and
|
||||
# replaced at the same address answered with the first one's set.
|
||||
cached = _suffix_cache[0]
|
||||
if cached is not None and cached[0] == id(db):
|
||||
if cached is not None and cached[0] is db:
|
||||
return cached[1]
|
||||
found = frozenset(
|
||||
sha1s[0]
|
||||
for tail, sha1s in db.get("indexes", {}).get("by_path_suffix", {}).items()
|
||||
if tail.startswith("Arcade/") and len(sha1s) == 1
|
||||
)
|
||||
_suffix_cache[0] = (id(db), found)
|
||||
_suffix_cache[0] = (db, found)
|
||||
return found
|
||||
|
||||
|
||||
|
||||
+8
-25
@@ -7,11 +7,14 @@ module existed nothing compared the two: the builder resolves the platform
|
||||
entry first and drops the profile entry on a bare filename match, so a profile
|
||||
that names the right file for a slot could never win and never even be heard.
|
||||
|
||||
The index is keyed by destination AND by name, the name carrying the union of
|
||||
what every same-named entry claims. Dolphin declares three IPL.bin separated
|
||||
only by their path, so a name-only key merges them, and a path-only key answers
|
||||
nothing when a candidate is known by name alone. Ambiguity resolves to the
|
||||
union and is never discarded, the same rule region.py already applies.
|
||||
Claims are keyed by destination, folded to one case only where the platform's
|
||||
filesystem folds it. Dolphin declares three IPL.bin separated only by their
|
||||
path, and each is its own slot. A profile claim resting on a bare filename
|
||||
proves nothing about content and contests no slot.
|
||||
|
||||
The pack builder, the install manifest and verify read slot_choices() through
|
||||
pack_overrides(). arbitrate() states the same decision for one conflict and
|
||||
feeds the report this module prints.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -126,26 +129,6 @@ def _normalize(destination: str, fold: bool = True) -> str:
|
||||
return cleaned.casefold() if fold else cleaned
|
||||
|
||||
|
||||
def build_claim_index(claims: list[Claim]) -> dict[str, list[Claim]]:
|
||||
"""Index claims by destination and by name, the name carrying the union.
|
||||
|
||||
A lookup by full destination answers for one slot. A lookup by bare name
|
||||
answers with every slot that shares it, which is what a candidate known
|
||||
only by its filename needs in order not to be dropped.
|
||||
"""
|
||||
index: dict[str, list[Claim]] = {}
|
||||
for claim in claims:
|
||||
keys = {_normalize(claim.destination)}
|
||||
if claim.name:
|
||||
keys.add(_normalize(claim.name))
|
||||
base = claim.destination.rsplit("/", 1)[-1]
|
||||
if base:
|
||||
keys.add(_normalize(base))
|
||||
for key in keys - {""}:
|
||||
index.setdefault(key, []).append(claim)
|
||||
return index
|
||||
|
||||
|
||||
def platform_claims(
|
||||
config: dict,
|
||||
db: dict,
|
||||
|
||||
+28
-6
@@ -24,6 +24,7 @@ import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
@@ -73,6 +74,22 @@ LARGE_FILE_ISSUE_URL = (
|
||||
)
|
||||
|
||||
|
||||
def large_file_issue_url(result: ValidationResult) -> str:
|
||||
"""The issue form with the values this run computed already filled in.
|
||||
|
||||
GitHub fills an issue form field from the query parameter named by its
|
||||
id: filename, sha1, md5 and size in missing-bios.yml.
|
||||
"""
|
||||
fields = {
|
||||
"title": f"[BIOS] Missing: {result.filename}",
|
||||
"filename": result.filename,
|
||||
"sha1": result.sha1,
|
||||
"md5": result.md5,
|
||||
"size": str(result.size),
|
||||
}
|
||||
return f"{LARGE_FILE_ISSUE_URL}&{urllib.parse.urlencode(fields)}"
|
||||
|
||||
|
||||
class ValidationResult:
|
||||
def __init__(self, filepath: str) -> None:
|
||||
self.filepath = filepath
|
||||
@@ -359,14 +376,15 @@ def validate_file(
|
||||
if ext in BLOCKED_EXTENSIONS:
|
||||
result.add_check(False, f"Blocked file extension: {ext}")
|
||||
|
||||
if result.size > MAX_FILE_SIZE:
|
||||
too_large = result.size > MAX_FILE_SIZE
|
||||
if too_large:
|
||||
# GitHub refuses a pushed file over 100 MiB, so a pull request cannot
|
||||
# carry it. Only a maintainer can add an asset to the release.
|
||||
result.add_check(
|
||||
False,
|
||||
f"File too large for git ({result.size:,} > {MAX_FILE_SIZE:,} bytes). "
|
||||
"Leave it out of the pull request and open an issue "
|
||||
f"({LARGE_FILE_ISSUE_URL}) with the hashes above and where the file "
|
||||
"Leave it out of the pull request and open an issue, its hashes "
|
||||
f"filled in ({large_file_issue_url(result)}), saying where the file "
|
||||
"can be obtained: a maintainer adds it to the `large-files` release.",
|
||||
)
|
||||
elif result.size == 0:
|
||||
@@ -450,8 +468,8 @@ def validate_file(
|
||||
result.outcome = "unknown"
|
||||
result.add_warning(
|
||||
"Not recognised: no platform list or emulator profile declares this "
|
||||
"file. Say in the pull request which emulator reads it, for which "
|
||||
"system, and where the dump comes from"
|
||||
f"file. Say in the {'issue' if too_large else 'pull request'} which "
|
||||
"emulator reads it, for which system, and where the dump comes from"
|
||||
)
|
||||
|
||||
normalized = os.path.normpath(relative)
|
||||
@@ -467,7 +485,11 @@ def validate_file(
|
||||
set(),
|
||||
)
|
||||
expected_folders = sorted(expected)
|
||||
if normalized.startswith("bios" + os.sep):
|
||||
if too_large:
|
||||
# The file stays out of the pull request, so no folder of bios/ is
|
||||
# the contributor's to choose: the release asset is placed by hand.
|
||||
pass
|
||||
elif normalized.startswith("bios" + os.sep):
|
||||
parts = normalized.split(os.sep)
|
||||
here = f"bios/{parts[1]}/{parts[2]}/" if len(parts) >= 4 else ""
|
||||
makers = {folder.split("/")[1] for folder in expected}
|
||||
|
||||
@@ -6330,7 +6330,13 @@ class DocsDescribeTheRegistriesTheyName(unittest.TestCase):
|
||||
from dedup import NODEDUP_DIRS
|
||||
tools = (ROOT / "wiki" / "tools.md").read_text(encoding="utf-8")
|
||||
line = next(l for l in tools.splitlines() if "NODEDUP_DIRS" in l)
|
||||
self.assertEqual(sorted(d for d in NODEDUP_DIRS if d not in line), [])
|
||||
# Compared as list items: `mame` was missing while the substring
|
||||
# test found it in `_mame_clones.json` on the same line.
|
||||
listed = line.split("`NODEDUP_DIRS`", 1)[1].split(":", 1)[1].rstrip(" |")
|
||||
self.assertEqual(
|
||||
sorted(item.strip() for item in listed.split(",")),
|
||||
sorted(NODEDUP_DIRS),
|
||||
)
|
||||
readme_gen = (ROOT / "scripts" / "generate_readme.py").read_text(encoding="utf-8")
|
||||
self.assertIn("NODEDUP_DIRS", readme_gen)
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ report is folded, since each of those decides whether a row says STALE.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -353,6 +354,43 @@ class PinParsingTests(unittest.TestCase):
|
||||
self.assertTrue(cf.specifier_allows(">=10.14", "12.0.1"))
|
||||
|
||||
|
||||
class SiteNpmPins(unittest.TestCase):
|
||||
"""The site's npm dependencies are pinned to exact versions in
|
||||
website/package.json, and the ci area compared only the PyPI and action
|
||||
pins: astro 7.4.2 stayed behind 7.4.7 with no STALE row."""
|
||||
|
||||
def test_both_dependency_kinds_are_read(self):
|
||||
pins = cf.parse_npm_pins(
|
||||
'{"dependencies": {"astro": "7.4.2"}, "devDependencies": {"x": "^1.0.0"}}'
|
||||
)
|
||||
self.assertEqual(pins, {"astro": "7.4.2", "x": "^1.0.0"})
|
||||
|
||||
def test_a_newer_release_is_stale(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
package = Path(tmp) / "package.json"
|
||||
package.write_text(json.dumps({"dependencies": {
|
||||
"astro": "7.4.2", "@astrojs/starlight": "0.42.9", "y": "^2.0.0",
|
||||
}}))
|
||||
latest = {"astro": "7.4.7", "@astrojs/starlight": "0.42.9"}
|
||||
def answer(url):
|
||||
name = url.split("registry.npmjs.org/", 1)[1].rsplit("/latest", 1)[0]
|
||||
return {"version": latest[name]}
|
||||
with mock.patch.object(cf, "SITE_PACKAGE", package), \
|
||||
mock.patch.object(cf.upstream, "_http_json", side_effect=answer):
|
||||
rows = {f.subject: f for f in cf._npm_findings()}
|
||||
self.assertEqual(rows["npm:astro"].status, cf.STALE)
|
||||
self.assertEqual(rows["npm:astro"].remote, "7.4.7")
|
||||
self.assertEqual(rows["npm:@astrojs/starlight"].status, cf.OK)
|
||||
self.assertEqual(rows["npm:y"].status, cf.UNKNOWN)
|
||||
|
||||
def test_the_ci_area_reads_the_site_pins(self):
|
||||
with mock.patch.object(cf, "_npm_findings", return_value=[]) as npm, \
|
||||
mock.patch.object(cf, "parse_pip_pins", return_value={}), \
|
||||
mock.patch.object(cf, "parse_action_pins", return_value={}):
|
||||
cf.check_ci("unused", offline=False)
|
||||
npm.assert_called_once()
|
||||
|
||||
|
||||
class RomFolderPins(unittest.TestCase):
|
||||
"""The rom folders of a setup were pinned by hand and nothing compared
|
||||
the pins with the platform file or the project's releases."""
|
||||
|
||||
@@ -75,7 +75,8 @@ class BeforeOpeningAPullRequest(unittest.TestCase):
|
||||
# CI holds added profiles to their upstream, not edited ones: --changed
|
||||
# failed a contributor on refs they never wrote.
|
||||
self.assertIn("profile_sync.py --added --base", workflow)
|
||||
self.assertIn("profile_sync.py --added --base upstream/main --strict", block)
|
||||
self.assertIn("profile_sync.py --added --strict", block)
|
||||
self.assertNotIn("upstream/main", section)
|
||||
self.assertNotIn("--changed --strict", block)
|
||||
|
||||
def test_it_names_the_site_build_and_the_packages_ci_runs(self):
|
||||
|
||||
@@ -1738,6 +1738,12 @@ class ArchitectureStatesTheArbitration(unittest.TestCase):
|
||||
section = page[page.index("### When both layers claim one destination"):]
|
||||
section = section[:section.index("\n## ")]
|
||||
self.assertIn("`slots.arbitrate()`", section)
|
||||
# The pack, the manifest and verify decide through slot_choices, and
|
||||
# a test in test_slots forbids them to call arbitrate: the page names
|
||||
# the function they read, not only the one the report prints.
|
||||
self.assertIn("`slots.slot_choices()`", section)
|
||||
self.assertIn("`slots.pack_overrides()`", section)
|
||||
self.assertNotIn("both decide through `slots.arbitrate()`", section)
|
||||
source = inspect.getsource(slots.arbitrate)
|
||||
# The three outcomes the section describes.
|
||||
for outcome in ("ADDRESSEE", "SERVES_BOTH", "FRONTEND_CHECKS_CONTENT"):
|
||||
@@ -1894,5 +1900,89 @@ class SkipFlagsNameTheSteps(unittest.TestCase):
|
||||
self.assertEqual(set(re.findall(self.STEP, text)), steps)
|
||||
|
||||
|
||||
class TheInstallerSamplesTravelWithTheManifests(unittest.TestCase):
|
||||
"""A manifest refresh (c624a820f) went out without the run samples of
|
||||
wiki/installer.md, which count what the manifests hold, and CI turned
|
||||
red. The rule lived in no page, and the nightly's commit rules had no
|
||||
place for the samples the gate's pipeline rewrites."""
|
||||
|
||||
def test_the_pages_and_the_nightly_carry_them(self):
|
||||
import autorun # noqa: PLC0415
|
||||
|
||||
guide = (WIKI / "testing-guide.md").read_text(encoding="utf-8")
|
||||
for name in ("wiki/installer.md", "derivable.json", "generate_installer_doc.py"):
|
||||
self.assertIn(name, guide)
|
||||
tools = (WIKI / "tools.md").read_text(encoding="utf-8")
|
||||
manifests = tools.index("--all --manifest --output-dir install/")
|
||||
self.assertIn("generate_installer_doc.py", tools[manifests:manifests + 300])
|
||||
nightly = autorun.load_jobs()["nightly"]
|
||||
patterns = [p for rule in nightly.commits for p in rule.paths]
|
||||
for path in ("wiki/installer.md", "derivable.json", "install/retroarch.json"):
|
||||
with self.subTest(path=path):
|
||||
self.assertTrue(any(autorun.path_matches(p, path) for p in patterns))
|
||||
|
||||
|
||||
class TheRemovalProofsAreTheExportersOwn(unittest.TestCase):
|
||||
"""tools.md listed the proofs that keep an entry out of
|
||||
removal_candidates without the platform's launcher records, which
|
||||
candidates.py checks right after a path."""
|
||||
|
||||
WORDS = {"path": "`path:`", "launcher": "`launchers:`", "name": "by name or alias",
|
||||
"hash": "by any hash", "directory": "`type: directory`",
|
||||
"data_directory": "a data directory"}
|
||||
|
||||
def test_the_page_names_each_proof_in_order(self):
|
||||
from exporter.candidates import _PROOF_ORDER # noqa: PLC0415
|
||||
|
||||
page = (WIKI / "tools.md").read_text(encoding="utf-8")
|
||||
start = page.index("No entry the platform declares is removed.")
|
||||
section = page[start:page.index("removal_candidates", start)]
|
||||
self.assertEqual(set(self.WORDS), set(_PROOF_ORDER))
|
||||
positions = [section.index(self.WORDS[kind]) for kind in _PROOF_ORDER]
|
||||
self.assertEqual(positions, sorted(positions))
|
||||
|
||||
|
||||
class APlatformListIsDescribedNotJudged(unittest.TestCase):
|
||||
"""adding-a-platform opened on files a platform's own list "leaves out",
|
||||
read first by the maintainer of that platform: "does not name" says the
|
||||
same without imputing an omission."""
|
||||
|
||||
def test_no_page_says_a_list_leaves_files_out(self):
|
||||
pattern = re.compile(r"\blists? (?:leaves|left|leave) out\b")
|
||||
for page in sorted(WIKI.glob("*.md")):
|
||||
with self.subTest(page=page.name):
|
||||
self.assertIsNone(pattern.search(page.read_text(encoding="utf-8")))
|
||||
|
||||
|
||||
class ArchivedIsWhatTheToolsDo(unittest.TestCase):
|
||||
"""The README and the setup pages said an archived platform is no longer
|
||||
scraped on a schedule, and adding-a-platform said `active` feeds a CI
|
||||
matrix and the scrapers. No workflow calls list_platforms.py and no tool
|
||||
reads `schedule:`: what `archived` changes is which full runs include it."""
|
||||
|
||||
def test_no_text_ties_the_status_to_automation(self):
|
||||
import generate_pack # noqa: PLC0415
|
||||
import generate_readme # noqa: PLC0415
|
||||
import generate_site # noqa: PLC0415
|
||||
import list_platforms # noqa: PLC0415
|
||||
|
||||
texts = {
|
||||
"adding-a-platform.md": (WIKI / "adding-a-platform.md").read_text(encoding="utf-8"),
|
||||
"generate_readme": inspect.getsource(generate_readme),
|
||||
"generate_site": inspect.getsource(generate_site),
|
||||
"generate_pack": inspect.getsource(generate_pack),
|
||||
"list_platforms": inspect.getsource(list_platforms),
|
||||
"_guides.yml": (ROOT / "platforms" / "_guides.yml").read_text(encoding="utf-8"),
|
||||
}
|
||||
for name, text in texts.items():
|
||||
with self.subTest(name):
|
||||
for claim in ("scraped on a schedule", "read again on a schedule",
|
||||
"CI matrix", "without the automation"):
|
||||
self.assertNotIn(claim, text)
|
||||
self.assertIn("`--include-archived`", texts["adding-a-platform.md"])
|
||||
for workflow in (ROOT / ".github" / "workflows").glob("*.yml"):
|
||||
self.assertNotIn("list_platforms", workflow.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -248,6 +248,21 @@ class SlicesLeaveWithTheirRelease(unittest.TestCase):
|
||||
self.assertNotIn(".zip.001", _which_pack())
|
||||
|
||||
|
||||
class TheReadmeLeadsBeginnersToTheirPages(unittest.TestCase):
|
||||
"""The README linked the installer reference and the FAQ's no-ZIP anchor
|
||||
only: a user stopped by an install error or a BIOS-not-found message had
|
||||
no link to the first install or troubleshooting page."""
|
||||
|
||||
def test_quick_install_links_the_start_pages(self):
|
||||
# The code block holds `#` comment lines, so the section runs to
|
||||
# the next level-two heading.
|
||||
section = _read("README.md").split("## Quick install", 1)[1].split("\n## ", 1)[0]
|
||||
for page in ("first-install", "troubleshooting", "faq"):
|
||||
with self.subTest(page=page):
|
||||
self.assertIn(f"wiki/{page}/)", section)
|
||||
self.assertTrue((REPO_ROOT / "wiki" / f"{page}.md").is_file())
|
||||
|
||||
|
||||
class EndUsersCheckTheirOwnFolder(unittest.TestCase):
|
||||
"""verify.py reads database.json and bios/. Advice about a user's own
|
||||
files points at the installer's --check, which hashes their folder."""
|
||||
|
||||
@@ -6034,7 +6034,11 @@ class TestBranchCheckIsWired(unittest.TestCase):
|
||||
import generate_readme # noqa: PLC0415
|
||||
|
||||
text = generate_readme.generate_contributing()
|
||||
self.assertIn("profile_sync.py --added --base upstream/main --strict", text)
|
||||
self.assertIn("profile_sync.py --added --strict", text)
|
||||
# The default base is fetched by the tool: a documented upstream
|
||||
# remote failed every clone that had origin alone.
|
||||
self.assertNotIn("--base upstream/main", text)
|
||||
self.assertNotIn("remote add upstream", text)
|
||||
self.assertNotIn("--emulator NAME --strict", text)
|
||||
workflow = (root / ".github" / "workflows" / "validate.yml").read_text()
|
||||
self.assertIn("profile_sync.py --added --base", workflow)
|
||||
|
||||
@@ -1452,5 +1452,23 @@ class UsaRegionServesTheAmericas(unittest.TestCase):
|
||||
self.assertEqual(region.rank(jpn, requested), 1)
|
||||
|
||||
|
||||
class RegionWordsInAPackReadme(unittest.TestCase):
|
||||
"""The pack README title-cased each slug: asia-ntsc read "Asia Ntsc"."""
|
||||
|
||||
def test_the_priority_line_reads_as_written(self):
|
||||
from packreadme import _build_readme # noqa: PLC0415
|
||||
|
||||
readme = _build_readme(
|
||||
"retroarch", "RetroArch", "system", 1, 1,
|
||||
regions=region.parse_requested("us,asia-ntsc,jp"),
|
||||
)
|
||||
self.assertIn("Region priority: North America, Asia (NTSC), Japan", readme)
|
||||
|
||||
def test_every_region_has_a_label(self):
|
||||
for slug in region.REGIONS:
|
||||
with self.subTest(slug=slug):
|
||||
self.assertNotRegex(region.label(slug), r"\bNtsc\b|\bPal\b|\bUk\b")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,103 @@
|
||||
"""A file that leaves the collection keeps its published address.
|
||||
|
||||
a2diskiing.zip (bd02f6a0) was rebuilt from its recipe on 2026-10-22 and
|
||||
filed under Apple II as d9bb1897. Its card's address, files/?sha1=bd02f6a0,
|
||||
answered that no file the site knows has the hash and asked for a
|
||||
contribution of a file the collection had held.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||
|
||||
import retired_files # noqa: E402
|
||||
|
||||
OLD = "bd" * 20
|
||||
NEW = "d9" * 20
|
||||
|
||||
|
||||
def _db(**files: dict) -> dict:
|
||||
return {"files": files}
|
||||
|
||||
|
||||
class WhatLeftIsRecorded(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.old = _db(**{
|
||||
OLD: {"name": "a2diskiing.zip", "path": "bios/Arcade/Arcade/a2diskiing.zip",
|
||||
"md5": "43" * 16, "crc32": "292ef826", "size": 589},
|
||||
"11" * 20: {"name": "kept.bin", "path": "bios/A/B/kept.bin"},
|
||||
})
|
||||
self.new = _db(**{
|
||||
NEW: {"name": "a2diskiing.zip", "path": "bios/Apple/Apple II/a2diskiing.zip"},
|
||||
"11" * 20: {"name": "kept.bin", "path": "bios/A/B/kept.bin"},
|
||||
})
|
||||
|
||||
def test_the_departure_and_the_file_in_its_place(self):
|
||||
left = retired_files.retired_between(self.old, self.new, "2026-10-22")
|
||||
self.assertEqual(list(left), [OLD])
|
||||
self.assertEqual(left[OLD]["left"], "2026-10-22")
|
||||
self.assertEqual(left[OLD]["successor"], NEW)
|
||||
self.assertEqual(left[OLD]["md5"], "43" * 16)
|
||||
|
||||
def test_the_first_departure_stays_and_a_return_leaves_the_record(self):
|
||||
first = retired_files.merge({}, retired_files.retired_between(self.old, self.new, "2026-10-22"),
|
||||
self.new)
|
||||
again = retired_files.merge(first, {OLD: {"left": "2026-11-01"}}, self.new)
|
||||
self.assertEqual(again[OLD]["left"], "2026-10-22")
|
||||
self.assertEqual(retired_files.merge(again, {}, self.old), {})
|
||||
|
||||
def test_generate_db_writes_it_beside_the_database(self):
|
||||
with tempfile.TemporaryDirectory(dir=REPO_ROOT / "tmp") as tmp:
|
||||
database = Path(tmp) / "database.json"
|
||||
path = retired_files.retired_path(database)
|
||||
self.assertEqual(path, Path(tmp) / "retired.json")
|
||||
self.assertEqual(retired_files.update(path, self.old, self.new, "2026-10-22"), 1)
|
||||
self.assertEqual(retired_files.update(path, self.old, self.new, "2026-10-23"), 0)
|
||||
record = json.loads(path.read_text())["files"]
|
||||
self.assertEqual(record[OLD]["left"], "2026-10-22")
|
||||
source = (REPO_ROOT / "scripts" / "generate_db.py").read_text(encoding="utf-8")
|
||||
self.assertIn("retired_files.update(", source)
|
||||
|
||||
|
||||
class TheLookupAnswersForIt(unittest.TestCase):
|
||||
def test_records_lead_to_the_card_in_its_place(self):
|
||||
import generate_site # noqa: PLC0415
|
||||
|
||||
retired = {OLD: {"name": "a2diskiing.zip", "path": "bios/Arcade/Arcade/a2diskiing.zip",
|
||||
"crc32": "292ef826", "left": "2026-10-22", "successor": "ee" * 20},
|
||||
"ee" * 20: {"name": "a2diskiing.zip", "path": "bios/Apple/x/a2diskiing.zip",
|
||||
"left": "2026-10-23", "successor": NEW}}
|
||||
files = {NEW: {"name": "a2diskiing.zip"}}
|
||||
locations = [{"sha1": NEW, "url": "/retrobios/systems/apple/#sha1-d9d9d9d9d9d9"}]
|
||||
records = generate_site.retired_records(retired, files, locations)
|
||||
found = {(r["sha1"], r["algorithm"]): r for r in records}
|
||||
# Followed to the file held today, through the copy that left after.
|
||||
self.assertEqual(found[(OLD, "sha1")]["url"], locations[0]["url"])
|
||||
self.assertEqual(found[(OLD, "sha1")]["successor"], NEW)
|
||||
self.assertEqual(found[(OLD, "crc32")]["value"], "292ef826")
|
||||
self.assertEqual(found[(OLD, "sha1")]["layer"], "retired")
|
||||
self.assertEqual(found[(OLD, "sha1")]["left"], "2026-10-22")
|
||||
|
||||
def test_the_committed_record_holds_the_reported_file(self):
|
||||
record = retired_files.load(REPO_ROOT / "retired.json")
|
||||
if not record:
|
||||
self.skipTest("retired.json is not written")
|
||||
self.assertIn("bd02f6a0ce8accba7ea5415fd22c2e8f23ec4b10", record)
|
||||
database = json.loads((REPO_ROOT / "database.json").read_text(encoding="utf-8"))
|
||||
self.assertFalse(set(record) & set(database["files"]))
|
||||
|
||||
def test_the_page_script_answers_with_it(self):
|
||||
script = (REPO_ROOT / "website" / "src" / "scripts" / "site.ts").read_text(encoding="utf-8")
|
||||
self.assertIn("item.layer === 'retired'", script)
|
||||
self.assertIn("left the collection on", script)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -110,7 +110,7 @@ class ASystemCardShowsTheAnswerFirst(unittest.TestCase):
|
||||
self.shown = self.text.replace(self.fold, "")
|
||||
|
||||
def test_the_open_part(self) -> None:
|
||||
for fact in ("<code>ppge_atlas.zim</code>", ">650.9 KB<", ">Download</a>",
|
||||
for fact in ("<code>ppge_atlas.zim</code>", ">666,539 B<", ">Download</a>",
|
||||
"<dt>On the setup's list</dt>", "<dt>Packed for its emulators by</dt>",
|
||||
"<dt>Emulators</dt>", f'<dt>SHA1</dt><dd class="rb-hash">{SHA1[:12]}</dd>'):
|
||||
self.assertIn(fact, self.shown)
|
||||
@@ -193,11 +193,23 @@ class AnInstallBoxShowsOneCommand(unittest.TestCase):
|
||||
self.assertEqual(re.findall(r'<TabItem label="([^"]+)"', tabs),
|
||||
["Windows", "macOS", "Linux", "Steam Deck"])
|
||||
self.assertEqual(tabs.count("--platform"), 4)
|
||||
self.assertEqual(tabs.count("--platform emudeck --system sony-psp"), 4)
|
||||
# RetroArch, which runs on each tab's system, not the first by name.
|
||||
self.assertEqual(tabs.count("--platform retroarch --system"), 4)
|
||||
|
||||
def test_the_default_setup(self) -> None:
|
||||
"""AmberELEC led 17 system pages, a handheld system under the
|
||||
Windows tab, for being first in alphabetical order."""
|
||||
self.assertEqual(gs._default_setup([("amberelec", "psx"), ("retroarch", "Sony - PlayStation")]),
|
||||
("retroarch", "Sony - PlayStation"))
|
||||
self.assertEqual(gs._default_setup([("amberelec", "x"), ("batocera", "x")]), ("batocera", "x"))
|
||||
self.assertEqual(gs._default_setup([("amberelec", "x"), ("knulli", "x")]), ("amberelec", "x"))
|
||||
select = re.search(r"<select>(.*?)</select>", self.text).group(1)
|
||||
self.assertEqual(re.findall(r'<option value="([^"]+)" selected', select), ["retroarch"])
|
||||
|
||||
def test_the_setup_list_carries_every_command(self) -> None:
|
||||
select = re.search(r"<select>(.*?)</select>", self.text).group(1)
|
||||
options = re.findall(r'<option value="([^"]+)" data-sh="([^"]+)" data-ps="([^"]+)">', select)
|
||||
options = re.findall(r'<option value="([^"]+)"(?: selected)? data-sh="([^"]+)" data-ps="([^"]+)">',
|
||||
select)
|
||||
self.assertEqual([value for value, _sh, _ps in options], ["emudeck", "retroarch", "retrobat"])
|
||||
self.assertIn(("retrobat", "--platform retrobat --system 'Neo Geo'",
|
||||
"--platform retrobat --system 'Neo Geo'"), options)
|
||||
|
||||
@@ -763,5 +763,76 @@ class SmallFixes(unittest.TestCase):
|
||||
self.assertEqual(said, [])
|
||||
|
||||
|
||||
class AnEmulatorPageNamesItsSetups(unittest.TestCase):
|
||||
"""emulators/handy named no setup that runs Handy: its lynxboot.img card
|
||||
names the setups whose own list holds it, not EmuDeck, ROCKNIX and
|
||||
NextUI, which pack it for the core."""
|
||||
|
||||
def test_setups_come_from_the_core_resolution(self):
|
||||
coverages = {
|
||||
"b": {"config": {"cores": ["handy"]}},
|
||||
"a": {"config": {"cores": ["handy", "other"]}},
|
||||
"c": {"config": {"cores": ["other"]}},
|
||||
}
|
||||
profiles = {"handy": {"type": "libretro"}, "other": {"type": "libretro"}}
|
||||
self.assertEqual(gs.emulator_setups(coverages, profiles)["handy"], ["a", "b"])
|
||||
|
||||
def test_the_page_names_them(self):
|
||||
from tests import sitepages # noqa: PLC0415
|
||||
|
||||
sitepages.require()
|
||||
text = sitepages.page("emulators/handy.md").read_text(encoding="utf-8")
|
||||
row = next(line for line in text.splitlines() if line.startswith("| Run by |"))
|
||||
for setup in ("emudeck", "rocknix", "nextui", "retroarch"):
|
||||
self.assertIn(f"/platforms/{setup}/", row)
|
||||
# Setup names on every emulator page outranked the setup's own page.
|
||||
self.assertIn("data-pagefind-ignore", row)
|
||||
|
||||
|
||||
class SizesTheCodeComparesAreExact(unittest.TestCase):
|
||||
"""raze validates DUKE3D.GRP by size, and its page showed 42.3 MB for
|
||||
44,356,548 and 44,348,015 bytes alike."""
|
||||
|
||||
def test_a_validated_size_carries_its_bytes(self):
|
||||
profile = {"emulator": "x", "type": "libretro", "systems": ["sony-playstation"], "files": [
|
||||
{"name": "DUKE3D.GRP", "size": 44348015, "validation": ["size", "crc32"]},
|
||||
{"name": "other.bin", "size": 44356548},
|
||||
{"name": "built.bin", "size": 2304, "validation": {"core": ["size"]}},
|
||||
]}
|
||||
page = _page("x", profile)
|
||||
self.assertIn("Size: 42.3 MB (44,348,015 bytes)", page)
|
||||
self.assertIn("Size: 2.2 KB (2,304 bytes)", page)
|
||||
self.assertNotIn("44,356,548", page)
|
||||
|
||||
|
||||
class ADataDirectoryNamesAFetchableOrigin(unittest.TestCase):
|
||||
"""pcee2's page linked github.com/PCSX2/pcsx2/archive/{version}.tar.gz,
|
||||
a 404, and buildbot zips showed `Revision: master (2047720058)`: a branch
|
||||
that means nothing for a buildbot URL and the server's ETag."""
|
||||
|
||||
def test_the_url_is_substituted_and_a_commit_is_linked(self):
|
||||
entry = {"source_url": "https://github.com/PCSX2/pcsx2/archive/{version}.tar.gz",
|
||||
"version": "6db638dc0373769b4c5d5b5dd7ed6da641b9a18e"}
|
||||
lines = gs.data_directory_origin(entry, {"sha": entry["version"],
|
||||
"version": entry["version"]})
|
||||
self.assertEqual(lines[0], "Fetched from <https://github.com/PCSX2/pcsx2/archive/"
|
||||
"6db638dc0373769b4c5d5b5dd7ed6da641b9a18e.tar.gz>")
|
||||
self.assertIn("(https://github.com/PCSX2/pcsx2/commit/6db638dc0373769b4c5d5b5dd7ed6da641b9a18e)",
|
||||
lines[1])
|
||||
self.assertNotIn("{version}", " ".join(lines))
|
||||
|
||||
def test_an_etag_is_no_revision(self):
|
||||
entry = {"source_url": "https://buildbot.libretro.com/assets/system/Dolphin.zip"}
|
||||
lines = gs.data_directory_origin(entry, {"sha": '"2047720058"', "version": "master"})
|
||||
self.assertEqual(lines, ["Fetched from <https://buildbot.libretro.com/assets/system/Dolphin.zip>"])
|
||||
|
||||
def test_every_registered_url_resolves(self):
|
||||
registry = yaml.safe_load((REPO_ROOT / "platforms" / "_data_dirs.yml").read_text())
|
||||
for ref, entry in (registry.get("data_directories") or registry).items():
|
||||
if isinstance(entry, dict):
|
||||
with self.subTest(ref=ref):
|
||||
self.assertNotIn("{", " ".join(gs.data_directory_origin(entry, {})))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -937,6 +937,19 @@ class ARouteWithoutANetwork(unittest.TestCase):
|
||||
self.assertEqual(_tab_labels(card), set())
|
||||
self.assertIn("**Linux**", card)
|
||||
|
||||
def test_a_linux_partition_card_says_why_and_offers_linux_alone(self):
|
||||
"""Lakka's /storage is the card's ext4 partition (Lakka-LibreELEC
|
||||
scripts/mkimage): its page offered Windows and macOS tabs for a
|
||||
folder neither can open."""
|
||||
for name in ("lakka", "retropie"):
|
||||
with self.subTest(setup=name):
|
||||
self.assertEqual(siteguides.GUIDES[name].card_run, ("linux",))
|
||||
card = _plain_page(f"platforms/{name}.md").split("No network?", 1)[1]
|
||||
card = card.split("## On the device", 1)[0]
|
||||
self.assertEqual(_tab_labels(card), set())
|
||||
self.assertNotIn("E:\\", card)
|
||||
self.assertIn("Only a Linux computer opens the card.", card)
|
||||
|
||||
def test_no_zip_is_promised_for_an_unmarked_setup(self):
|
||||
for name in SETUPS:
|
||||
if REGISTRY[name].get("release_pack") is True:
|
||||
|
||||
@@ -785,6 +785,53 @@ class AHashFindsEveryPageThatNamesIt(unittest.TestCase):
|
||||
self.assertNotIn("no.json", names)
|
||||
|
||||
|
||||
class AnArchiveMemberIsFoundByItsCrc(unittest.TestCase):
|
||||
"""A pasted CRC of logica2.u2 (8484f426), which the collection holds
|
||||
inside a500.zip, answered that no file the site knows has it: the
|
||||
lookup held the members a profile's `contents:` declares, 7,389 of
|
||||
33,403."""
|
||||
|
||||
def test_each_member_is_a_record_leading_to_its_archive(self):
|
||||
import zipfile # noqa: PLC0415
|
||||
|
||||
import generate_site # noqa: PLC0415
|
||||
|
||||
with tempfile.TemporaryDirectory(dir=Path(__file__).resolve().parents[1] / "tmp") as tmp:
|
||||
root = Path(tmp)
|
||||
archive = root / "bios" / "Arcade" / "Arcade" / "a500.zip"
|
||||
archive.parent.mkdir(parents=True)
|
||||
with zipfile.ZipFile(archive, "w") as handle:
|
||||
handle.writestr("logica2.u2", b"rom")
|
||||
handle.writestr("sub/kick.rom", b"kick")
|
||||
entry = {"name": "a500.zip", "path": "bios/Arcade/Arcade/a500.zip"}
|
||||
records = generate_site._archive_member_records(entry, "ab" * 20, "/retrobios/x/", root)
|
||||
plain = generate_site._archive_member_records(
|
||||
{"name": "a.bin", "path": "bios/a.bin"}, "cd" * 20, "/retrobios/y/", root)
|
||||
self.assertEqual(plain, [])
|
||||
import zlib # noqa: PLC0415
|
||||
|
||||
self.assertEqual(records, [
|
||||
{"value": f"{zlib.crc32(b'rom'):08x}", "algorithm": "crc32", "layer": "archive-member",
|
||||
"name": "logica2.u2", "archive": "a500.zip", "sha1": "ab" * 20, "url": "/retrobios/x/"},
|
||||
{"value": f"{zlib.crc32(b'kick'):08x}", "algorithm": "crc32", "layer": "archive-member",
|
||||
"name": "kick.rom", "archive": "a500.zip", "sha1": "ab" * 20, "url": "/retrobios/x/"},
|
||||
])
|
||||
|
||||
def test_the_lookup_and_the_schema_carry_the_layer(self):
|
||||
import json as _json # noqa: PLC0415
|
||||
|
||||
import generate_site # noqa: PLC0415
|
||||
|
||||
source = Path(generate_site.__file__).read_text(encoding="utf-8")
|
||||
self.assertIn("records += _archive_member_records(entry, sha1, item[\"url\"])", source)
|
||||
schema = _json.loads((Path(__file__).resolve().parents[1] / "schemas"
|
||||
/ "site-api-envelope.schema.json").read_text())
|
||||
self.assertIn('"archive-member"', _json.dumps(schema))
|
||||
script = (Path(__file__).resolve().parents[1] / "website" / "src" / "scripts"
|
||||
/ "site.ts").read_text(encoding="utf-8")
|
||||
self.assertIn("byLayer('archive-member')", script)
|
||||
|
||||
|
||||
class AFileHasAnAddressByContent(unittest.TestCase):
|
||||
"""mt32_control.rom moved from the Commodore page to the Roland page and
|
||||
every link to its card broke: the anchor was content-based, the page
|
||||
|
||||
@@ -206,13 +206,21 @@ class FaqInstallsFromWhatYouHold(unittest.TestCase):
|
||||
import install # noqa: PLC0415
|
||||
|
||||
section = " ".join(self._section("Fill an SD card from a computer?").split())
|
||||
self.assertIn("The interface finds a muOS, Onion, spruceOS, MinUI, KNULLI or ROCKNIX "
|
||||
"card by itself.", section)
|
||||
self.assertIn("The interface finds a muOS, Onion, spruceOS, MinUI, KNULLI, ROCKNIX, Lakka "
|
||||
"or MiSTer card.", section)
|
||||
known = {rule.name for rule in install._card_rules()}
|
||||
known |= {install.display_name(name) for name, _marker, _prefix in install._CARD_RULES}
|
||||
for name in ("muOS", "Onion", "spruceOS", "MinUI", "KNULLI", "ROCKNIX"):
|
||||
for name in ("muOS", "Onion", "spruceOS", "MinUI", "KNULLI", "ROCKNIX", "MiSTer FPGA"):
|
||||
with self.subTest(card=name):
|
||||
self.assertIn(name, known)
|
||||
# A card told by a rule of its share (MiSTer since 73b82375d) is named
|
||||
# here and in the README, or its user passes --dest the card's root.
|
||||
readme = (REPO_ROOT / "README.md").read_text(encoding="utf-8")
|
||||
for rule in install._SHARE_RULES:
|
||||
word = rule.name.split()[0]
|
||||
with self.subTest(share=rule.name):
|
||||
self.assertIn(word, section)
|
||||
self.assertRegex(readme, rf"# SD card mounted on a PC: .*\b{word}\b")
|
||||
# The label the first screen gives a card found: "Onion SD card".
|
||||
self.assertEqual(install._card_word(("retroarch", Path("/x/BIOS"), "Onion")), "SD card")
|
||||
self.assertEqual(install._card_word(("batocera", Path("/x/bios"), "Batocera")),
|
||||
|
||||
@@ -649,6 +649,24 @@ class TheContractsValidate(unittest.TestCase):
|
||||
[e.message for e in envelope.iter_errors(json.loads((API / name).read_text()))], [])
|
||||
|
||||
|
||||
class TheIndexSaysWhatTheCatalogLists(unittest.TestCase):
|
||||
"""index.json called catalog.json every published file, while the 990
|
||||
one-item documents of emulators/, platforms/, systems/ and lookup/ are
|
||||
in none of its 80 items."""
|
||||
|
||||
def test_the_description_matches_the_items(self):
|
||||
index = json.loads(_built(API / "index.json").read_text())
|
||||
catalog = json.loads(_built(API / "catalog.json").read_text())
|
||||
described = next(i["description"] for i in index["items"] if i["url"] == "catalog.json")
|
||||
self.assertNotIn("Every published file", described)
|
||||
listed = {item["url"].split("/", 1)[0] for item in catalog["items"]}
|
||||
for folder in ("emulators", "platforms", "systems", "lookup"):
|
||||
with self.subTest(folder=folder):
|
||||
self.assertIn(f"{folder}/", described)
|
||||
self.assertNotIn(folder, listed)
|
||||
self.assertTrue(any((API / folder).glob("*.json")))
|
||||
|
||||
|
||||
class EveryEnvelopeCarriesTheCommit(unittest.TestCase):
|
||||
"""8.5: emulators/ and systems/ lacked `commit` (716 documents)."""
|
||||
|
||||
|
||||
@@ -503,5 +503,34 @@ class NoTooltipInsideAName(unittest.TestCase):
|
||||
self.assertIn('>BIOS</abbr>', marked)
|
||||
|
||||
|
||||
class TheArcadeCacheHoldsItsDatabase(unittest.TestCase):
|
||||
"""The set of sha1s filed under Arcade/ was cached by id(db): a database
|
||||
freed and replaced at the same address answered with the first one's set,
|
||||
and an arcade card of the second landed in another section."""
|
||||
|
||||
@staticmethod
|
||||
def _db(sha1: str) -> dict:
|
||||
return {"files": {}, "indexes": {"by_path_suffix": {"Arcade/x.zip": [sha1]}}}
|
||||
|
||||
def test_a_new_database_at_a_reused_address_is_read_again(self):
|
||||
saved = siterender._suffix_cache[0]
|
||||
self.addCleanup(siterender._suffix_cache.__setitem__, 0, saved)
|
||||
first = self._db("a" * 40)
|
||||
self.assertTrue(siterender._in_arcade("a" * 40, first))
|
||||
address = id(first)
|
||||
del first
|
||||
# CPython hands a freed dict's address to a later one of its size,
|
||||
# unless the cache still holds the first database.
|
||||
for _ in range(1000):
|
||||
second = self._db("b" * 40)
|
||||
if id(second) == address:
|
||||
break
|
||||
del second
|
||||
else:
|
||||
second = self._db("b" * 40)
|
||||
self.assertTrue(siterender._in_arcade("b" * 40, second))
|
||||
self.assertFalse(siterender._in_arcade("a" * 40, second))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -304,5 +304,22 @@ class BuildCount(unittest.TestCase):
|
||||
self.assertIn("system ids linked", source)
|
||||
|
||||
|
||||
class AnEntryListingSystemsIsFiledUnderEach(unittest.TestCase):
|
||||
"""MAME declares a device set under `systems:`, the machines of several
|
||||
systems loading it. The site read only `system:`, so ym2608.zip was filed
|
||||
under neither Arcade nor NEC PC-88, and kept on pages it does not serve."""
|
||||
|
||||
def test_the_site_reads_the_common_rule(self):
|
||||
import generate_site as gs # noqa: PLC0415
|
||||
from common import entry_system_ids # noqa: PLC0415
|
||||
|
||||
entry = {"name": "ym2608.zip", "systems": ["arcade", "pc80"]}
|
||||
self.assertEqual(gs._entry_systems("mame", entry), ["arcade", "pc80"])
|
||||
self.assertEqual(
|
||||
gs._entry_systems("mame", entry), entry_system_ids(entry, {"systems": []})
|
||||
)
|
||||
self.assertEqual(gs._entry_systems("mame", {"name": "a", "system": "pc80"}), ["pc80"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -184,6 +184,19 @@ class ThePermalinkIsAnIcon(unittest.TestCase):
|
||||
self.assertEqual(page.count('href="#rb-link-icon"'), 2)
|
||||
|
||||
|
||||
class TheCardGivesTheSizeInBytes(unittest.TestCase):
|
||||
"""The badge rounded to 0.1 KB and no line of a system card gave the
|
||||
bytes an emulator compares."""
|
||||
|
||||
def test_the_badge_is_exact(self):
|
||||
for size, shown in ((524288, "512 KB"), (44348015, "44,348,015 B"),
|
||||
(2 * 1024 * 1024, "2 MB"), (2304, "2,304 B"), (10, "10 B")):
|
||||
with self.subTest(size=size):
|
||||
card = {**_sony_file("scph5501.bin", "4"), "size": size}
|
||||
title = gs._system_file_card(card, FileConsumers({"files": {}}))[2]
|
||||
self.assertIn(f'<span class="rb-badge rb-badge-muted">{shown}</span>', title)
|
||||
|
||||
|
||||
class TheCardLinesSayWhatTheyMean(unittest.TestCase):
|
||||
""""Platforms:" and "Also installed by:" listed different setups, as
|
||||
lower-case ids, and only a code comment told them apart."""
|
||||
|
||||
+22
-25
@@ -75,31 +75,6 @@ def tearDownModule():
|
||||
shutil.rmtree(_ROOT, ignore_errors=True)
|
||||
|
||||
|
||||
class TestClaimIndex(unittest.TestCase):
|
||||
"""Keyed by destination and by name, the name carrying the union."""
|
||||
|
||||
def _claims(self):
|
||||
return [
|
||||
slots.Claim("profile", "GC/USA/IPL.bin", "IPL.bin", emulator="dolphin"),
|
||||
slots.Claim("profile", "GC/JAP/IPL.bin", "IPL.bin", emulator="dolphin"),
|
||||
]
|
||||
|
||||
def test_destination_key_separates_same_named_entries(self):
|
||||
index = slots.build_claim_index(self._claims())
|
||||
self.assertEqual(len(index["gc/usa/ipl.bin"]), 1)
|
||||
self.assertEqual(len(index["gc/jap/ipl.bin"]), 1)
|
||||
|
||||
def test_bare_name_answers_with_the_union(self):
|
||||
index = slots.build_claim_index(self._claims())
|
||||
self.assertEqual(len(index["ipl.bin"]), 2)
|
||||
|
||||
def test_lookup_ignores_case_and_leading_separator(self):
|
||||
index = slots.build_claim_index(
|
||||
[slots.Claim("platform", "/BIOS/Disk.rom", "Disk.rom")]
|
||||
)
|
||||
self.assertIn("bios/disk.rom", index)
|
||||
|
||||
|
||||
class TestConflicts(unittest.TestCase):
|
||||
"""A proven profile claim contradicting what the baseline would ship."""
|
||||
|
||||
@@ -411,6 +386,28 @@ class TestArbitration(unittest.TestCase):
|
||||
# The hashed declaration keeps the file the frontend checks.
|
||||
self.assertEqual(overrides, {"IPL.bin": "bios/Console/GC/JAP/IPL.bin"})
|
||||
|
||||
def test_the_report_and_the_pack_decide_alike(self):
|
||||
"""The report prints arbitrate(), the pack, the manifest and verify
|
||||
read slot_choices(): both must name the same file for a slot."""
|
||||
for mode in ("existence", "md5"):
|
||||
config = {
|
||||
"verification_mode": mode,
|
||||
"cores": ["dolphin"],
|
||||
"systems": {"console": {"files": [
|
||||
{"name": "IPL.bin", "destination": "GC/JAP/IPL.bin", "md5": "m" * 32},
|
||||
]}},
|
||||
}
|
||||
profiles = {"dolphin": {"type": "libretro", "systems": ["console"], "files": [
|
||||
{"name": "IPL.bin", "path": "GC/JAP/IPL.bin"},
|
||||
]}}
|
||||
with self.subTest(mode=mode):
|
||||
[conflict] = slots.find_conflicts(config, profiles, REGIONS_DB)
|
||||
decision = slots.arbitrate(conflict, mode)
|
||||
served = slots.pack_overrides(config, profiles, REGIONS_DB).get(
|
||||
"GC/JAP/IPL.bin", conflict.platform_claim.local_path
|
||||
)
|
||||
self.assertEqual(served, decision.winner.local_path)
|
||||
|
||||
def test_the_reported_line_gives_the_ground_for_the_decision(self):
|
||||
kept = slots.format_decision(slots.arbitrate(self._conflict(), "md5"))
|
||||
self.assertIn("verifies content", kept)
|
||||
|
||||
@@ -115,6 +115,30 @@ class OversizedFileAdvice(unittest.TestCase):
|
||||
self.assertIn("open an issue", message)
|
||||
self.assertIn(validate_pr.LARGE_FILE_ISSUE_URL, message)
|
||||
|
||||
def test_the_issue_form_is_filled_and_nothing_points_to_the_pr(self):
|
||||
"""The link opened an empty form, and the same report said to put the
|
||||
file in the pull request and under bios/."""
|
||||
import urllib.parse # noqa: PLC0415
|
||||
|
||||
path = self.tmp / "big.bin"
|
||||
path.write_bytes(b"x" * 8)
|
||||
validate_pr.MAX_FILE_SIZE = 4
|
||||
result = validate_pr.validate_file(
|
||||
"big.bin", None, {"sha1": set(), "md5": set(), "names": set()},
|
||||
)
|
||||
message = next(m for s, m in result.checks if s == "FAIL" and "too large" in m)
|
||||
url = message.split("(", 2)[2].split(")", 1)[0]
|
||||
query = urllib.parse.parse_qs(urllib.parse.urlsplit(url).query)
|
||||
template = (REPO_ROOT / ".github" / "ISSUE_TEMPLATE" / "missing-bios.yml").read_text()
|
||||
for field, value in (("filename", "big.bin"), ("sha1", result.sha1),
|
||||
("md5", result.md5), ("size", "8")):
|
||||
with self.subTest(field=field):
|
||||
self.assertIn(f"id: {field}\n", template)
|
||||
self.assertEqual(query[field], [value])
|
||||
others = " ".join(m for _s, m in result.checks)
|
||||
self.assertNotIn("pull request which", others)
|
||||
self.assertNotIn("not under bios/", others)
|
||||
|
||||
def test_the_linked_issue_template_exists(self):
|
||||
template = validate_pr.LARGE_FILE_ISSUE_URL.rsplit("template=", 1)[1]
|
||||
self.assertTrue(
|
||||
|
||||
@@ -291,6 +291,7 @@
|
||||
"systems/other-fbneo.md": "systems/other.md",
|
||||
"systems/other-future-pinball-fploader.md": "systems/other.md",
|
||||
"systems/other-gopher2600.md": "systems/other.md",
|
||||
"systems/other-groovymame.md": "systems/other.md",
|
||||
"systems/other-idtech4a.md": "systems/other.md",
|
||||
"systems/other-ines.md": "systems/other.md",
|
||||
"systems/other-mame-2.md": "systems/other.md",
|
||||
|
||||
@@ -21,6 +21,11 @@ type Item = {
|
||||
owner?: string;
|
||||
sha1?: string;
|
||||
system_name?: string;
|
||||
// On an archive-member record: the collection archive holding the member.
|
||||
archive?: string;
|
||||
// On a retired record: the day the file left, and the file in its place.
|
||||
left?: string;
|
||||
successor?: string;
|
||||
};
|
||||
|
||||
const siteRoot = () => new URL(import.meta.env.BASE_URL.replace(/\/?$/, '/'), location.origin);
|
||||
@@ -521,7 +526,9 @@ const renderMatches = async (status: HTMLElement, result: HTMLElement, matched:
|
||||
// naomi2.zip by the MD5 of a ROM inside it, which no file of the
|
||||
// collection has.
|
||||
const searched = new Set(matched.map((item) => item.value));
|
||||
const carried = [...new Set(matched.flatMap((item) => (item.sha1 && !searched.has(item.sha1) ? [item.sha1] : [])))];
|
||||
// A member's record names its archive, which is not the file searched for.
|
||||
const carried = [...new Set(matched.flatMap((item) =>
|
||||
(item.sha1 && !['archive-member', 'retired'].includes(item.layer) && !searched.has(item.sha1) ? [item.sha1] : [])))];
|
||||
const found = carried.length ? [...matched, ...(await lookupValues(carried).catch(() => [] as Item[]))] : matched;
|
||||
const held = distinct(found.filter((item) => item.layer === 'collection'), (item) => item.sha1 ?? item.url);
|
||||
const files = await Promise.all(held.map((item) => heldFile(item).catch(() => undefined)));
|
||||
@@ -550,8 +557,11 @@ const renderMatches = async (status: HTMLElement, result: HTMLElement, matched:
|
||||
const readEntries = emulators.map((item) => readerEntry(reader.get(item.owner ?? ''), item, valuesOf(item)));
|
||||
const setups = new Set(platforms.map((item) => item.owner));
|
||||
|
||||
const members = distinct(byLayer('archive-member'), (item) => `${item.sha1}|${item.name}`);
|
||||
const counted = [
|
||||
held.length ? count(held.length, 'file in the collection', 'files in the collection') : 'not in the collection',
|
||||
held.length ? count(held.length, 'file in the collection', 'files in the collection')
|
||||
: members.length ? count(members.length, 'member of a collection archive', 'members of collection archives')
|
||||
: 'not in the collection',
|
||||
...(platforms.length ? [`${count(platforms.length, 'declaration', 'declarations')} by ${count(setups.size, 'setup', 'setups')}`] : []),
|
||||
...(owners.length ? [count(owners.length, 'emulator that reads it', 'emulators that read it')] : []),
|
||||
];
|
||||
@@ -606,7 +616,19 @@ const renderMatches = async (status: HTMLElement, result: HTMLElement, matched:
|
||||
),
|
||||
);
|
||||
});
|
||||
if (!held.length) blocks.push(paragraph('No file in the collection has this hash.'));
|
||||
const retired = held.length ? [] : distinct(byLayer('retired'), (item) => item.sha1 ?? item.value);
|
||||
if (!held.length && !members.length && !retired.length) blocks.push(paragraph('No file in the collection has this hash.'));
|
||||
blocks.push(...retired.map(retiredParagraph));
|
||||
if (members.length) {
|
||||
// A ROM the collection holds inside a set: the set's card, by its name.
|
||||
const list = document.createElement('ul');
|
||||
for (const item of members) {
|
||||
const row = document.createElement('li');
|
||||
row.append(code(item.name ?? item.value), ' in ', link(item.url, code(item.archive ?? '')));
|
||||
list.append(row);
|
||||
}
|
||||
blocks.push(paragraph(strong('Inside an archive of the collection')), list);
|
||||
}
|
||||
const shipped = byLayer('data-directory');
|
||||
if (shipped.length) {
|
||||
const list = document.createElement('ul');
|
||||
@@ -640,6 +662,15 @@ const renderMatches = async (status: HTMLElement, result: HTMLElement, matched:
|
||||
result.replaceChildren(...blocks);
|
||||
};
|
||||
|
||||
// A file the collection held: its address keeps answering, with the day
|
||||
// it left and the card of the file in its place.
|
||||
const retiredParagraph = (item: Item) =>
|
||||
paragraph(
|
||||
code(item.name || item.value), ` left the collection on ${item.left}. `,
|
||||
item.successor ? link(item.url, 'Open the card of the file in its place') : link(item.url, 'Open the page it was on'),
|
||||
'.',
|
||||
);
|
||||
|
||||
const WANTED = 'provenance/#wanted-catalogued-dumps-this-project-does-not-have';
|
||||
|
||||
// Nothing the site knows has the value: where a dump of one's own can go.
|
||||
@@ -749,8 +780,12 @@ const resolveFileAddress = () => {
|
||||
const matches = items.filter((item) =>
|
||||
prefix ? item.algorithm === 'sha1' && item.value.startsWith(value) : item.value === value);
|
||||
const cards = matches.filter((item) => item.layer === 'collection' || item.layer === 'data-directory');
|
||||
const retired = matches.filter((item) => item.layer === 'retired' && item.algorithm === 'sha1');
|
||||
if (new Set(cards.map((item) => item.url)).size === 1) {
|
||||
location.replace(new URL(cards[0].url, siteRoot()));
|
||||
} else if (!cards.length && retired.length) {
|
||||
status.textContent = 'This file is no longer in the collection.';
|
||||
result.replaceChildren(...distinct(retired, (item) => item.value).map(retiredParagraph));
|
||||
} else if (matches.length) {
|
||||
return renderMatches(status, result, prefix ? cards : matches);
|
||||
} else {
|
||||
@@ -1086,7 +1121,7 @@ const copyButtons = () => {
|
||||
};
|
||||
|
||||
// A system's install box: one command, for the setup picked in its list,
|
||||
// under the site's OS tabs. The box is built with the first setup's command;
|
||||
// under the site's OS tabs. The box is built with its selected setup's command;
|
||||
// the choice rewrites the options of each command and its copy text, and is
|
||||
// remembered for the next box and the next page. Without this script the
|
||||
// picker stays hidden and the list of every command shows instead.
|
||||
|
||||
@@ -7,7 +7,7 @@ order: 20
|
||||
|
||||
Adding a platform gives its users a pack and a one-line install with the files
|
||||
their setup checks, in the place it checks them, plus every file its emulators
|
||||
need that its own list leaves out. The work is to read how the platform
|
||||
need that its own list does not name. The work is to read how the platform
|
||||
declares and checks BIOS files, then teach this project to read that
|
||||
declaration. A platform can be a frontend like Batocera, a manager like EmuDeck
|
||||
or a firmware database like BizHawk.
|
||||
@@ -732,10 +732,13 @@ This executes in sequence, numbered as the run prints them (`STEP_LABELS` in
|
||||
|
||||
`--offline` skips steps 2 to 2b2.
|
||||
|
||||
A new platform is archived-by-default nowhere: `status: active` puts it in
|
||||
`list_platforms.py`, the CI matrix, and the scrapers. Set
|
||||
`status: archived` to keep the configuration without the automation. Archived
|
||||
platforms are excluded unless `--all` or `--include-archived` is passed.
|
||||
A new platform is `status: active`. `status: archived` keeps the platform
|
||||
out of the runs that cover every platform (`verify.py --all`,
|
||||
`generate_pack.py --all`, `slots.py`, `pipeline.py`, `list_platforms.py`)
|
||||
unless `--include-archived` is passed. The installer manifests, the site, the
|
||||
native export and `check_freshness.py` cover archived platforms as well. No
|
||||
tool scrapes a platform on its own, archived or not: the `schedule:` field is
|
||||
the cadence a maintainer keeps by hand.
|
||||
|
||||
A new platform has no release ZIP: `release_pack: true` is set for the setups
|
||||
most people download, and the installer serves every other one from its
|
||||
|
||||
@@ -52,6 +52,7 @@ website/ Astro Starlight project; pages, data and navigation
|
||||
_mame_clones.json MAME parent/clone set mappings
|
||||
database.json file index built from bios/ (SHA1 primary key)
|
||||
derivable.json files a held dump already contains, from provenance_report.py
|
||||
retired.json files that left the collection, written by generate_db.py
|
||||
release.json files, sizes and assets of each pack of the last release
|
||||
allowed_signers public key that signs SHA256SUMS.txt
|
||||
dist/ generated packs (gitignored)
|
||||
@@ -196,9 +197,10 @@ whoever asked for it:
|
||||
- A pack built with `--emulator` answers to that emulator, and takes its file.
|
||||
|
||||
`scripts/slots.py` prints each contested destination and the reason for its
|
||||
winner (`python scripts/slots.py --platform retrobat`). `generate_pack.py` and
|
||||
`verify.py` both decide through `slots.arbitrate()`, so the pack and the report
|
||||
agree.
|
||||
winner (`python scripts/slots.py --platform retrobat`), as `slots.arbitrate()`
|
||||
decides it. `generate_pack.py`, the install manifest and `verify.py` all read
|
||||
`slots.slot_choices()` through `slots.pack_overrides()`, which applies the same
|
||||
order, so the pack, the installer and the report agree.
|
||||
|
||||
A platform YAML is regenerated by its scraper and never edited by hand: an
|
||||
edit would be lost at the next scrape, and the file would stop describing the
|
||||
|
||||
+2
-2
@@ -77,8 +77,8 @@ setup's files and checks each one. `--dry-run` shows the list first.
|
||||
|
||||
## Fill an SD card from a computer?
|
||||
|
||||
The interface finds a muOS, Onion, spruceOS, MinUI, KNULLI or ROCKNIX card by
|
||||
itself. Name your setup and the card's BIOS folder. For RetroArch on a Switch,
|
||||
The interface finds a muOS, Onion, spruceOS, MinUI, KNULLI, ROCKNIX, Lakka or
|
||||
MiSTer card. Else name your setup and its BIOS folder. For RetroArch on a Switch,
|
||||
with the card in drive `E:`, run:
|
||||
|
||||
<!-- os-tabs windows: --platform retroarch --dest E:\retroarch\cores\system -->
|
||||
|
||||
@@ -65,7 +65,7 @@ Every module in `tests/`, by the area it guards.
|
||||
| Pack probe | the cores a platform runs for a system, staging grown to a fixed point, runs mapped to pack and manifest entries, the line and its schema, the probe lock per run, that no page shows its lines. One class builds a real pack narrowed to PlayStation and lays it out both ways, from the archive and with `install.py --dest`, and skips without the collection | `test_pack_probe` |
|
||||
| Automation runner, disk cleanup, asset sync and bit-flip repair | fake jobs, git repositories with a bare origin and session worktrees, a committed file with one bit flipped, canned CI runs in place of `gh` | `test_autorun` |
|
||||
| Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` |
|
||||
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_cited_lines`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_pin_build`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_arcade_boards`, `test_site_availability`, `test_site_bootstrap`, `test_site_built_gate`, `test_site_components`, `test_site_console_folders`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_css`, `test_site_density_cards`, `test_site_emulator_page`, `test_site_emulators`, `test_site_entry`, `test_site_entry_tables`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_integration`, `test_site_nav`, `test_site_picker`, `test_site_platform_density`, `test_site_platform_hashes`, `test_site_reference`, `test_site_render`, `test_site_search`, `test_site_system_links`, `test_site_system_names`, `test_site_system_sections`, `test_site_systems`, `test_site_validation`, `test_site_weight`, `test_site_wiki`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_version_basis`, `test_workflow_paths` |
|
||||
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_cited_lines`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_pin_build`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_retired_files`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_arcade_boards`, `test_site_availability`, `test_site_bootstrap`, `test_site_built_gate`, `test_site_components`, `test_site_console_folders`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_css`, `test_site_density_cards`, `test_site_emulator_page`, `test_site_emulators`, `test_site_entry`, `test_site_entry_tables`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_integration`, `test_site_nav`, `test_site_picker`, `test_site_platform_density`, `test_site_platform_hashes`, `test_site_reference`, `test_site_render`, `test_site_search`, `test_site_system_links`, `test_site_system_names`, `test_site_system_sections`, `test_site_systems`, `test_site_validation`, `test_site_weight`, `test_site_wiki`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_version_basis`, `test_workflow_paths` |
|
||||
| Games on the card | small DATs, listxml and setup sources, a fake fetcher, fixture install lists, and the committed data against its schemas | `test_arcade_parents`, `test_game_catalogs`, `test_game_headers`, `test_game_regions`, `test_games_index`, `test_romfolders_card`, `test_romfolders_es` |
|
||||
|
||||
## Test architecture
|
||||
@@ -463,9 +463,12 @@ refactor: extract shared utils to common.py
|
||||
|
||||
Files are staged by name, never with `git add -A` or `git add .`, which sweep in
|
||||
deletions and work that is not yours. Generated artifacts (`database.json`,
|
||||
`install/*.json`, `README.md`, `CONTRIBUTING.md`, `exports/`) are committed
|
||||
`retired.json`, `derivable.json`, `install/*.json`, `README.md`, `CONTRIBUTING.md`, `exports/`,
|
||||
the generated tables and run samples of `wiki/installer.md`) are committed
|
||||
with the data they derive from, never alone: a manifest committed before the
|
||||
files it points at gives the installer a 404.
|
||||
files it points at gives the installer a 404. The run samples count what the
|
||||
manifests hold, so `generate_installer_doc.py` runs after every manifest
|
||||
regeneration and its output goes in the same commit.
|
||||
|
||||
## CI integration
|
||||
|
||||
|
||||
+12
-7
@@ -243,6 +243,7 @@ python scripts/generate_pack.py --all --offline # cache-only, never open t
|
||||
# Install manifests (consumed by install.py)
|
||||
python scripts/generate_pack.py --all --manifest --output-dir install/
|
||||
python scripts/generate_pack.py --manifest-targets --output-dir install/targets/
|
||||
python scripts/generate_installer_doc.py # run samples of wiki/installer.md, from the manifests
|
||||
```
|
||||
|
||||
Packs include platform baseline files plus files required by the platform's cores.
|
||||
@@ -364,9 +365,11 @@ test reads the original and the updated System.dat and es_bios.xml back to
|
||||
check that the list is the diff.
|
||||
|
||||
No entry the platform declares is removed. One the truth does not pair is
|
||||
looked for in the profiles of every core the platform runs: by name or
|
||||
alias, by the basename or tail of a `path:`, by any hash (archive members
|
||||
included), under a `type: directory` entry or a data directory. An entry
|
||||
looked for, in this order: by the tail of a `path:` in the profiles of every
|
||||
core the platform runs, by a name the platform's own launch code reads (its
|
||||
`launchers:` records, such as the 22 names Batocera's DuckStation generator
|
||||
tries), then in those profiles by name or alias, by any hash (archive
|
||||
members included), under a `type: directory` entry or a data directory. An entry
|
||||
none of them answers is written to `exports/<platform>.json` under
|
||||
`removal_candidates`, with the cores examined for its system and a reason.
|
||||
The verdict is `undetermined` when a core the platform names has no
|
||||
@@ -481,7 +484,7 @@ python scripts/profile_sync.py --all --stale-before 2026-06-01 --limit 20 --tria
|
||||
python scripts/profile_sync.py --emulator vice --full-diff --ref vice/src/c64/c64rom.h --context 6
|
||||
python scripts/profile_sync.py --emulator vice --strict
|
||||
python scripts/profile_sync.py --changed --strict
|
||||
python scripts/profile_sync.py --added --base upstream/main --strict
|
||||
python scripts/profile_sync.py --added --strict
|
||||
```
|
||||
|
||||
Per part of a ref: `ANCHORED` (same content, same lines), `SHIFTED` (same
|
||||
@@ -694,8 +697,9 @@ directories are checked against their upstream, dump catalogs and romset
|
||||
snapshots against the catalog they were read from (Redump DAT versions, the
|
||||
No-Intro daily rebuild, the newest TOSEC pack, the newest MAME release, the
|
||||
git blobs of the FBNeo DATs), the game catalogs of `games/catalogs/` against
|
||||
the No-Intro pack and the Redump datfiles they were read from, and the CI
|
||||
toolchain against PyPI and the actions' latest releases.
|
||||
the No-Intro pack and the Redump datfiles they were read from, the CI
|
||||
toolchain against PyPI and the actions' latest releases, and the site's exact
|
||||
npm pins in `website/package.json` against the npm registry.
|
||||
|
||||
The `native` area needs no network. It reads the cache `export_native.py`
|
||||
patches from and reports an original that is absent, that came from another
|
||||
@@ -1330,7 +1334,8 @@ before a release (wiki/release-process.md). Its lines stay in
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `common.py` | Shared library: hash computation, file resolution, platform config loading, emulator profiles, target filtering |
|
||||
| `dedup.py` | Deduplicate `bios/` (`--dry-run`, `--bios-dir`): byte-identical copies are deleted and one kept. Identical MAME sets under other names stay one file, named in `_mame_clones.json`. Trees addressed by path are left alone (`NODEDUP_DIRS`): BBK, Bungie, C-Dogs SDL, Ikemen GO, PinMAME, RPG Maker, ScummVM, Softdisk, TheXTech, splitwolf, Theme Hospital, c64-emu, opentyrian, tzdata |
|
||||
| `retired_files.py` | The record of files that left the collection, `retired.json`, which `generate_db.py` appends to and the site's lookup reads. `--backfill` rebuilds it from the history of `database.json` since content addresses were published (`--output` names another file) |
|
||||
| `dedup.py` | Deduplicate `bios/` (`--dry-run`, `--bios-dir`): byte-identical copies are deleted and one kept. Identical MAME sets under other names stay one file, named in `_mame_clones.json`. Trees addressed by path are left alone (`NODEDUP_DIRS`), under a folder of that name at any depth: BBK, Bungie, C-Dogs SDL, Ikemen GO, mame, PinMAME, RPG Maker, ScummVM, Softdisk, TheXTech, splitwolf, Theme Hospital, c64-emu, opentyrian, tzdata |
|
||||
| `identity.py` | One content filed under two identities: groups `bios/` by content and reports a group that crosses manufacturers outside `KNOWN_SHARED` (`--strict` exits 1, pipeline step 3a) |
|
||||
| `slots.py` | Destinations the platform and a core both claim, and how the pack settles each (`--platform`, `--include-archived`, `--target`, `--json`, `--strict`, and pipeline step 3b passes the run's own `--include-archived` and `--target`) |
|
||||
| `region_audit.py` | Confront every profile `region:` with the regions No-Intro names for the same dump (`--json`, `--db`, `--emulators-dir`). An opinion, never an authority: the code decides, and a disagreement is a line to read in the emulator's source |
|
||||
|
||||
Reference in new issue
Block a user