diff --git a/README.md b/README.md index 261d7b23..98d0df9c 100644 --- a/README.md +++ b/README.md @@ -66,17 +66,17 @@ The Files column counts what a pack holds once extracted, the figure a file mana | RetroPie * | 12,533 | 8.1 GB | `~/RetroPie/BIOS/` | Download: [part 1 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part3of3.zip) | | RomM | 1,087 | 1.5 GB | `library/bios/` | [Download](../../releases/download/v2026.10.10/RomM_5.3.1_BIOS_Pack.zip) | -The AmberELEC pack already carries its own `bios/` folder, so it extracts into `/storage/roms/` rather than into the BIOS folder. +The AmberELEC pack holds its own `bios/` folder. Unzip it into `/storage/roms/`. -The EmuELEC pack already carries its own `bios/` folder, so it extracts into `/storage/roms/` rather than into the BIOS folder. +The EmuELEC pack holds its own `bios/` folder. Unzip it into `/storage/roms/`. -The MinUI pack already carries its own `Bios/` folder, so it extracts into `/` rather than into the BIOS folder. +The MinUI pack holds its own `Bios/` folder. Unzip it into `/`. -The MyMinUI pack already carries its own `Bios/` folder, so it extracts into `/` rather than into the BIOS folder. +The MyMinUI pack holds its own `Bios/` folder. Unzip it into `/`. -The NextUI pack already carries its own `Bios/` folder, so it extracts into `/` rather than into the BIOS folder. +The NextUI pack holds its own `Bios/` folder. Unzip it into `/`. -The RetroDECK pack already carries its own `bios/` folder, so it extracts into `~/retrodeck/` rather than into the BIOS folder. +The RetroDECK pack holds its own `bios/` folder. Unzip it into `~/retrodeck/`. \* Archived: the configuration is kept and packs are still built, but upstream is no longer scraped on a schedule. @@ -206,6 +206,6 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. The scripts and tooling are released under the [MIT License](LICENSE). The BIOS and firmware files are not covered by that license: they are third-party system software, preserved and provided for personal backup, archival, and interoperability with emulation software. [NOTICE](NOTICE) sets out their status and how to ask for a file to be removed. -The reasoning, and where it is weakest, is in the [FAQ](https://abdess.github.io/retrobios/wiki/faq/#is-this-legal). +The reasoning, and where it is weakest, is in [the project's legal position](https://abdess.github.io/retrobios/wiki/legal/). *Auto-generated on 2026-10-22T10:00:19Z* diff --git a/mkdocs.yml b/mkdocs.yml index 17260560..8d27f28c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -141,6 +141,7 @@ nav: - Self-hosted game library: - RomM: platforms/romm.md - Explore: + - explore.md - Systems: - systems/index.md - Consoles and computers: @@ -855,7 +856,7 @@ nav: - wiki/index.md - How each platform checks files: wiki/verification-modes.md - What the packs do not hold: gaps.md - - Cross-reference: cross-reference.md + - Cores by platform: cross-reference.md - Dump provenance: provenance.md - Updated platform files: exports.md - Data and API: data.md diff --git a/scripts/generate_readme.py b/scripts/generate_readme.py index 519d4202..3723ed55 100644 --- a/scripts/generate_readme.py +++ b/scripts/generate_readme.py @@ -92,6 +92,10 @@ def compute_coverage( "verified": ok, "untested": untested, "missing": missing, + "missing_not_firmware": { + category: len(dests) + for category, dests in (result.get("missing_not_firmware") or {}).items() + }, "present": present, "percentage": pct, "core_present": core_present, @@ -116,7 +120,7 @@ _REPO = Path(__file__).resolve().parent.parent # stores it here and the second reads it back, instead of verifying every # platform twice (about a minute of the site build). COVERAGE_CACHE_DIR = _REPO / "tmp" / "coverage-cache" -_COVERAGE_CACHE_FORMAT = "1" +_COVERAGE_CACHE_FORMAT = "2" def _coverage_inputs(db_path: Path, platforms_dir: str, emulators_dir: str) -> str: @@ -293,8 +297,8 @@ def extract_notes(platforms_dir: str) -> list[str]: display = config.get("platform", key) if config.get("base_destination") == "" and display in folders: notes.append( - f"The {display} pack already carries its own `{own_root(config)}/` folder," - f" so it extracts into `{folders[display]}` rather than into the BIOS folder." + f"The {display} pack holds its own `{own_root(config)}/` folder." + f" Unzip it into `{folders[display]}`." ) return notes @@ -810,8 +814,8 @@ def generate_readme(db: dict, platforms_dir: str, db_path: str = "database.json" " backup, archival, and interoperability with emulation software." " [NOTICE](NOTICE) sets out their status and how to ask for a file" " to be removed.", - f"The reasoning, and where it is weakest, is in the" - f" [FAQ]({SITE_URL}wiki/faq/#is-this-legal).", + f"The reasoning, and where it is weakest, is in" + f" [the project's legal position]({SITE_URL}wiki/legal/).", "", f"*Auto-generated on {ts}*", ] diff --git a/scripts/generate_site.py b/scripts/generate_site.py index 80c13779..4e33e7b3 100644 --- a/scripts/generate_site.py +++ b/scripts/generate_site.py @@ -82,6 +82,7 @@ from siteguides import ( GUIDES, Guide, INSTALL_BASE, + TERMINALS, install_command, platform_args, terminal_tabs, @@ -414,105 +415,92 @@ def generate_home( ) -> str: """The page someone lands on knowing nothing: what a BIOS file is, whether they need one, and the three steps to get the right ones. - Figures about the collection come last.""" + + The figures about the collection, the cards to the deeper pages and the + call for help sat below the steps and doubled the page: they are on the + Explore index, one tab further.""" registry = registry or {} - total_files = db.get("total_files", 0) - total_size = db.get("total_size", 0) - emulator_count = len(unique_emulator_profiles(profiles)) names = sorted((cov["platform"] for cov in coverages.values()), key=str.casefold) + named = [n for n in ("RetroArch", "Batocera", "EmuDeck") if n in names] + others = len(names) - len(named) + works_with = ", ".join(named) + (f" and {others} more" if others else "") lines = [ - "# BIOS files for your emulators, in the right folder", + "# BIOS files for your emulators", "", - "Many consoles start their games with the help of a small program built " - "into the machine, called a BIOS or firmware. An emulator needs a copy of " - "that file to start the games of those consoles. RetroBIOS gives you the " - "right files for the setup you already have, in the folder where your " - "emulators look for them.", + "Some consoles need a small program to start their games. It is called " + "the **BIOS**. Your emulator needs a copy of it. We put the right copy " + "in the right folder.", "", "[Get your files](which-pack.md){ .md-button .md-button--primary } " - "[What is a BIOS file?](wiki/what-are-bios-files.md){ .md-button }", + "[What is a BIOS?](wiki/what-are-bios-files.md){ .md-button }", "", - f"Free, with no account. It works with {', '.join(names[:-1])} and {names[-1]}.", + f"Free. No account. Works with {works_with}.", "", - "", + "## Do I need it?", "", - "## Do I need this?", + "- **Yes**, if your emulator says a BIOS is missing.", + "- **Yes** for PlayStation 1 and 2, Saturn, Dreamcast and Neo Geo.", + "- **Usually not** for NES, Super Nintendo, Mega Drive and Game Boy.", "", - "- **Yes**, if a game or an emulator says a BIOS file is missing.", - "- **Usually yes** for PlayStation, PlayStation 2, Saturn, Dreamcast, Neo " - "Geo and many home computers. Details: [the files each system " - "needs](systems/index.md).", - "- **Usually not** for NES, Super Nintendo and Mega Drive cartridges: those " - "consoles start games without a BIOS. Their add-ons do need one, such as " - "the Famicom Disk System and the Mega-CD.", + "Details: [Every console, one by one](systems/index.md)", "", "## Three steps", "", - '
', - "", - "- **1. Find your setup**", - "", - " Pick your device, or the system you play on. Its page shows the folder " - "where the files go.", - "", - " [Pick your setup](which-pack.md#pick-your-setup)", - "", - "- **2. Get the files**", - "", - " One command downloads only what is missing and checks each file. You " - "can also download a ZIP pack and extract it yourself.", - "", - " [First install, step by step](wiki/first-install.md)", - "", - "- **3. Start a game**", - "", - " Your emulator finds the files by itself. If a game still does not " - "start, the help page goes through the usual causes.", - "", - " [When a game does not start](wiki/troubleshooting.md)", - "", - "
", + "1. **Pick your device.** Its page shows your BIOS folder.", + "2. **Run one command**, or download a ZIP file.", + "3. **Start a game.**", "", "## Pick your device", "", '
', "", ] - for group, (label, description) in DEVICE_GROUPS.items(): + for group, (label, _description) in DEVICE_GROUPS.items(): members = _group_members(coverages, registry, group) if not members: continue lines += [ f"- **{label}**", "", - f" {description}", - "", " " + ", ".join( f"[{coverages[name]['platform']}](platforms/{name}.md)" for name in members ), "", - f" [See where the files go](which-pack.md#{GROUP_ANCHORS[group]})", - "", ] lines += [ "
", "", - "## Is this safe?", + "## Is it safe?", "", - "The installer writes only inside your BIOS folder, checks every file " - "before saving it, keeps any file of yours it replaces in a " - "retrobios-backup folder beside it, and never touches games or saves. " - "[How the command is checked](wiki/faq.md#is-piping-the-installer-into-a-shell-safe).", + "- It writes only in your BIOS folder.", + "- It checks every file before saving it.", + "- It keeps any file it replaces, in a `retrobios-backup` folder.", + "- It never touches your games or saves.", "", - "Laws on copying console firmware differ by country. " - "[The project's position](wiki/faq.md#is-this-legal) says where it stands " - "and how to ask for a file to be removed.", + "[How the command is checked](wiki/faq.md#is-piping-the-installer-into-a-shell-safe). " + "[Is it legal?](wiki/faq.md#is-this-legal)", "", - "## Go further", + *_next_block([ + "[Get your files](which-pack.md)", + "[What is a BIOS?](wiki/what-are-bios-files.md)", + ]), + f'
Generated on {_timestamp()}.
', + ] + return "\n".join(lines) + "\n" + + +def generate_explore_index(db: dict, coverages: dict, profiles: dict) -> str: + """The Explore tab opens here: where to look for one console, one + emulator or one installer option, and the figures of the collection.""" + total_files = db.get("total_files", 0) + total_size = db.get("total_size", 0) + emulator_count = len(unique_emulator_profiles(profiles)) + lines = [ + "# Explore", "", - "These pages go into detail, for when you want to know more than where the " - "files go.", + "One console, one emulator, or one installer option: each has its page, " + "with the files it needs and the code that reads them.", "", '
', "", @@ -520,13 +508,19 @@ def generate_home( "", " What each console and computer needs, file by file.", "", - " Details: [Browse the systems](systems/index.md)", + " [Browse the systems](systems/index.md)", "", "- **Emulators**", "", " What each emulator reads, with the source code that reads it.", "", - " Details: [Browse the emulators](emulators/index.md)", + " [Browse the emulators](emulators/index.md)", + "", + "- **Installer options**", + "", + " Fewer files, one region, another folder, a check without download.", + "", + " [All installer options](wiki/installer.md)", "", "- **How files are checked**", "", @@ -536,12 +530,14 @@ def generate_home( "", "- **Data and API**", "", - " Every file and hash as JSON, CSV and SQLite, for your own tools.", + " Every file and hash as JSON, CSV and SQLite.", "", " Reference: [Data and API](data.md)", "", "
", "", + "## The collection", + "", '
', "", *_stat(f"{total_files:,}", COLLECTION_FILES_LABEL), @@ -554,14 +550,9 @@ def generate_home( "", "## Help the project", "", - "Files, emulator profiles and corrections are welcome. " - "For contributors: [How to contribute](contributing.md).", + "A file you hold, an emulator profile or a correction fills a gap for " + "every setup at once. For contributors: [How to contribute](contributing.md).", "", - *_next_block([ - "[Get your files](which-pack.md)", - "[What BIOS files are](wiki/what-are-bios-files.md)", - ]), - f'
Generated on {_timestamp()}.
', ] return "\n".join(lines) + "\n" @@ -1429,6 +1420,7 @@ def _browser_title(relative: Path, title: str) -> str: "systems/index.md": "Systems", "emulators/index.md": "Emulators", "wiki/index.md": "Reference", + "explore.md": "Explore", } if key in index_titles: return index_titles[key] @@ -1470,10 +1462,11 @@ def _page_description(markdown: str, relative: Path, title: str) -> str: specific = { "index.md": "The BIOS files your emulators need, for RetroArch, Batocera, EmuDeck and more: the folder they go in, one command that installs and checks them, and packs to download.", "data.md": "Versioned RetroBIOS JSON API, CSV exports, SQLite snapshot, schemas, checksums and data interpretation guidance.", - "cross-reference.md": "Cross-reference from retrogaming platforms to emulator cores, systems, upstream projects and profiled firmware files.", + "cross-reference.md": "Every emulator core each retrogaming platform runs, the systems it emulates and the upstream project it comes from.", "gaps.md": "Current RetroBIOS verification gaps, platform-to-emulator divergences, missing files and documented source limitations.", "exports.md": "Each platform's own BIOS list (System.dat, batocera-systems, es_bios.xml and the rest) with the hashes and files the emulator source code shows, offered to its maintainers.", "provenance.md": "Hash-based comparison of RetroBIOS metadata with No-Intro, Redump and TOSEC catalog snapshots.", + "explore.md": "Systems, emulators and installer options to explore, with the figures of the BIOS collection.", "which-pack.md": "Pick your device or setup and get its BIOS files: the folder they go in, one command that installs and checks them, or a ZIP pack to extract.", } if key in specific: @@ -1875,7 +1868,7 @@ def page_level(relative: Path) -> str: return "start" if relative.parts[0] == "platforms": return "setup" - if relative.parts[0] in ("systems", "emulators"): + if relative.parts[0] in ("systems", "emulators") or path == "explore.md": return "explore" if path in _REFERENCE_PAGES: return "reference" @@ -1978,8 +1971,7 @@ def generate_platform_index(coverages: dict, registry: dict | None = None) -> st lines = [ "# Your setup", "", - "Pick the system you play on. Its page says where the BIOS files go, " - "gives the command that installs them and links its pack.", + "Pick what you play on. Its page gives the folder, the command and the ZIP.", "", ] for group, (label, description) in DEVICE_GROUPS.items(): @@ -2314,19 +2306,24 @@ def _core_folders_section(name: str, config: dict) -> list[str]: def _check_sentence(display: str, mode: str, config: dict, no_list: bool) -> str: """What the frontend itself checks, from the mode its code uses.""" if no_list: - return (f"{display} publishes no BIOS list, so it does not check files " - "itself. Each emulator core looks for its own files when a game starts.") + return (f"{display} has no BIOS list, so it checks nothing itself. Each " + "emulator looks for its own files when a game starts.") if config.get("hash_rule") == "size_and_any_hash": - return (f"{display} itself compares the size and a hash (MD5, SHA1 or " - "CRC32) of each file with its list, so a file with other contents " - "does not count there.") - algorithm = digest_algorithm(mode) - if algorithm: - return (f"{display} itself compares the {algorithm.upper()} of each file " - "with its list, so a file with other contents does not count there.") - return (f"{display} itself only checks that each file is there, by its name. " - "The check above also reads the contents, so it finds a file that has " - "the right name and other contents.") + return (f"{display} checks each file's size and fingerprint too. It " + "reports a file with the wrong size or contents.") + if digest_algorithm(mode): + return (f"{display} checks each file's fingerprint too. It reports a file " + "with the wrong contents.") + return (f"{display} only checks that a file with the right name is there. " + "`--check` also checks what is inside.") + + +def _folder_rows(rows: tuple[tuple[str, str], ...], indent: str = "") -> list[str]: + return [ + f"{indent}| Where | BIOS folder |", + f"{indent}|-------|-------------|", + *(f"{indent}| {label} | {path} |" for label, path in rows), + ] def _platform_guide(name: str, display: str, cov: dict, registry: dict, no_list: bool) -> list[str]: @@ -2334,15 +2331,9 @@ def _platform_guide(name: str, display: str, cov: dict, registry: dict, no_list: guide = GUIDES[name] entry = registry.get(name, {}) extract_to = entry.get("extract_to") - lines = [ - "## Where the files go", - "", - "| Where | BIOS folder |", - "|-------|-------------|", - *(f"| {label} | {path} |" for label, path in guide.folders), - ] + lines = ["## Where the files go", "", *_folder_rows(guide.folders)] if extract_to: - lines.append(f"| Extract the pack into | `{extract_to}` |") + lines.append(f"| Unzip the pack into | `{extract_to}` |") lines.append("") for note in guide.notes: lines += [note, ""] @@ -2350,35 +2341,38 @@ def _platform_guide(name: str, display: str, cov: dict, registry: dict, no_list: for line in (note, "")] if name == "retroarch": lines += [ - "Handheld systems that start RetroArch for their games keep its files " - "in the folder below. The command finds it by itself.", + '??? note "Other builds: Microsoft Store, Mac Steam, handhelds"', "", - "| System | BIOS folder |", - "|--------|-------------|", - *(f"| {system} | `{folder}` |" for system, folder in _retroarch_handhelds(registry)), + *_folder_rows(guide.more_folders, " "), + *(f" | {system} | `{folder}` |" for system, folder in _retroarch_handhelds(registry)), + "", + " LaunchBox and other launchers use the folder RetroArch reads.", + "", + ] + elif guide.more_folders: + lines += [ + '??? note "Other layouts"', + "", + *_folder_rows(guide.more_folders, " "), "", ] lines += [ *(["", ""] if guide.on_card else []), "## Install", "", - f"The command finds the {display} BIOS folder, downloads the files that " - "are missing or differ, checks each one, and resumes where it stopped.", + "The command finds the folder, downloads what is missing, and checks " + "each file.", "", ] if guide.on_card: lines += [ - f"Put the {display} SD card in a computer. On Windows, note its drive " - "letter (for example **E:**). Then run the line with the root of the " - "card after `--dest`.", + "Put the SD card in a computer. On Windows, note its letter, like " + "**E:**. Put the card's path after `--dest`.", "", ] lines += [ *terminal_tabs(guide.run, platform_args(name, guide)), *trust_box("../"), - "To install fewer systems, add `--system` with the system ids listed in " - "the reference below.", - "", "## Download", "", ] @@ -2386,21 +2380,26 @@ def _platform_guide(name: str, display: str, cov: dict, registry: dict, no_list: pack = release_record.pack_for(name, record) if pack and pack.get("assets"): parts = len(pack["assets"]) + bullets = [ + f"{_fmt_size(pack['download_size'])} to download, " + f"{_fmt_size(pack['extracted_size'])} once unzipped." + ] + if parts > 1: + bullets.append("You need both parts." if parts == 2 else f"You need all {parts} parts.") + target = f"`{extract_to}`" if extract_to else "your BIOS folder" + bullets.append(f"Unzip {'them all' if parts > 1 else 'it'} into {target}.") + if guide.download_hint: + bullets.append(guide.download_hint) lines += [ _pack_buttons(name, f"Download {display} Pack"), "", - f"{_fmt_size(pack['download_size'])} to download" - + (f" in {parts} parts" if parts > 1 else "") - + f", {_fmt_size(pack['extracted_size'])} once extracted. " - + (f"Extract it into `{extract_to}`. " if extract_to else "") - + ("Every part is needed: extract them all into the same folder." - if parts > 1 else ""), + *(f"- {b}" for b in bullets), "", ] elif record.get("tag"): lines += [ - f"The {record['tag']} release has no {display} pack yet. The command " - "above installs the same files.", + f"No {display} pack in release {record['tag']} yet. The command above " + "installs the same files.", "", ] else: @@ -2409,19 +2408,16 @@ def _platform_guide(name: str, display: str, cov: dict, registry: dict, no_list: lines += [ "## How to check", "", - "The same command with `--check` reads the BIOS folder, checks every file " - "and writes nothing. It ends on a line such as " - "`5865/5865 present (5865 verified, 0 wrong hash)`.", + "Add `--check` to the command. It changes nothing.", "", *terminal_tabs(guide.run, check_args, steps=False), + "All good when you see `0 wrong hash`.", + "", _check_sentence(display, cov["mode"], cov["config"], no_list), "", *_next_block([ - "[First install, step by step](../wiki/first-install.md), with what " - "each step prints", + "[First install, step by step](../wiki/first-install.md)", "[When a game does not start](../wiki/troubleshooting.md)", - "[Every file by system](#reference-every-file-by-system), further down " - "this page", ]), ] return lines @@ -2470,9 +2466,7 @@ def generate_platform_page( [ '!!! warning "Archived platform"', "", - " The configuration is kept and packs are still built, but " - "upstream is no longer scraped on a schedule, so the file list " - "reflects the last sync rather than today's upstream.", + f" {display}'s list is kept as last read. Its pack is still built.", "", ] ) @@ -2490,9 +2484,11 @@ def generate_platform_page( "", level_label("reference"), "", - "What follows is for checking the work: the counts, where the list was " - "read from, and each file with its hashes.", + "For checking the work: the counts, where the list comes from, and " + "every file with its hashes.", "", + *(line for note in (GUIDES[name].reference_notes if name in GUIDES else ()) + for line in (note, "")), ] if no_list: # MinUI and its forks hand each core a folder and name no file: a @@ -3061,7 +3057,7 @@ def generate_provenance_page( "not by catalog membership. Plenty of files this project ships are " "outside any catalog by nature: composites a core assembles for itself " "(the MiSTer X68000 `boot.rom` is `cgrom.dat` joined to `iplrom.dat`), " - "modern console firmware updates, game data such as `prboom.wad`, and " + "modern console firmware updates, game data such as Diablo's `diabdat.mpq`, and " "arcade sets tracked by MAME driver source instead.", "", "When the two views disagree, this project follows the code, because " @@ -4251,14 +4247,16 @@ def _render_metadata_rows(profile: dict) -> list[str]: ("resources_directory", "Resources directory"), ("rom_path", "ROM path"), ("game_count", "Game count"), - ("verification", "Checked by"), + ("verification", "Checks the file by"), ("analysis_date", "Analysis date"), ("analysis_commit", "Analysis commit"), ]: val = profile.get(field) if val is None or val == "" or isinstance(val, (dict, list)): continue - if isinstance(val, str) and val.startswith("http"): + if field == "bios_size" and isinstance(val, int): + lines.append(f"| {label} | {_fmt_size(val)} ({val:,} bytes) |") + elif isinstance(val, str) and val.startswith("http"): lines.append(f"| {label} | [{val}]({val}) |") else: lines.append(f"| {label} | {val} |") @@ -5008,9 +5006,10 @@ def _render_problem_files( if p["status"] == "missing": entry = missing_files.setdefault(fname, { "name": fname, "required": p["required"], - "platforms": [], "reason": p["reason"], + "platforms": [], "reason": p["reason"], "categories": set(), }) entry["platforms"].append(p["platform"]) + entry["categories"].add(p.get("category", "bios")) if p["required"]: entry["required"] = True elif p["status"] == "untested": @@ -5038,7 +5037,7 @@ def _render_problem_files( "## Problem files", "", f"{len(missing_files)} missing, {len(untested_files)} untested, " - f"{len(mismatch_files)} hash mismatch.", + f"{len(mismatch_files)} discrepancy.", "", ]) @@ -5047,14 +5046,22 @@ def _render_problem_files( f'### Missing ' f"{_count(len(missing_files), 'file')}", "", - "| File | Required | Platforms |", - "|------|----------|-----------|", + "Required is the platform's own flag. Category says what the " + "platform's emulators do with the file: `bios` is firmware they " + "load, `game_data` comes from a copy of the game, `unread` is " + "opened by none of them. Only a missing `bios` entry stops a " + "console from starting.", + "", + "| File | Category | Required | Platforms |", + "|------|----------|----------|-----------|", ]) for fname in sorted(missing_files): f = missing_files[fname] req = "yes" if f["required"] else "no" plats = ", ".join(sorted(set(f["platforms"]))) - lines.append(f"| `{fname}` | {req} | {plats} |") + cats = f["categories"] + category = "bios" if "bios" in cats else ", ".join(sorted(cats)) + lines.append(f"| `{fname}` | {category} | {req} | {plats} |") lines.append("") if untested_files: @@ -5062,7 +5069,7 @@ def _render_problem_files( f'### Untested ' f"{_count(len(untested_files), 'file')}", "", - "Present but hash not verified. Declared is what the platform " + "Present, but not the expected hash. Declared is what the platform " "asks for, the collection copy is what the path holds.", "", "| File | Platforms | Reason | Declared | Collection copy |", @@ -5081,10 +5088,11 @@ def _render_problem_files( if mismatch_files: lines.extend([ - f'### Hash mismatch ' + f'### Discrepancy ' f"{_count(len(mismatch_files), 'file')}", "", - "Platform says OK but emulator validation disagrees.", + "The platform's own check passes, but the emulator's size or " + "hash check fails. Each row names the check that fails.", "", "| File | Platforms | Discrepancy | Accepted dump in the collection |", "|------|----------|-------------|-------------------|", @@ -5118,25 +5126,44 @@ def _render_problem_files( return lines -def _platform_lists_sentence(total: int, verified: int, untested: int, missing: int) -> str: +def _platform_lists_sentence( + total: int, verified: int, untested: int, missing: int, + not_firmware: dict[str, int] | None = None, +) -> str: """The platforms' own lists in one sentence, every count said. Four tiles opened the page: "Verified (100%)" for 5,437 of 5,438 and "Missing 0", above a table of red cells, on the page about what is missing. The gaps are in the files the emulators load, and the sentence - says so. + says so. The missing count then read as missing firmware, while every + entry behind it was game data or a file no emulator opens: the split by + category is said with it. """ def _is(count: int) -> str: return "is" if count == 1 else "are" + not_firmware = {k: v for k, v in (not_firmware or {}).items() if v} parts = [f"{verified:,} {_is(verified)} held and pass the platform's own check"] if untested: parts.append(f"{untested:,} {_is(untested)} held but not verified " "(listed under Untested below)") parts.append(f"{missing:,} {_is(missing)} missing") listed = ", ".join(parts[:-1]) + f" and {parts[-1]}" + split = "" + if missing and not_firmware: + labels = {"game_data": "game data", + "unread": "opened by none of the platform's emulators"} + firmware = missing - sum(not_firmware.values()) + pieces = [f"{firmware:,} firmware"] + [ + f"{count:,} {labels.get(category, category)}" + for category, count in sorted(not_firmware.items()) + ] + split = ( + f" Of the missing, {', '.join(pieces[:-1])} and {pieces[-1]}: " + "only missing firmware stops a console from starting." + ) return ( - f"Of the {total:,} files on the platforms' own BIOS lists, {listed}. " + f"Of the {total:,} files on the platforms' own BIOS lists, {listed}.{split} " "The files the emulators load beyond those lists are counted per " "platform in the table below." ) @@ -5169,6 +5196,7 @@ def generate_gap_analysis( total_untested = 0 total_missing_verify = 0 total_files_verify = 0 + not_firmware_totals: dict[str, int] = {} platform_problems: list[dict] = [] settled_problems: list[dict] = [] @@ -5177,6 +5205,8 @@ def generate_gap_analysis( total_untested += cov["untested"] total_missing_verify += cov["missing"] total_files_verify += cov["total"] + for category, count in (cov.get("missing_not_firmware") or {}).items(): + not_firmware_totals[category] = not_firmware_totals.get(category, 0) + count for d in cov["details"]: if d.get("settled_by") and d["status"] != "ok": @@ -5195,6 +5225,7 @@ def generate_gap_analysis( "name": d["name"], "status": d["status"], "required": d.get("required", True), + "category": d.get("category", "bios"), "reason": d.get("reason", ""), "discrepancy": d.get("discrepancy", ""), "system": d.get("system", ""), @@ -5205,15 +5236,24 @@ def generate_gap_analysis( lines = [ "# Gap analysis", "", - "What each platform and its emulators still lack, and where the " - "collection's copies come from.", + "What each pack still cannot give you, and why. Most gaps are not " + "firmware: a platform's list can also name game files, such as " + "Diablo's `diabdat.mpq`, which you copy from your own game, or files " + "no emulator of that platform opens. Those are counted apart. A missing " + "firmware file means a console that does not start, and each one is " + "named below with the platforms that list it.", + "", + "Each file here is a request: if you hold one of the files under " + "\"Files to source\", [adding it](contributing.md) fills the gap for " + "every platform at once.", "", "[Download gaps CSV](downloads/gaps.csv){ .md-button } " "[Open gaps API](api/v1/gaps.json){ .md-button } " "[All data exports](data.md){ .md-button }", "", _platform_lists_sentence( - total_files_verify, total_verified, total_untested, total_missing_verify + total_files_verify, total_verified, total_untested, total_missing_verify, + not_firmware_totals, ), "", ] @@ -5259,7 +5299,7 @@ def generate_gap_analysis( "required and optional files alike since both ship. The first column " "is the BIOS list the platform publishes. The second counts files its " "emulators load that this list never mentions, found by reading their " - "source code, and it is routinely several times larger. A short " + "source code, and it is often several times larger. A short " "fraction is flagged and named in the sections below. That second " "number is a floor, not a ceiling: an emulator that accepts any file " "handed to it names none in its code, so nothing there can be counted. " @@ -5267,7 +5307,7 @@ def generate_gap_analysis( "from its source code " "([how each one works](wiki/verification-modes.md)).", "", - "## Corroboration against emulator source", + "## How much of each list the emulator code confirms { #corroboration-against-emulator-source }", "", "| Platform | On its BIOS list | Documented in a profile | Content the code checks |", "|----------|-----------------:|------------------------:|------------------------:|", @@ -5402,7 +5442,7 @@ def generate_gap_analysis( ) lines.extend([ - "## Core complement", + "## Files emulators load that no platform lists { #core-complement }", "", f"Files loaded by emulators but not declared by any platform. " f"{total_undeclared:,} files across {_count(len(emulator_gaps), 'emulator')}, " @@ -5416,7 +5456,7 @@ def generate_gap_analysis( "Whatever is not in the collection is an acquisition target, named " "per emulator below.", "", - "### Provenance", + "### Where the files come from", "", "| Source | Count | Description |", "|--------|------:|-------------|", @@ -5785,14 +5825,17 @@ def generate_cross_reference( }) lines = [ - "# Cross-reference", + "# Cores by platform", "", - f"Platform > Core > Systems > Upstream emulator. " - f"{total_cores} cores across {len(coverages)} platforms, " - f"tracing back to {total_upstreams} upstream projects.", + "Every core each platform runs, the systems it emulates, and the project " + "it comes from. A core built for RetroArch is often a port of another " + "emulator, and the two can differ in the files they read and the checks " + "they make, so each core names its upstream project. " + f"{total_cores} cores across {len(coverages)} platforms come from " + f"{total_upstreams} upstream projects.", "", - "The libretro core is a port of the upstream emulator. " - "Files, features, and validation may differ between the two.", + "The files a platform's emulators load and its own list does not name are " + "counted on [What the packs do not hold](gaps.md).", "", "[Download cross-reference CSV](downloads/cross-reference.csv){ .md-button } " "[Open emulator API](api/v1/emulators.json){ .md-button } " @@ -6095,58 +6138,49 @@ def _group_keys(registry: dict, group: str, display: dict[str, str]) -> list[str ) +def _own_root_folders(registry: dict) -> dict[str, str]: + """Platforms whose pack holds its own top folder, with where it unzips. + + The same rule as extract_notes: a list written from an empty base + destination carries its root in every path.""" + folders = {} + for key, entry in registry.items(): + config = load_platform_config(key, "platforms") + if config.get("base_destination") == "" and entry.get("extract_to"): + folders[key] = entry["extract_to"] + return folders + + def generate_which_pack() -> str: """The device picker: every registered platform, grouped by the device it - runs on, with its BIOS folder, its pack and its page.""" + runs on, with its BIOS folder, its pack and its page. + + The command comes first: it finds the setup by itself, and the tables + below it serve the reader who downloads a ZIP.""" registry = load_platform_registry("platforms") record = release_record.load_record() display = {key: load_platform_config(key, "platforms").get("platform", key) for key in registry} - check_sh, check_ps = install_command("sh", "--check"), install_command("powershell", "--check") - card = install_command("sh", "--platform retroarch --dest /path/to/sdcard/retroarch/cores/system") + own_root = _own_root_folders(registry) lines = [ "# Get your files", "", - "Each setup keeps its BIOS files in a folder of its own. Pick yours below " - "to see where that folder is. Then get the files in one of two ways:", + "Two ways to fill your BIOS folder.", "", - "- **One command**, the easiest. It finds your setup and its BIOS folder, " - "downloads only the files that are missing, and checks each one before " - "saving it.", - "- **A ZIP pack.** You download the pack of your setup and extract it into " - "the BIOS folder yourself.", - "", - "", - "", - "## Pick your setup", - "", - '
', - "", - ] - for group, (label, description) in DEVICE_GROUPS.items(): - members = _group_keys(registry, group, display) - if not members: - continue - lines += [ - f"- **{label}**", - "", - f" {description}", - "", - f" [{', '.join(display[k] for k in members)}](#{GROUP_ANCHORS[group]})", - "", - ] - lines += [ - "
", + "- **[One command](#quick-install)**: finds your folder and checks each " + "file.", + "- **[A ZIP file](#manual-download)**: download it, unzip it into your " + "BIOS folder.", "", "## One command { #quick-install }", "", - "Run this line in a terminal. It finds your setup by itself. On a system " - "that starts into a game menu, such as Batocera, the line runs on the " - "device: the page of your setup gives the exact command.", + "A **terminal** is a window where you type commands.", "", *terminal_tabs(("windows", "macos", "linux", "steamdeck", "android"), {}), + "On Batocera, Recalbox and other game systems, run it on the device. " + "Your setup's page shows how.", + "", *trust_box(""), - "At a terminal, the installer shows the setup it found and asks what to " - "install:", + "It then asks what to install. Press **Enter** for everything.", "", "```", " Enter) install everything", @@ -6155,30 +6189,38 @@ def generate_which_pack() -> str: " q) quit", "```", "", - "Press **Enter** to install everything. Press **c** to pick only some " - "consoles. The **i** line appears when the installer finds emulator cores " - "installed, and keeps only the files they read.", + "**Check your folder later.** Add `--check`. It changes nothing.", "", - "To see what is already in your BIOS folder, add `--check`. It reads the " - "folder, checks every file and writes nothing:", + *terminal_tabs(("windows", "macos", "linux", "steamdeck", "android"), + dict.fromkeys(TERMINALS, "--check"), steps=False), + '??? note "Fill an SD card from a computer"', "", - "```sh", - check_sh, - "```", + " Name your setup and the BIOS folder on the card. For RetroArch on a " + "Nintendo Switch:", "", - "```powershell", - check_ps, - "```", + " ```sh", + " " + install_command( + "sh", "--platform retroarch --dest /path/to/sdcard/retroarch/cores/system"), + " ```", "", - "A memory card from another device can be filled from a computer: name the " - "setup with `--platform` and the BIOS folder on the card with `--dest`. " - "For RetroArch on a Nintendo Switch:", + "Details: [All installer options](wiki/installer.md).", "", - "```sh", - card, - "```", + "## Pick your setup", + "", + '
', "", ] + for group, (label, _description) in DEVICE_GROUPS.items(): + members = _group_keys(registry, group, display) + if not members: + continue + lines += [ + f"- **{label}**", + "", + f" [{', '.join(display[k] for k in members)}](#{GROUP_ANCHORS[group]})", + "", + ] + lines += ["
", ""] for group, (label, description) in DEVICE_GROUPS.items(): members = _group_keys(registry, group, display) if not members: @@ -6188,60 +6230,57 @@ def generate_which_pack() -> str: "", description, "", - "| Setup | What it is | Runs on | BIOS folder | Pack |", - "|-------|-----------|---------|-------------|------|", + "| Setup | Runs on | BIOS folder | ZIP |", + "|-------|---------|-------------|-----|", ] for key in members: guide = GUIDES[key] + folder = (f"Unzip into `{own_root[key]}`" if key in own_root + else _folder_cell(key, guide)) lines.append( - f"| [{display[key]}](platforms/{key}.md) | {guide.summary} | " - f"{_runs_on(registry[key], guide)} | {_folder_cell(key, guide)} | " + f"| [{display[key]}](platforms/{key}.md) | " + f"{_runs_on(registry[key], guide)} | {folder} | " f"{_pack_cell(key, record)} |" ) - for setup, summary, uses, runs_on, folder in ALSO_RUNS.get(group, ()): + for setup, _summary, uses, runs_on, folder in ALSO_RUNS.get(group, ()): lines.append( - f"| {setup} | {summary} It uses the [{display[uses]}](platforms/{uses}.md) " - f"files. | {runs_on} | {folder} | {_pack_cell(uses, record)} |" + f"| {setup} | {runs_on} | {folder}, same as " + f"[{display[uses]}](platforms/{uses}.md) | {_pack_cell(uses, record)} |" ) lines.append("") if group == "handheld": lines += [ - "Many other handheld systems start RetroArch for their games. On " - "them, the RetroArch files go in the folder below, and the command " - "finds it by itself.", + '??? note "Other handhelds that run RetroArch"', "", - "| System | Runs | BIOS folder |", - "|--------|------|-------------|", - *(f"| {name} | [RetroArch](platforms/retroarch.md) | `{folder}` |" + " They use the RetroArch files, in the folder below. The command " + "finds it.", + "", + " | System | Runs | BIOS folder |", + " |--------|------|-------------|", + *(f" | {name} | [RetroArch](platforms/retroarch.md) | `{folder}` |" for name, folder in _retroarch_handhelds(registry)), "", ] lines += [ - "## Download a pack { #manual-download }", + "## Download a ZIP { #manual-download }", "", - f"Each pack in the tables above comes from the [latest release]({RELEASE_URL}). " - "Download the pack of your setup and extract it into its BIOS folder. " - "This site can describe files newer than that release: a release is " - "published by hand, once its checks pass.", + "1. Find your setup in the tables above.", + "2. Download its ZIP. A big one comes in parts, like " + "`Pack.part1of2.zip` and `Pack.part2of2.zip`. Download every part.", + "3. Unzip every part into the folder named in the table.", "", - "A pack over 2 GB comes in several parts (`PackName.part1of2.zip`, " - "`PackName.part2of2.zip`), and every part is needed. Each part is an " - "ordinary ZIP: extract them all into the same folder.", + "A row that says **Unzip into** has a ZIP that holds its own top folder. " + "Unzip it there, not into the BIOS folder.", "", - *[line for note in extract_notes("platforms") for line in (note, "")], - "## Fewer files", + f"The ZIPs come from the [latest release]({RELEASE_URL}). The command may " + "install newer files.", "", - "Each pack holds everything its setup runs: the BIOS list the setup " - "publishes, and every file its emulators load. The command can install " - "less: press **c** when it asks, or name the consoles with `--system`.", + "## Only some consoles?", "", - "Details: [All installer options](wiki/installer.md#choosing-what-to-install). " - "Reference: [Build your own packs](wiki/advanced-usage.md).", + "Run the command and press **c** when it asks.", "", *_next_block([ "[First install, step by step](wiki/first-install.md)", - "The page of your setup, from the tables above: where the files go and " - "how to check them", "[When a game does not start](wiki/troubleshooting.md)", ]), ] @@ -6349,6 +6388,7 @@ def generate_mkdocs_nav( ]}, {"Your setup": _platform_nav(coverages, registry or {})}, {"Explore": [ + "explore.md", {"Systems": _system_nav(manufacturers)}, {"Emulators": _emulator_nav(profiles)}, {"All installer options": "wiki/installer.md"}, @@ -6357,7 +6397,7 @@ def generate_mkdocs_nav( "wiki/index.md", {"How each platform checks files": "wiki/verification-modes.md"}, {"What the packs do not hold": "gaps.md"}, - {"Cross-reference": "cross-reference.md"}, + {"Cores by platform": "cross-reference.md"}, {"Dump provenance": "provenance.md"}, {"Updated platform files": "exports.md"}, {"Data and API": "data.md"}, @@ -6502,6 +6542,8 @@ def _generate(args: argparse.Namespace) -> None: ) ) + write_if_changed(str(docs / "explore.md"), generate_explore_index(db, coverages, profiles)) + stats = compute_stats(db, coverages, profiles) stats["composition"] = compute_composition(db) write_if_changed(str(docs / "stats.json"), generate_stats(stats)) diff --git a/scripts/siteguides.py b/scripts/siteguides.py index 3b8b87f6..b9c06631 100644 --- a/scripts/siteguides.py +++ b/scripts/siteguides.py @@ -21,25 +21,23 @@ INSTALL_BASE = "https://raw.githubusercontent.com/Abdess/retrobios/main" DEVICE_GROUPS: dict[str, tuple[str, str]] = { "computer": ( "Computers and Steam Deck", - "An emulator app on a Windows, Mac or Linux computer, or on a Steam Deck.", + "An emulator app on a computer or a Steam Deck.", ), "gaming-os": ( "Game systems for PCs and small boards", - "A system that starts straight into a game menu, installed on a PC, a " - "Raspberry Pi or another single-board computer.", + "A system that starts straight into a game menu, on a PC or a Raspberry Pi.", ), "handheld": ( "Handhelds and TV boxes", - "A handheld console or a TV box, and the system it starts when you turn it on.", + "The system your handheld or TV box starts.", ), "fpga": ( "MiSTer FPGA", - "A MiSTer board, which recreates console hardware in a chip instead of " - "emulating it in software.", + "A board that recreates old consoles in hardware, not in software.", ), "server": ( "Self-hosted game library", - "A RomM server, which plays your games in a web browser.", + "A RomM server. It plays games in your web browser.", ), } @@ -72,43 +70,43 @@ ALSO_RUNS: dict[str, tuple[tuple[str, str, str, str, str], ...]] = { TERMINALS: dict[str, tuple[str, str, str]] = { "windows": ( "Windows", - "Open the Start menu, type **PowerShell** and open it. Paste the line " - "and press Enter.", + "Open the Start menu. Type **PowerShell** and open it. Paste the line. " + "Press Enter.", "powershell", ), "macos": ( "macOS", - "Open **Terminal**, in Applications then Utilities. Paste the line and " - "press Return.", + "Open **Terminal**, in Applications, then Utilities. Paste the line. " + "Press Return.", "sh", ), "linux": ( "Linux", - "Open a terminal. Paste the line and press Enter.", + "Open a terminal. Paste the line. Press Enter.", "sh", ), "steamdeck": ( "Steam Deck", - "Switch to Desktop Mode and open **Konsole**, the terminal. Paste the " - "line with Ctrl+Shift+V and press Enter.", + "Switch to Desktop Mode. Open **Konsole**. Paste with Ctrl+Shift+V. " + "Press Enter.", "sh", ), "android": ( "Android (Termux)", - "Install **Termux**, then run `pkg install python` and " - "`termux-setup-storage` once. Paste the line and press Enter.", + "Install **Termux**, a terminal app. Run `pkg install python` and " + "`termux-setup-storage` once. Then paste the line. Press Enter.", "sh", ), "device": ( "On the device", - "Open a terminal on the device, or connect to it from a computer on the " - "same network with SSH. Paste the line and press Enter.", + "Open a terminal on the device. Or connect from a computer with SSH, a " + "terminal over the network. Paste the line. Press Enter.", "sh", ), "server": ( "On the server", - "Open a terminal on the machine that holds the RomM library. Give " - "`--dest` the `bios` folder of that library.", + "Open a terminal on the machine that holds your RomM library. Put the " + "path of its `bios` folder after `--dest`.", "sh", ), } @@ -119,7 +117,14 @@ CARD_TERMINALS = ("macos", "linux", "windows") @dataclass(frozen=True) class Guide: - """What a platform page says before its file reference.""" + """What a platform page says before its file reference. + + `folders` are the common cases, `more_folders` the rarer builds shown + folded under them. `notes` stay in the guide, one plain fact each. + `reference_notes` carry the source citations and the details a reader + checking the work wants, and open the Reference part of the page. + `download_hint` is one more bullet under the pack download. + """ summary: str folders: tuple[tuple[str, str], ...] @@ -128,79 +133,82 @@ class Guide: dest: str = "" on_card: bool = False extra_args: dict[str, str] = field(default_factory=dict) + more_folders: tuple[tuple[str, str], ...] = () + reference_notes: tuple[str, ...] = () + download_hint: str = "" +NETWORK_HINT = "From a computer, use the network folder above." + GUIDES: dict[str, Guide] = { "retroarch": Guide( - summary="A multi-system emulator. It loads a different core for each console.", + summary="An app that runs many consoles. It uses a different **core**, " + "a small emulator, for each one.", folders=( - ("Windows, installer or portable build", "`system\\` next to `retroarch.exe`"), - ("Windows, Microsoft Store build", - "`%LOCALAPPDATA%\\Packages\\1e4cf179-f3c2-404f-b9f3-cb2070a5aad8_*\\LocalState\\system\\`"), + ("Windows", "`system\\` next to `retroarch.exe`"), ("macOS", "`~/Documents/RetroArch/system/`"), - ("macOS, Steam or portable build", "`~/Library/Application Support/RetroArch/system/`"), ("Linux", "`~/.config/retroarch/system/`"), - ("Linux Flatpak and Steam Deck", "`~/.var/app/org.libretro.RetroArch/config/retroarch/system/`"), + ("Steam Deck and Linux Flatpak", "`~/.var/app/org.libretro.RetroArch/config/retroarch/system/`"), ("Android", "`RetroArch/system/` on the internal storage"), ("Nintendo Switch", "`retroarch/cores/system/` on the SD card"), ), + more_folders=( + ("Windows, Microsoft Store build", + "`%LOCALAPPDATA%\\Packages\\1e4cf179-f3c2-404f-b9f3-cb2070a5aad8_*\\LocalState\\system\\`"), + ("macOS, Steam or portable build", "`~/Library/Application Support/RetroArch/system/`"), + ), run=("windows", "macos", "linux", "steamdeck", "android"), notes=( - "RetroArch shows the folder it reads in **Settings > Directory > " - "System/BIOS**. The same value is the `system_directory` line of " - "`retroarch.cfg`.", + "**Not sure?** In RetroArch, open **Settings > Directory > " + "System/BIOS**. It shows the folder.", + ), + reference_notes=( + "RetroArch reads its BIOS folder from the `system_directory` line of " + "`retroarch.cfg`, the value **Settings > Directory > System/BIOS** " + "shows.", "On Windows, `retroarch.cfg` is read next to `retroarch.exe` first, " "then from `%APPDATA%\\RetroArch\\`. A `system_directory` left at its " "default is the `system` folder next to `retroarch.exe` (`:\\system` " "in RetroArch's `frontend/drivers/platform_win32.c`, where `:` stands " "for the folder of the program).", - "LaunchBox and other launchers that start RetroArch use the folder " - "RetroArch itself reads.", ), ), "batocera": Guide( - summary="A system that starts straight into a game menu. It runs on PCs, " - "Raspberry Pi boards and handhelds.", + summary="A system that starts straight into a game menu.", folders=( ("On Batocera", "`/userdata/bios/`"), - ("From Windows, over the network", "`\\\\BATOCERA\\share\\bios\\`"), - ("From macOS or Linux, over the network", "`smb://batocera/share/bios/`"), + ("From Windows, over your home network", "`\\\\BATOCERA\\share\\bios\\`"), + ("From a Mac or Linux, over your home network", "`smb://batocera/share/bios/`"), ), run=("device",), - notes=( - "From a computer on the same network, the network folder is the " - "easiest way in: download the pack and extract it there.", - ), + download_hint=NETWORK_HINT, ), "recalbox": Guide( summary="A system that starts straight into a game menu and sets up its " - "emulators itself.", + "emulators.", folders=( ("On Recalbox", "`/recalbox/share/bios/`"), - ("From Windows, over the network", "`\\\\RECALBOX\\share\\bios\\`"), + ("From Windows, over your home network", "`\\\\RECALBOX\\share\\bios\\`"), ), run=("device",), - notes=( - "From a computer on the same network, download the pack and extract " - "it into the network folder.", - ), + download_hint=NETWORK_HINT, ), "retrobat": Guide( - summary="A Windows frontend built on EmulationStation. It comes with " - "RetroArch and standalone emulators.", + summary="A game menu for Windows. It comes with RetroArch and other " + "emulators.", folders=( ("Inside the RetroBat folder", "`bios\\`"), ("Default install", "`C:\\RetroBat\\bios\\`"), ), run=("windows",), - notes=( + notes=("The command finds RetroBat wherever you installed it.",), + reference_notes=( "RetroBat records its own folder each time it starts, and the " - "installer reads it there, wherever RetroBat was installed.", + "installer reads it there.", ), ), "emudeck": Guide( - summary="Installs and sets up many emulators, and adds each game to the " - "Steam library.", + summary="Installs your emulators and adds your games to Steam.", folders=( ("Steam Deck and Linux", "`~/Emulation/bios/`"), ("Windows, by default", "`C:\\Emulation\\bios\\`"), @@ -208,22 +216,24 @@ GUIDES: dict[str, Guide] = { ), run=("steamdeck", "linux", "windows", "macos"), notes=( - "The `bios` folder sits inside the Emulation folder chosen in " - "EmuDeck. When EmuDeck was set up on an SD card, that folder is on " - "the card. The installer reads the location from EmuDeck's settings " + "The `bios` folder is inside the Emulation folder you chose in " + "EmuDeck. If you chose the SD card, it is on the card.", + ), + reference_notes=( + "The installer reads the Emulation folder from EmuDeck's settings " "(`emulationPath` in `~/.config/EmuDeck/settings.sh`).", ), ), "lakka": Guide( - summary="A system built around RetroArch. It starts straight into the " - "RetroArch menu.", + summary="A system that starts straight into RetroArch.", folders=(("On Lakka", "`/storage/system/`"),), run=("device",), - notes=("The folder is reachable over SSH and over the network share.",), + notes=("You can also reach the folder from a computer, over your home network.",), + reference_notes=("The folder is reachable over SSH and over the network share.",), ), "retrodeck": Guide( - summary="One app with its emulators included, installed from Flathub " - "or Discover.", + summary="One app with its emulators inside. Install it from Flathub or " + "Discover.", folders=( ("Steam Deck and Linux", "`~/retrodeck/bios/`"), ("Installed on a microSD card", "for example `/run/media/mmcblk0p1/retrodeck/bios/`"), @@ -231,101 +241,111 @@ GUIDES: dict[str, Guide] = { run=("steamdeck", "linux"), ), "romm": Guide( - summary="A web-based game library. It plays games in the browser " - "through EmulatorJS.", - folders=( - ("In the RomM library", "`bios/`, one subfolder per platform"), + summary="A game library on your own server. It plays games in the web " + "browser.", + folders=(("In the RomM library", "`bios/`, one subfolder per platform"),), + more_folders=( ("With `firmware: \"{platform}/bios\"` in `config.yml`", "`/bios/` for each platform"), ), run=("server",), dest="/path/to/library/bios", - notes=("BIOS files can also be managed in RomM's web interface.",), + notes=("You can also add BIOS files in RomM's web page.",), ), "rocknix": Guide( - summary="A Linux system for ARM and x86 handhelds, among them the " - "RG35XX (H700) and the RG552 (RK3399).", + summary="A system for handhelds.", folders=(("On ROCKNIX", "`/storage/roms/bios/`"),), run=("device",), - notes=("The folder is reachable over SSH and over the network share.",), + notes=("You can also reach the folder from a computer, over your home network.",), + reference_notes=( + "ROCKNIX runs on ARM and x86 handhelds, among them the RG35XX (H700) " + "and the RG552 (RK3399). The folder is reachable over SSH and over " + "the network share.", + ), ), "amberelec": Guide( - summary="A Linux system for Anbernic handhelds: RG351P, RG351V, " - "RG351MP, RG353, RG503 and RG552.", + summary="A system for Anbernic handhelds.", folders=(("On AmberELEC", "`/storage/roms/`"),), run=("device",), - notes=( - "The AmberELEC list names files under `/storage/roms`: `bios/` for " - "most, `neogeo/` for the rest. The pack holds those folders, so it " - "is extracted into `/storage/roms/` itself.", + reference_notes=( + "AmberELEC runs on the RG351P, RG351V, RG351MP, RG353, RG503 and " + "RG552. Its list names files under `/storage/roms`: `bios/` for most, " + "`neogeo/` for the rest. The pack holds those folders.", ), ), "emuelec": Guide( - summary="A Linux system for Amlogic TV boxes, the Odroid Go Advance, " - "GameForce and RK356x devices.", + summary="A system for TV boxes and some handhelds.", folders=(("On EmuELEC", "`/storage/roms/`"),), run=("device",), - notes=( - "The EmuELEC list names files under `/storage/roms`: `bios/` for " - "most, `neogeo/` and `ports/` for the rest. The pack holds those " - "folders, so it is extracted into `/storage/roms/` itself.", + reference_notes=( + "EmuELEC runs on Amlogic TV boxes, the Odroid Go Advance, GameForce " + "and RK356x devices. Its list names files under `/storage/roms`: " + "`bios/` for most, `neogeo/` and `ports/` for the rest. The pack " + "holds those folders.", ), ), "knulli": Guide( - summary="A system for handhelds based on Batocera, among them the " - "RG35XX family (H700) and the TrimUI Smart Pro.", + summary="A system for handhelds, based on Batocera.", folders=(("On KNULLI", "`/userdata/bios/`"),), run=("device",), + reference_notes=( + "KNULLI runs on handhelds among them the RG35XX family (H700) and the " + "TrimUI Smart Pro.", + ), ), "retropie": Guide( - summary="Scripts that install RetroArch and standalone emulators on " - "Raspberry Pi OS.", + summary="Installs RetroArch and other emulators on a Raspberry Pi.", folders=(("On the Raspberry Pi", "`~/RetroPie/BIOS/`"),), run=("device",), notes=( - "Use the RetroPie pack, not the RetroArch one. RetroPie also " - "installs standalone emulators (openMSX, PCSX2, XRoar, Amiberry and " - "others) whose files only the RetroPie pack carries.", - "A stock RetroPie 4.8 image ships Python 3.7, older than the " - "installer needs. On it, download the RetroPie pack and extract it " - "into `~/RetroPie/BIOS/`.", + "Use the RetroPie pack, not the RetroArch one. On RetroPie 4.8, " + "Python is too old for the command: use the pack there.", + ), + reference_notes=( + "RetroPie also installs standalone emulators (openMSX, PCSX2, XRoar, " + "Amiberry and others) whose files only the RetroPie pack carries.", "RetroPie is archived in this project: its configuration is kept and " "its pack is still built, but its upstream list is no longer read " "again on a schedule.", ), ), "bizhawk": Guide( - summary="A multi-system emulator with rerecording and debugging tools, " - "hosted by TASVideos.", + summary="An emulator app for many consoles, from TASVideos.", folders=( ("Inside the BizHawk folder", "`Firmware\\`"), ("Default place the installer looks", "`%USERPROFILE%\\BizHawk\\Firmware\\`"), ), run=("windows", "linux"), + reference_notes=("BizHawk carries rerecording and debugging tools.",), ), "misterfpga": Guide( - summary="FPGA cores that recreate console hardware on a DE10-Nano board.", + summary="A board that recreates old consoles in hardware, not in software.", folders=(("On the MiSTer SD card", "`/media/fat/games/`, one subfolder per core"),), run=("device",), notes=( - "Each file goes in the subfolder of its core, for example " - "`/media/fat/games/3DO/boot.rom`. Only the files the MiSTer BIOS " - "database lists with a download address are in the pack. The others " - "come with the MiSTer distribution and install themselves.", + "Each file goes in its console's folder, like " + "`/media/fat/games/3DO/boot.rom`.", + ), + reference_notes=( + "MiSTer runs FPGA cores on a DE10-Nano board. Only the files the " + "MiSTer BIOS database lists with a download address are in the pack. " + "The others come with the MiSTer distribution and install themselves.", ), ), "minui": Guide( - summary="A frontend for handhelds that runs from the SD card.", + summary="A simple game menu for handhelds. It runs from the SD card.", folders=(("On the SD card", "`Bios//`, one folder per console"),), run=CARD_TERMINALS, on_card=True, notes=( + "Each console has its own folder, named after the tag in its Roms " + "folder. `Roms/Sony PlayStation (PS)/` uses `Bios/PS/`.", + ), + reference_notes=( "MinUI gives each console its own folder, named after the tag in " "parentheses of the Roms folder the game sits in. A game in " - "`Roms/Sony PlayStation (PS)/` runs with `Bios/PS/` as its BIOS " - "folder, one in `Roms/Game Boy (GB)/` with `Bios/GB/` (MinUI " - "`workspace/all/minarch/minarch.c:2943`). The pack holds the `Bios/` " - "folder: extract it at the root of the card.", + "`Roms/Game Boy (GB)/` runs with `Bios/GB/` as its BIOS folder (MinUI " + "`workspace/all/minarch/minarch.c:2943`).", "Without `--platform`, the installer reads the release name MinUI " "writes in `.system/version.txt` on the card. A console started by a " "standalone program instead of a libretro core (DraStic for NDS, " @@ -333,29 +353,33 @@ GUIDES: dict[str, Guide] = { ), ), "nextui": Guide( - summary="A frontend for handhelds derived from MinUI. It runs from the SD card.", + summary="A game menu for handhelds, based on MinUI. It runs from the SD card.", folders=(("On the SD card", "`Bios//`, one folder per console"),), run=CARD_TERMINALS, on_card=True, notes=( + "Each console has its own folder, named after the tag in its Roms " + "folder. `Roms/Sony PlayStation (PS)/` uses `Bios/PS/`.", + ), + reference_notes=( "NextUI gives each console its own folder, named after the tag in " - "parentheses of the Roms folder the game sits in: `Bios/PS/` for " - "`Roms/Sony PlayStation (PS)/` (NextUI " - "`workspace/all/minarch/ma_core.c:76`). The pack holds the `Bios/` " - "folder: extract it at the root of the card.", + "parentheses of the Roms folder the game sits in (NextUI " + "`workspace/all/minarch/ma_core.c:76`).", ), ), "myminui": Guide( - summary="A frontend for handhelds derived from MinUI. It runs from the SD card.", + summary="A game menu for handhelds, based on MinUI. It runs from the SD card.", folders=(("On the SD card", "`Bios//`, one folder per console"),), run=CARD_TERMINALS, on_card=True, notes=( + "Each console has its own folder, named after the tag in its Roms " + "folder. `Roms/Sony PlayStation (PS)/` uses `Bios/PS/`.", + ), + reference_notes=( "MyMinUI gives each console its own folder, named after the tag in " - "parentheses of the Roms folder the game sits in: `Bios/PS/` for " - "`Roms/Sony PlayStation (PS)/` (MyMinUI " - "`workspace/all/minarch/minarch.c:3532`). The pack holds the `Bios/` " - "folder: extract it at the root of the card.", + "parentheses of the Roms folder the game sits in (MyMinUI " + "`workspace/all/minarch/minarch.c:3532`).", ), ), } @@ -409,23 +433,23 @@ def platform_args(name: str, guide: Guide) -> dict[str, str]: return args -TRUST_NOTE = ( - "The line downloads a small script from this project on GitHub. The script " - "checks the installer against a fingerprint it carries, then the installer " - "downloads the files your setup needs and checks each one before saving it. " - "It writes only inside your BIOS folder. A BIOS file whose contents differ " - "from the expected one is replaced with a checked copy, and yours is kept in " - "a retrobios-backup folder beside it. Files it does not know are never moved " - "or deleted. Games and saves are never touched." +TRUST_ITEMS = ( + "It checks each file before saving it.", + "It writes only in your BIOS folder.", + "It never touches games or saves.", + "It never moves or deletes files it does not know.", + "It keeps any file it replaces in a `retrobios-backup` folder, beside your " + "BIOS folder. The same command with `--restore` puts it back.", ) def trust_box(prefix: str, indent: str = "") -> list[str]: - """What the command does, under every command on a guide page.""" + """Is it safe, under every command on a guide page: five plain facts.""" return [ - f'{indent}??? note "What this command does"', + f'{indent}??? note "Is it safe?"', "", - f"{indent} {TRUST_NOTE} " - f"[Read the script first]({prefix}wiki/faq.md#is-piping-the-installer-into-a-shell-safe).", + *(f"{indent} - {item}" for item in TRUST_ITEMS), + "", + f"{indent} [Read the script first]({prefix}wiki/faq.md#is-piping-the-installer-into-a-shell-safe)", "", ] diff --git a/tests/test_audit_regressions.py b/tests/test_audit_regressions.py index 58b44745..78388d20 100644 --- a/tests/test_audit_regressions.py +++ b/tests/test_audit_regressions.py @@ -5333,7 +5333,8 @@ class DocumentedExamplesHoldTogether(unittest.TestCase): def test_the_first_install_example_counts_one_number(self): import re - page = (ROOT / "wiki" / "first-install.md").read_text(encoding="utf-8") + # The annotated run moved from First install to the installer page. + page = (ROOT / "wiki" / "installer.md").read_text(encoding="utf-8") downloading = re.search(r"Downloading (\d+) files", page).group(1) done = re.search(r"Done\. (\d+) files installed", page).group(1) self.assertEqual(done, downloading) diff --git a/tests/test_doc_pages.py b/tests/test_doc_pages.py index a25a43ef..cd208c02 100644 --- a/tests/test_doc_pages.py +++ b/tests/test_doc_pages.py @@ -674,15 +674,17 @@ class FirstInstallWalkthrough(unittest.TestCase): Batocera run printed 399 files not installed. Its disk sizes (Batocera 4.0 GB) contradicted its own example (8.6 GB), and its no-platform case said a piped command cannot ask, where install.sh hands it the terminal - and the installer shows a menu.""" + and the installer shows a menu. The annotated run now lives in the + installer page, under Reading a run: First install keeps the steps.""" def setUp(self): self.text = _page("first-install.md") + self.run = _page("installer.md") def test_no_hand_kept_size_before_the_run(self): import install - section = _section(self.text, "Before starting") + section = _section(self.run, "Before the run") # The confirmation threshold is a constant of the installer, not a pack size. self.assertEqual(install.CONFIRM_FROM_BYTES, 1024 ** 3) self.assertIn("reaches 1 GB", section) @@ -691,10 +693,12 @@ class FirstInstallWalkthrough(unittest.TestCase): def test_the_run_is_dated_and_consistent_with_the_manifest(self): import install - self.assertRegex(self.text, r"published on \d{4}-\d{2}-\d{2}") - step3 = _section(self.text, "Step 3. Let it work") - step4 = _section(self.text, "Step 4. Read the end of the run") - step5 = _section(self.text, "Step 5. Confirm it later") + self.assertRegex(_section(self.run, "Reading a run { #reading-a-run }"), + r"published on\s+\d{4}-\d{2}-\d{2}") + step3 = _section(self.run, "The file index and the download") + step4 = _section(self.run, "The end of the run") + step5 = _section(self.run, "A later check") + self.assertIn(re.search(r"Done\. \d+ files installed", step4).group(0), self.text) count, size = re.search(r"\n (\d+) files \(([\d.]+ [KMG]B)\)", step3).groups() self.assertIn(f"0/{count} present", step3) self.assertIn(f"Downloading {count} files ({size})", step3) diff --git a/tests/test_doc_surfaces.py b/tests/test_doc_surfaces.py index b87abebe..a565740c 100644 --- a/tests/test_doc_surfaces.py +++ b/tests/test_doc_surfaces.py @@ -93,7 +93,7 @@ class TheDownloadTablesFollowTheRegistry(unittest.TestCase): row = next(line for line in page.splitlines() if line.startswith(f"| [{name}](")) with self.subTest(os=os_name, platform=platform): - self.assertIn(label, row.split(" | ")[2]) + self.assertIn(label, row.split(" | ")[1]) class DownloadLinksNameTheFile(unittest.TestCase): @@ -208,10 +208,10 @@ class EndUsersCheckTheirOwnFolder(unittest.TestCase): files points at the installer's --check, which hashes their folder.""" USER_SECTIONS = ( - ("wiki/troubleshooting.md", "## Game won't start / black screen"), + ("wiki/troubleshooting.md", "## 1. Check your BIOS folder { #checking-what-you-installed }"), + ("wiki/troubleshooting.md", "## 2. Fix the files"), ("wiki/faq.md", "## My game shows a black screen"), ("wiki/faq.md", "## How do I know which BIOS I need?"), - ("wiki/troubleshooting.md", "## Checking what you installed"), ) def test_user_sections_never_send_to_verify(self): diff --git a/tests/test_minui_platform.py b/tests/test_minui_platform.py index 8ab441f2..a167ae6e 100644 --- a/tests/test_minui_platform.py +++ b/tests/test_minui_platform.py @@ -429,7 +429,7 @@ class TheSitePageOfACardPlatform(unittest.TestCase): self.assertNotIn("0/0", page) self.assertNotIn("## Systems overview", page) self.assertIn("| `Bios/PS/` | [pcsx_rearmed](../emulators/pcsx_rearmed.md) |", page) - self.assertIn("| Extract the pack into | `/` |", page) + self.assertIn("| Unzip the pack into | `/` |", page) if __name__ == "__main__": diff --git a/tests/test_site_copy.py b/tests/test_site_copy.py new file mode 100644 index 00000000..a4518587 --- /dev/null +++ b/tests/test_site_copy.py @@ -0,0 +1,442 @@ +"""The pages a newcomer reads stay short, plain and quick to act on. + +The Start here pages ran 460 to 1,950 words, the first command of the +download page sat 2.8 phone screens down, and a sentence of 57 words listed +every platform's check. Each budget below is a number on the built page: +words before the first action, sentence and block length, words per page, and +a list of words a newcomer does not know or does not need. + +Run ``python -m tests.test_site_copy --report docs`` to print the measures of +every page held here. +""" + +from __future__ import annotations + +import os +import re +import statistics +import sys +import unittest +from dataclasses import dataclass, field +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +DOCS = REPO_ROOT / "docs" + +# Level 0: the Start here pages, with their own ceiling on words. +START_PAGES = { + "index.md": 220, + "which-pack.md": 400, + "wiki/what-are-bios-files.md": 300, + "wiki/first-install.md": 400, + "wiki/troubleshooting.md": 300, + "wiki/faq.md": 450, + "wiki/glossary.md": 200, + "wiki/getting-started.md": 120, +} +# Level 1: the guide part of each setup page, above its Reference heading. +SETUP_CEILING = 220 +# The setup index is a card per setup, one sentence each. +SETUP_INDEX_CEILING = 300 + +BUDGETS = { + # longest sentence, median sentence, longest block, words before the first action + "start": (20, 12, 35, 50), + "setup": (25, 14, 40, 90), +} +HOME_BEFORE_ACTION = 40 +LIST_ITEMS = {"start": 5, "setup": 6} +HEADING_WORDS = 8 + +# Words that say nothing, or talk about the page instead of the task. +FILLER = ( + "just", "simply", "easy", "easily", "easiest", "basically", "actually", + "really", "very", "quite", "note that", "please note", "keep in mind", + "of course", "obviously", "clearly", "in order to", "worth", "goes through", + "covers", "explains", "says more", "says where it stands", "what follows", + "this page", "these pages", "each section", "the last section", + "the rest of this page", "further down", "the usual causes", "come up", + "one path", "a few minutes", "is the answer", "the normal way", "not a fault", + "does not pretend", "plainly", "for when you want", "for your own tools", + "welcome", "robust", "seamless", "comprehensive", "leverage", "streamline", + "ensure", "utilize", "facilitate", "via", "e.g.", "i.e.", "etc.", "vs.", + "respectively", "deliberately", +) +# Words a newcomer meets only with their meaning in the same sentence. The +# value is the gloss that makes the word allowed there, None when no gloss +# does: such a word belongs one level down. +JARGON = { + "checksum": None, "md5": None, "sha1": None, "sha-1": None, "sha-256": None, + "crc32": None, "hash": r"fingerprint", "manifest": None, "bootstrap": None, + "execution policy": None, "libretro": None, "standalone": None, "hle": None, + "dump": None, "revision": None, "verified": None, "frontend": None, + "system id": None, "upstream": None, "scrape": None, "yaml": None, + "json": None, "csv": None, "sqlite": None, "repository": None, + "release asset": None, "rom set": None, "casefold": None, + "case-sensitive": None, "mount": None, "read-only": None, "proxy": None, + "https_proxy": None, "sudo": r"administrator", "ssh": r"terminal over the network", + "termux": r"terminal app", "core": r"small emulator|one emulator|its own emulator", + "interoperability": None, "fair use": None, "dmca": None, "circumvent": None, + "rerecording": None, "fpga cores": None, "de10-nano": None, "amlogic": None, + "rk356x": None, "h700": None, "rk3399": None, "emulationstation": None, + "extract": None, "extracted": None, +} +# Level 0 says ZIP and setup, a setup page names its own pack and platform. +START_ONLY = {"pack": None, "packs": None, "platform": None, "platforms": None} +# A word glossed in place: "X, a ...", "X (a ...", "called X". +GENERIC_GLOSS = r"(?:{w}\*{{0,2}}, an? |{w}\*{{0,2}} \(an? |called \*{{0,2}}{w})" + +WORD = re.compile(r"[A-Za-z0-9][A-Za-z0-9'._/\\:%~-]*") +SENTENCE_END = re.compile(r"(?<=[.!?])\s+(?=[A-Z`\"(*\[])") + + +@dataclass +class Block: + kind: str # paragraph, item, term, definition, heading, table + text: str + collapsed: bool = False + defines: str = "" + hidden: bool = False # in a tab other than the first: read instead of the first + + +@dataclass +class Page: + blocks: list[Block] = field(default_factory=list) + before_action: int | None = None + list_runs: list[int] = field(default_factory=list) + + def prose(self) -> list[Block]: + return [b for b in self.blocks if b.kind in ("paragraph", "item", "term", "definition")] + + def words(self) -> int: + return sum(count_words(b.text) for b in self.blocks if b.kind != "table" and not b.hidden) + + def sentences(self) -> list[str]: + return [s for b in self.prose() for s in split_sentences(b.text)] + + +def strip_inline(text: str) -> str: + """Link text kept, targets, attribute lists, images and markup dropped.""" + text = re.sub(r"!\[[^\]]*\]\([^)]*\)(\{[^}]*\})?", "", text) + text = re.sub(r"\[([^\]]*)\]\([^)]*\)", r"\1", text) + text = re.sub(r"\{\s*[#.:][^}]*\}", "", text) + text = re.sub(r"<[^>]+>", "", text) + return text.replace("**", "").strip() + + +def count_words(text: str) -> int: + return len(WORD.findall(re.sub(r"`[^`]*`", "code", text))) + + +def split_sentences(text: str) -> list[str]: + return [s for s in SENTENCE_END.split(text.strip()) if WORD.search(s)] + + +def guide_part(markdown: str) -> str: + """A page up to its Reference heading, without front matter.""" + if markdown.startswith("---\n"): + markdown = markdown.split("\n---\n", 1)[1] + match = re.search(r"^## Reference\b", markdown, re.M) + return markdown[: match.start()] if match else markdown + + +def parse(markdown: str) -> Page: + """Blocks of prose a reader sees, code and HTML left out.""" + text = guide_part(markdown) + text = re.sub(r"", "", text, flags=re.S) + text = re.sub(r"", "", text, flags=re.S) + page = Page() + seen_words = 0 + collapsed_indent: int | None = None + tab_indent: int | None = None # the tabs of a group a reader picks one of + hidden_indent: int | None = None + fence: str | None = None + current: list[str] = [] + kind = "paragraph" + run = 0 + + def flush() -> None: + nonlocal current, seen_words, kind + if current: + body = strip_inline(" ".join(current)) + if WORD.search(body): + term = page.blocks[-1].text.lower() if kind == "definition" and page.blocks else "" + block = Block(kind, body, collapsed_indent is not None, term, + hidden_indent is not None) + page.blocks.append(block) + if (page.before_action is None and not block.collapsed + and not block.hidden and kind not in ("table", "heading")): + seen_words += count_words(body) + current = [] + if kind in ("item", "definition", "heading"): + kind = "paragraph" + + def act() -> None: + flush() + if page.before_action is None: + page.before_action = seen_words + + for raw in text.splitlines(): + line = raw.rstrip() + stripped = line.strip() + indent = len(line) - len(line.lstrip()) + if fence: + if stripped.startswith(fence): + fence = None + continue + if stripped and tab_indent is not None and indent <= tab_indent and not stripped.startswith("=== "): + flush() + tab_indent = hidden_indent = None + if stripped.startswith(("```", "~~~")): + fence = stripped[:3] + if hidden_indent is None: + act() + continue + if collapsed_indent is not None and stripped and indent <= collapsed_indent: + flush() + collapsed_indent = None + if not stripped: + flush() + continue + if re.match(r"(\?\?\?\+?|!!!) ", stripped): + flush() + if stripped.startswith("??? "): + collapsed_indent = indent + continue + if stripped.startswith("=== "): + flush() + if tab_indent == indent: + hidden_indent = indent + else: + tab_indent, hidden_indent = indent, None + continue + if stripped.startswith("<") or stripped.startswith(" list[str]: + """Filler, and jargon met without its meaning beside it.""" + found = [] + text = re.sub(r"`[^`]*`", "", block.text) + if block.kind == "heading" and text.rstrip().endswith("."): + return [] # a heading that quotes an installer message + if block.kind == "term": + return [] + for sentence in split_sentences(text) or [text]: + lower = sentence.lower() + for phrase in FILLER: + if re.search(rf"(? dict: + lengths = [count_words(s) for s in page.sentences()] + return { + "words": page.words(), + "before_action": page.before_action, + "max_block": max((count_words(b.text) for b in page.prose()), default=0), + "median_sentence": statistics.median(lengths) if lengths else 0, + "max_sentence": max(lengths, default=0), + } + + +def held_pages(docs: Path) -> list[tuple[str, str, int]]: + """(relative path, level, ceiling) for every page the budgets hold.""" + pages = [(name, "start", ceiling) for name, ceiling in START_PAGES.items()] + pages += [ + (str(path.relative_to(docs)), "setup", SETUP_CEILING) + for path in sorted((docs / "platforms").glob("*.md")) + if path.name != "index.md" + and "\n## Where the files go" in path.read_text(encoding="utf-8") + ] + pages.append(("platforms/index.md", "setup", SETUP_INDEX_CEILING)) + return [(name, level, ceiling) for name, level, ceiling in pages if (docs / name).is_file()] + + +def _docs() -> Path: + if not (DOCS / "index.md").is_file(): + if os.environ.get("CI"): + raise AssertionError("CI generates docs/ before the suite") + raise unittest.SkipTest("docs/ not generated (python scripts/generate_site.py)") + return DOCS + + +def _load(name: str) -> Page: + return parse((_docs() / name).read_text(encoding="utf-8")) + + +class CopyBudgets(unittest.TestCase): + def test_sentences_stay_short(self): + for name, level, _ceiling in held_pages(_docs()): + longest, median, _block, _before = BUDGETS[level] + page = _load(name) + with self.subTest(page=name): + long = [s for s in page.sentences() if count_words(s) > longest] + self.assertEqual(long, [], f"sentences over {longest} words") + self.assertLessEqual(measure(page)["median_sentence"], median) + + def test_blocks_stay_short(self): + for name, level, _ceiling in held_pages(_docs()): + limit = BUDGETS[level][2] + page = _load(name) + with self.subTest(page=name): + long = [b.text for b in page.prose() if count_words(b.text) > limit] + self.assertEqual(long, [], f"blocks over {limit} words") + + def test_the_first_action_comes_early(self): + for name, level, _ceiling in held_pages(_docs()): + limit = HOME_BEFORE_ACTION if name == "index.md" else BUDGETS[level][3] + page = _load(name) + if page.before_action is None: + continue # a page of definitions has nothing to run + with self.subTest(page=name): + self.assertLessEqual(page.before_action, limit) + + def test_pages_stay_under_their_ceiling(self): + for name, _level, ceiling in held_pages(_docs()): + with self.subTest(page=name): + self.assertLessEqual(_load(name).words(), ceiling) + + def test_lists_of_choices_stay_short(self): + for name, level, _ceiling in held_pages(_docs()): + with self.subTest(page=name): + self.assertLessEqual(max(_load(name).list_runs, default=0), LIST_ITEMS[level]) + + def test_headings_stay_short(self): + for name, _level, _ceiling in held_pages(_docs()): + page = _load(name) + long = [ + b.text for b in page.blocks + if b.kind == "heading" and not b.text.endswith(".") + and count_words(re.sub(r"`[^`]*`", "", b.text)) > HEADING_WORDS + ] + with self.subTest(page=name): + self.assertEqual(long, []) + + +class PlainWords(unittest.TestCase): + def test_no_filler_and_no_unexplained_jargon(self): + offenders = [] + for name, level, _ceiling in held_pages(_docs()): + for block in _load(name).blocks: + if block.kind == "table": + continue + words = banned(block, level) + if words: + offenders.append(f"{name}: {sorted(set(words))} in: {block.text[:90]}") + self.assertEqual(offenders, [], "\n".join(offenders[:40])) + + +class TheMeasureItself(unittest.TestCase): + """The parser decides every number above, so it is held on a sample.""" + + SAMPLE = ( + "---\nlevel: start\n---\n\n# Title\n\nOne two three. Four five.\n\n" + "- Item **one**.\n- Item [two](x.md).\n\n```sh\nnot counted at all\n```\n\n" + "??? note \"Hidden\"\n\n Collapsed words here.\n\n" + "=== \"A\"\n\n First tab.\n\n=== \"B\"\n\n Second tab words.\n\n" + "| A | B |\n|---|---|\n| cell | cell |\n\n## Reference: deep\n\nNot read.\n" + ) + + def test_blocks_and_counts(self): + page = parse(self.SAMPLE) + kinds = [b.kind for b in page.blocks] + self.assertEqual(kinds, ["heading", "paragraph", "item", "item", "paragraph", + "paragraph", "paragraph", "table"]) + self.assertEqual([b.hidden for b in page.blocks[5:7]], [False, True]) + self.assertEqual(page.words(), 1 + 5 + 2 + 2 + 3 + 2) + self.assertEqual(page.before_action, 5 + 2 + 2) + self.assertEqual(page.list_runs, [2]) + self.assertTrue(page.blocks[4].collapsed) + self.assertEqual(page.sentences()[:2], ["One two three.", "Four five."]) + + def test_a_gloss_allows_a_term(self): + self.assertEqual(banned(Block("paragraph", "Install Termux, a terminal app."), "start"), []) + self.assertEqual(banned(Block("paragraph", "Open Termux."), "start"), ["termux"]) + self.assertEqual(banned(Block("paragraph", "Run `sudo` first."), "start"), []) + self.assertEqual(banned(Block("paragraph", "It is very easy."), "start"), ["easy", "very"]) + self.assertEqual(banned(Block("paragraph", "The RetroArch pack."), "setup"), []) + + +def report(docs: Path) -> str: + rows = ["| Page | Words | Before action | Max block | Median / max sentence |", + "|---|---:|---:|---:|---|"] + for name, _level, _ceiling in held_pages(docs): + m = measure(parse((docs / name).read_text(encoding="utf-8"))) + before = "none" if m["before_action"] is None else m["before_action"] + rows.append(f"| {name} | {m['words']} | {before} | {m['max_block']} | " + f"{m['median_sentence']:g} / {m['max_sentence']} |") + return "\n".join(rows) + + +if __name__ == "__main__": + if len(sys.argv) == 3 and sys.argv[1] == "--report": + print(report(Path(sys.argv[2]))) + else: + unittest.main() diff --git a/wiki/adding-a-platform.md b/wiki/adding-a-platform.md index 6180c07c..870fcc1f 100644 --- a/wiki/adding-a-platform.md +++ b/wiki/adding-a-platform.md @@ -171,6 +171,14 @@ page. The groups and their labels are `DEVICE_GROUPS` in (what it is, where its BIOS folder is, where the install command runs). A test fails when a registered platform has no group or no guide. +The guide text is read by someone who knows their device and little else. +`summary` is one plain sentence, with no chip or board names. `notes` hold +one fact each that the reader acts on. A source citation, a list of models or +how the installer finds the folder goes in `reference_notes`, which open the +Reference part of the page, and a rare build goes in `more_folders`, shown +folded. The guide part of every setup page is held to the word budgets of +[Pages for newcomers](testing-guide.md#pages-for-newcomers). + A platform without a scraper (`scraper: null`, `schedule: null`) keeps its YAML as a hand-maintained file. Every registered platform has a scraper today, so none is edited by hand: RetroPie inherits RetroArch's file set and its scraper diff --git a/wiki/faq.md b/wiki/faq.md index f49c89e8..d4a3870a 100644 --- a/wiki/faq.md +++ b/wiki/faq.md @@ -4,209 +4,105 @@ level: start # FAQ -The questions people ask before installing anything. Questions about how files -are identified and checked are in the -Reference: [technical questions](technical-faq.md). - ## Do I need BIOS files? -For many consoles, yes: PlayStation, PlayStation 2, Saturn, Mega-CD, Neo Geo, -Dreamcast and many home computers. For cartridge consoles such as the NES or -the Super Nintendo, usually not. [What BIOS files are](what-are-bios-files.md) -explains why, console by console. +For many consoles, yes: PlayStation 1 and 2, Saturn, Dreamcast, Neo Geo and +many home computers. For NES or Super Nintendo cartridges, usually not. +[What is a BIOS?](what-are-bios-files.md) ## How do I know which BIOS I need? -Two approaches: +Run the installer with `--dry-run`. It lists what it would install and changes +nothing. -1. **Ask the installer.** `--dry-run` lists what it would install for your platform and writes nothing, `--list-systems` gives the file count per system, and `--emulator` narrows to what one core reads: +```sh +curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --dry-run +``` - ```bash - curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --emulator beetle_psx --dry-run - ``` +Your setup's page also lists every file, console by console. -2. **Check the project site.** Each platform page lists all required and optional BIOS files per system, and each emulator page the files that core loads, with the source references and expected values. +## Is the command safe? { #is-piping-the-installer-into-a-shell-safe } -## Is piping the installer into a shell safe? +- It checks the installer's fingerprint before running it. If the download + does not match its expected fingerprint, it stops. +- It checks every file before saving it. +- It writes only in your BIOS folder. -The one-liner runs a small bootstrap, not the installer. That bootstrap fetches -`install.py` over HTTPS only, refuses anything over 2 MB, and requires its -SHA-256 to equal the value written inside the bootstrap itself. A substituted -installer aborts with "the downloaded installer does not match its expected -fingerprint" before a single line of it runs. +To read the script before you run it: -The installer then verifies every file it downloads against the size, SHA-256 -and SHA-1 the manifest declares, and moves it into place only once those match. -The manifest is treated as untrusted: a destination cannot be absolute, carry a -drive letter or climb out of the BIOS directory. - -Reading the bootstrap before running it is two lines: - -```bash +```sh curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh -o install.sh less install.sh && sh install.sh ``` -Details: [All installer options](installer.md), with the full behaviour. +Details: [How the installer checks itself](installer.md#what-the-bootstrap-does). -## Will the installer change my other files? +## Will it change my other files? -No. It writes only inside the BIOS folder and `retrobios-backup` beside it, and games and saves are never -touched. Inside that folder, a BIOS file whose contents differ from the -expected one is replaced with a checked copy, and your old file is moved to a -`retrobios-backup` folder beside the BIOS folder. Files the installer does not -know are never read, moved or deleted. Copies into the folders of standalone -emulators happen only when you ask for them with `--standalone-copies`. +No. It never touches games or saves, or a file it does not know. It replaces a +BIOS file with the wrong contents, and keeps yours. -## What if Windows or an antivirus blocks the command? +## Windows or my antivirus blocks the command -The PowerShell execution policy governs script files, and the PowerShell line -writes none, so the policy does not block it. Where a security tool still -blocks it, download the pack of your setup from -[Get your files](../which-pack.md) and extract it into the BIOS folder: it holds -the same files. +Download the ZIP of your setup instead, from [Get your files](../which-pack.md). +It holds the same files. -## Can I run the installer again? +## Can I run it again? -Yes, and re-running is the normal way to update. A file already present with -the expected hash is left untouched, one whose contents do not match is -replaced by the verified copy (the old one is kept), and files the manifest does not name are never -read, moved or deleted. `--check` does the same inspection and exits without -writing. +Yes. That is how you update. It downloads only what is missing or wrong. ## Can I undo it? -Yes. A file you had that the installer replaced is kept in a dated folder -inside `retrobios-backup`, beside the BIOS folder. The end of the run prints -that folder and the command that puts the files back, the same command with -`--restore` and the folder. Copying the contents of the dated folder into the folder that holds -`retrobios-backup` does the same. Files that were not there before can be -deleted from the BIOS folder. The installer changed nothing outside it, apart -from the backup folder. -[More on kept files](installer.md#files-you-already-had). +Yes. It keeps every file it replaces in a `retrobios-backup` folder, beside +your BIOS folder. The end of the run prints the line that puts them back: the +same command with `--restore` and that folder. -## Can I install onto an SD card or another machine's drive? +Details: [Files you already had](installer.md#files-you-already-had). -Yes, `--dest` takes any path and the one-liner forwards arguments: +## Fill an SD card from a computer? -```bash +Yes. Name your setup and the BIOS folder on the card: + +```sh curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh \ | sh -s -- --platform retroarch --dest /path/to/sdcard/retroarch/cores/system ``` -`--dest` is the BIOS folder itself: on a card, `retroarch/cores/system` for -RetroArch on Switch, `RetroArch/system` on Android. +On Windows the line is different. Details: [Passing options](installer.md#passing-options). -On Windows the arguments need a script block, since `iex` would read them as -its own. Details: [the Windows form](installer.md#passing-options). +## Does a BIOS work on every setup? -## Can I use BIOS from one platform on another? +Yes. Only the folder changes, and sometimes the name. -Yes. BIOS files are console-specific, not platform-specific. A PlayStation BIOS works in RetroArch, Batocera, Recalbox, and any other platform that emulates PlayStation. The only differences between platforms are: +## Why is a file missing from the ZIP? -- **Where the file goes** (each platform has its own BIOS directory) -- **What filename is expected** (usually the same, occasionally different) -- **How verification works** (MD5 check vs. existence check) +Some files come with the emulator when you install it. The ZIP leaves those +out. -The packs differ per platform because each platform declares its own set of supported systems and expected files. +## Required or optional? -## Why is a file the emulator reads not in the pack? - -A pack brings what the user has to obtain. A file the emulator installs itself, a -font, a shader, a game database, a free replacement BIOS, is already on the -machine once the emulator is installed, so the pack leaves it out. This is decided -per build and per platform: the PPSSPP standalone release ships its `assets/` -folder, the PPSSPP libretro core does not, so RetroArch packs carry those files and -a standalone PPSSPP pack does not. When a platform's own package installs data -beside a core (Batocera with blueMSX), that platform's pack leaves it out too. The -emulator's page on the site marks these files as installed with the emulator. - -## How often are packs updated? - -Upstream sources (libretro System.dat, batocera-systems, etc.) are re-scraped by hand and the diff is reviewed before it lands. Releases happen as needed when new BIOS files are added or profiles are updated. - -## What's the difference between required and optional? - -**Required** means the emulator will not start games for that system without -the file. **Optional** means the emulator works without it, but with reduced -accuracy or missing features (e.g., boot screen animation, wrong font -rendering, or degraded audio). - -## What's HLE? - -HLE (High-Level Emulation) is a software reimplementation of what the original -BIOS does. Some cores can boot games without a real BIOS file by using their -built-in HLE fallback. The trade-off is lower accuracy: some games may have -glitches or fail to boot entirely. The file is still included in packs because -the real BIOS gives better results. - -## What's a hash? - -A hash is a fingerprint computed from a file's contents. If even one byte -differs, the hash changes completely. Batocera, Recalbox, EmuDeck and several -other setups compare the MD5 hash of each BIOS file with their list and refuse -a file that does not match. RetroArch only checks that the file is there. The -kinds of hash and where each is used are in the -Reference: [technical questions](technical-faq.md#whats-a-hashchecksum). - -## Are these files verified against original hardware dumps? - -The hashes here come from what emulator code loads and accepts. Whether a file -matches a catalogued dump of the original chip is a separate question, and the -system pages mark the files that do. The full answer is in the -Reference: [technical questions](technical-faq.md#are-these-files-verified-against-original-hardware-dumps). +- **Required:** the emulator does not start games without it. +- **Optional:** games start, with small differences, like no start-up screen. ## My game shows a black screen -Most often a BIOS file is missing, or is not the version the emulator expects. -[When a game does not start](troubleshooting.md) goes through it step by step, -starting with a check of what is in your BIOS folder. +Most of the time, a BIOS file is missing or is the wrong version. +[When a game does not start](troubleshooting.md) shows how to check. ## Is this legal? -Redistributing firmware is not settled law, and this page does not pretend otherwise. What follows is the reasoning the project acts on, with the strength of each argument stated plainly so anyone can weigh it. None of it is legal advice, and none of it has been tested in court. +Laws on copying console firmware differ by country. The question is not +settled, and nothing here is legal advice. Details: [The project's legal position](legal.md). -### Emulation and BIOS redistribution +To ask for a file to be removed, [open an issue](https://github.com/Abdess/retrobios/issues) +and name the file. A file is removed when the owner of its rights asks. -- **Writing an emulator is lawful.** *Sony v. Connectix* (2000) and *Sega v. Accolade* (1992) held that copying a work during reverse engineering, to reach the functional elements needed for interoperability, is fair use. Both concern the act of copying while developing, and *Connectix* specifically involved copying Sony's BIOS in the course of that work. Neither decision addresses distributing firmware to third parties, so they support the legality of emulation itself rather than of this collection. -- **Fair use (US, 17 USC 107) is a defence, argued case by case.** The project's position is that non-commercial redistribution for personal backup and archival weighs favourably on purpose and on market effect, since these files are not sold separately. It weighs poorly on the amount used: each file is copied whole. Manufacturers do still monetise some of this firmware indirectly, through re-releases, subscription services and mini consoles, so the market factor is contestable rather than clear. -- **Interoperability in the EU.** The Software Directive (2009/24/EC, Art. 5-6) permits decompilation and use to achieve interoperability, subject to conditions, and Art. 6 restricts passing the resulting information to others. It supports the emulation use case, but it is not a general redistribution permission. -- **Discontinued hardware.** Most firmware here is for hardware no longer sold or supported. This bears on the fair-use market factor and on the practical likelihood of complaint. "Abandonware" is not a legal doctrine in any jurisdiction, and nothing here rests on it. +## Why keep recent consoles too? -### Encryption keys (Switch prod.keys, 3DS AES keys, Wii U keys) - -This is the most contested area, and the weakest part of the project's position. - -- **Keys are unlikely to be copyrightable.** A bare numeric value has no author's creative expression, so copyright is a poor fit. This says nothing about the DMCA, which is the provision that actually applies. -- **DMCA 1201(f) is narrow.** Section 1201(f) allows a person who has lawfully obtained the right to use a copy to circumvent for the sole purpose of achieving interoperability with an independently created program, and 1201(f)(3) allows the means to be made available to others only for that same purpose. Publishing keys openly is not obviously within that allowance, and the trafficking prohibitions in 1201(a)(2) and 1201(b) remain in play. This is the clearest legal risk the project carries. -- **Library of Congress exemptions do not cover this repository.** The triennial exemptions at 37 CFR 201.40 let an eligible library, archive or museum circumvent access controls to preserve a lawfully acquired video game whose server support has ended, with access limited to the institution's premises. The Ninth Triennial Proceeding (2024 cycle) renewed them and declined to extend them to off-premises remote access. A public repository is neither an eligible institution nor an on-premises reading room, so these exemptions show which way preservation policy is moving without authorising what is done here. -- **Keys come from consumer hardware.** They are extracted from retail devices their owners bought. *Chamberlain v. Skylink* (2004) held that a 1201 claim requires some nexus to copyright infringement, in a dispute over a garage-door opener. Later cases have narrowed its reach, and it is a weak foundation for a general post-sale right. -- **Trade secret is not a live issue.** Keys embedded in millions of shipped devices and widely republished are not kept secret by reasonable measures. - -### Recent firmware (Switch 19.0.0, PS3UPDAT, PSVUPDAT) - -- **Free download is not a licence.** Manufacturers publish these updates on open CDNs without authentication, which makes them easy to obtain but does not grant a right to redistribute them, and PlayStation firmware ships under an end-user licence agreement. The practical argument is that mirroring a file the manufacturer already gives away for free causes no identifiable loss, not that doing so is expressly permitted. -- **Functional necessity.** Emulators cannot run the software these systems were built for without this firmware. That is why the files are collected. It is a statement of purpose, not a legal defence in itself. -- **Yuzu context.** The Yuzu settlement (2024) concerned the emulator and its alleged facilitation of piracy. Yuzu settled without admitting liability, so it set no precedent either way on firmware or key redistribution. It does show that rights holders in this space litigate. - -### Summary - -This project preserves BIOS files, firmware and keys for personal use, archival and interoperability. The strongest ground is the older, discontinued firmware, where fair use and the absence of a market both point the same way. The weakest is the encryption keys, where DMCA 1201 applies and the interoperability exemption is narrower than the use made of it here. The project accepts that risk deliberately, in the belief that these files are worth preserving while they can still be obtained. - -This reflects the project's good-faith understanding, not legal advice. - -### Asking for a file to be removed - -A rights holder, or anyone acting for one, can [open an issue](https://github.com/Abdess/retrobios/issues) identifying the file by path or hash. A file whose removal is requested by its rights holder is removed from the repository and from the next release. No formal notice is required and none will be demanded. State the file and the basis of the claim. The same channel is open to anyone else with a concern about a specific file. - -## Isn't the Switch or PS3 too recent for a retro project? - -Age is not what decides inclusion here: availability is. Firmware distribution depends on manufacturer servers, and those disappear: Nintendo closed the 3DS and Wii U eShops in March 2023, Sony shut the PSP store in 2021. Once a CDN goes offline, the files only survive where someone archived them. - -The project tries to archive files while they are still available rather than after they have become rare. "Retro" is a moving window: the PlayStation 2 was current hardware when its BIOS was first preserved, and today's consoles will be tomorrow's retro systems. A file is included for the same reason at any age: an emulator needs it, and nothing guarantees it stays available. +Files disappear when stores close, like the 3DS and Wii U shops in 2023. We keep +them while they can still be found. !!! tip "Next" - [Get your files](../which-pack.md) - - [When a game does not start](troubleshooting.md) - - [Glossary](glossary.md) + - [Words used on this site](glossary.md) diff --git a/wiki/first-install.md b/wiki/first-install.md index 8df22bf1..1454e2b6 100644 --- a/wiki/first-install.md +++ b/wiki/first-install.md @@ -4,89 +4,66 @@ level: start # First install -One path, from an empty BIOS folder to files the emulator can read. Every step -shows what the screen looks like when it worked. +From an empty BIOS folder to games that start. -If something on this page does not match what happens, the last section covers -the cases that come up. +**You need:** internet and some free space. The installer says how much. -## Before starting +## 1. Open a terminal and paste -Three things, and a few minutes: +A **terminal** is a window where you type commands. Paste the line for your +system, then press Enter. -- The BIOS folder the emulator reads. The installer finds it by itself on most - systems. On a handheld, that usually means plugging the SD card into a - computer. -- `curl` or `wget`, and Python 3.8 or newer. The command below checks for both - and stops with a message if either is absent. -- Room on the drive. The size depends on the platform and changes with each - release: the run prints it before downloading (Step 3), and asks first when it - reaches 1 GB. The installer also measures the free space and refuses before - writing anything if there is not enough. +=== "Windows" -Nothing is written outside the BIOS folder, and games and saves are never -touched. + Open the Start menu. Type **PowerShell** and open it. -## Step 1. Run the command + ```powershell + irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex + ``` -The command runs in a terminal. On Windows, open the Start menu, type -PowerShell and open it. On a Mac, open Terminal from Applications, then -Utilities. On a Steam Deck, switch to Desktop Mode and open Konsole. Paste the -line and press Enter. +=== "Mac" -On Linux, macOS, a Steam Deck, or a handheld running Batocera, Recalbox, -KNULLI, ROCKNIX, AmberELEC, EmuELEC or Lakka: + Open **Terminal**. It is in Applications, then Utilities. -```bash -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -``` + ```sh + curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh + ``` -On Windows, in PowerShell: +=== "Linux and Steam Deck" -```powershell -irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1 | iex -``` + On a Steam Deck, switch to Desktop Mode and open **Konsole**. -PowerShell's execution policy does not block that line. The policy governs -script files, and this command never writes one. Changing the policy is not -part of installing anything here. + ```sh + curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh + ``` -On Android, from Termux, three commands instead of one: +=== "Android" -```bash -pkg install python -termux-setup-storage -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -``` + Install **Termux**, a terminal app. Run these three lines in it. -Before running anything, that first line downloads the installer, compares it -against the SHA-256 written inside the command's own script, and refuses to -continue if the two differ. To read the script first: + ```sh + pkg install python + termux-setup-storage + curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh + ``` -```bash -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | less -``` +On Batocera, Recalbox and other game systems, follow +[your setup's page](../which-pack.md). -## Step 2. Watch it find the platform +## 2. Check the folder it found -The first lines name what it found and where it will write: +It shows your setup and its BIOS folder: ``` -RetroBIOS 8c0e9c762d2f - -Detecting platform... Found Batocera at /userdata/bios ``` -The code after RetroBIOS identifies the installer: it is the start of the -SHA-256 the command checked before running it. +- **Right folder?** Go on. +- **Wrong folder?** Press Ctrl+C to stop. Then see [If it stops](#if-it-stops). -That path is the one thing worth reading twice. If it is not where the emulator -keeps its BIOS files, stop with Ctrl+C and see the last section. - -At a terminal, the installer then asks what to install: +## 3. Choose what to install ``` Enter) install everything @@ -95,170 +72,79 @@ At a terminal, the installer then asks what to install: q) quit ``` -Enter installs everything, which is what the rest of this page shows. **c** -lets you pick consoles, emulator cores and regions. The **i** line appears -only when the installer finds emulator cores installed. +**Enter** installs everything. **c** lets you pick consoles. -## Step 3. Let it work +## 4. Wait -The examples on this page follow the Batocera list as published on 2026-10-21. -The figures change with each release. +It shows one line per file. You can stop with Ctrl+C at any time. Run the +command again to carry on. -Two phases follow. The first reads what is already in the folder, which takes a -few minutes on a memory card because every file is hashed: - -``` -Fetching file index for batocera... - 14481 files (9.1 GB) - Not installable: 399 files (79 required, not found: 399). - -Checking existing files... - 0/14481 present (0 verified, 0 wrong hash) -``` - -`Not installable` counts the files of the list that the collection does not -hold. They are named at the end of the run (Step 4). - -On a first install the present count is zero, which is expected, not a fault. -On a later run it counts what is already there, and only the rest is -downloaded. A download of 1 GB or more is confirmed first -(`Download 9.1 GB to /userdata/bios? [Y/n]`). - -The second phase downloads what is missing, one line per file: - -``` -Downloading 14481 files (9.1 GB)... - Press Ctrl+C to stop. Running the same command again carries on from here. - [1/14481] 3do_arcade_saot.bin ok - [2/14481] 3dobios.zip ok -``` - -Ctrl+C is safe at any point. Files already installed stay, and running the -command again picks up where it stopped. - -## Step 4. Read the end of the run - -This is what a finished run looks like: +## 5. Look for "Done." ``` Done. 14481 files installed, 0 files already up to date. Location: /userdata/bios -399 files were not installed: the collection does not hold them, or its copy fails their check. - Needed to start an emulator (79 files): - dark-forces (theforceengine): DARK.GOB, SOUNDS.GOB, SPRITES.GOB, TEXTURES.GOB - dark-forces-ii (openjkdf2): openjkdf2/episode/JK1.gob, openjkdf2/episode/JK1CTF.gob, openjkdf2/episode/JK1MP.gob, openjkdf2/resource/Res1hi.gob, openjkdf2/resource/Res2.gob - diablo (devilutionx): devilutionx/diabdat.mpq - doom3 (dhewm3): base/pak000.pk4, base/pak002.pk4, base/pak003.pk4, base/pak004.pk4 - droidworks (openjkdf2): dwCD.GOB, dwHD.GOB, dwMin.GOB, dwStream.GOB - fallout (fallout1-ce): fallout1-ce/critter.dat, fallout1-ce/master.dat - fallout2 (fallout2-ce): fallout2-ce/critter.dat, fallout2-ce/master.dat - fury (fury): fury.grp - ... and 9 more systems (--check --verbose lists them all) - Each emulator's page says what these files are: https://abdess.github.io/retrobios/emulators/ -To check this later, run the same command with --check. ``` -The word `Done` and a location are the confirmation. Any other ending (`Stopped`, -`Finished with errors`, `Nothing was installed`) means something was left undone, -and the run says which case it was. +`Done.` means it worked. It may list files it could not install. -The lines between `Location:` and the `--check` hint name the files of the list -that the collection does not hold. `Needed to start an emulator` gives, for each -system, the emulators that read it and the files they cannot start without. On -the run above, all 79 belong to games (Dark Forces, Diablo, Doom 3, Tomb Raider -and the others): the engine reads them from a copy of the game itself. The -other 320 are optional, and the emulators reading them start without them. +## Check it later -## Step 5. Confirm it later +Add `--check` to the same command. It changes nothing. -The same command with `--check` reads the folder and writes nothing: - -```bash +```sh curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --check ``` -``` -Checking existing files... - 14481/14481 present (14481 verified, 0 wrong hash) +All good when you see `0 wrong hash`. - All files up to date. -``` +## Undo -The files the collection does not hold are listed after it again, as long as -they are absent from the folder. `0 wrong hash` is the line that matters. A number there means a file is present -whose contents the platform will reject, and running the install again replaces -it. Your old file is kept in `retrobios-backup`, beside the BIOS folder. +It keeps every file it replaces in a `retrobios-backup` folder, beside your +BIOS folder. Nothing is deleted. -## When the screen says something else +To put your files back, run the command with `--restore` and that folder. The +end of the run prints the exact line. + +## If it stops ### No supported platform detected. -At a terminal, a menu follows that line. The platform is picked by its number, -then the run asks for the BIOS folder, offering the platform's usual folder on -this system as the default when it has one. +At a terminal, a menu follows. Pick your setup by its number, then type its +BIOS folder. -Without a terminal, or with `--no-input`, the run stops there and prints the -command to use instead, with the platform and the folder named: - -```bash -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh \ - | sh -s -- --platform retroarch --dest /path/to/system -``` - -The accepted names come from `--list-platforms`. The page of each setup, -listed on [Get your files](../which-pack.md), gives its BIOS folder. +With `--no-input`, or without a terminal, it stops and prints the command to +use. ### Error: Python 3.8 or newer is required. -No Python 3.8 or newer was found: Python is missing, or older. The message -names the command that installs it on the system it runs on: `sudo apt install -python3` on Debian and Ubuntu, `sudo dnf install python3` on Fedora, `sudo -pacman -S python` on Arch, `xcode-select --install` on macOS and `pkg install -python` in Termux, where it offers to run that command itself (Termux installs -without root). On Windows, `install.ps1` offers to run `winget install -Python.Python.3.12`. On a handheld running Batocera or Recalbox it is already there, so -this message usually means the command ran on the computer rather than on the -console. +The message shows the command that installs Python. Run it, then run the +installer again. -A stock RetroPie 4.8 image is Debian Buster, where `python3` is 3.7.3 and apt -has nothing newer, so the same command changes nothing. The RetroPie pack from -the [releases page](https://github.com/Abdess/retrobios/releases/latest), -extracted into `~/RetroPie/BIOS/`, installs the same files. +A RetroPie 4.8 image has Python 3.7.3, and its update command finds nothing +newer. Use the [RetroPie ZIP](../platforms/retropie.md#download) there, +unzipped into `~/RetroPie/BIOS/`. ### Error: the downloaded installer does not match its expected fingerprint. -Nothing was run and nothing was written. The download did not match the -fingerprint the script carries, which is almost always a transfer cut short. -Running the command again, on another network if possible, is the answer. +The download was cut short. Nothing was changed. Run the command again. ### Cannot create ... -The user running the command cannot write there. On Batocera, Recalbox and -ROCKNIX the BIOS folder belongs to root, so the same command with `sudo` in -front of it succeeds. +You do not have the right to write there. On Batocera, Recalbox and ROCKNIX, +put `sudo` and a space before the command. It then runs as the administrator. ### Not enough space on the drive holding ... -The run stopped before writing. The message names how much is needed and how -much is free. `--dest` installs to another drive. +It stopped before writing. Free some space, or use `--dest` to pick another +drive. -### The game still shows a black screen +### The game still does not start -A file can be in place and still be the wrong one for that game, or the console -may need a file that nobody has dumped yet. -[When a game does not start](troubleshooting.md) goes through that case. +See [When a game does not start](troubleshooting.md). -## What comes next +!!! tip "Next" -- The page of your setup, from [Get your files](../which-pack.md): where its - folder is and how the setup itself checks the files -- [When a game does not start](troubleshooting.md) -- [FAQ](faq.md), for what the words mean and why a pack is this large -- Details: [All installer options](installer.md), for every option, what it - detects and what it is allowed to write - -To remove what the installer wrote, delete the files from the BIOS folder. -A file you had that it replaced is in `retrobios-backup`, beside that folder: -the end of the run printed the `--restore` command that puts it back -([details](installer.md#files-you-already-had)). Nothing else outside the -BIOS folder was changed. + - [Your setup's page](../which-pack.md): its folder and how it checks files + - [When a game does not start](troubleshooting.md) + - Details: [Reading the installer's output, line by line](installer.md#reading-a-run) diff --git a/wiki/getting-started.md b/wiki/getting-started.md index 1e763bf6..f1630420 100644 --- a/wiki/getting-started.md +++ b/wiki/getting-started.md @@ -4,25 +4,20 @@ level: start # Getting started -What this page held now lives on the pages written for each step. +Three pages take you from nothing to games that start. -## What are BIOS files? +[Get your files](../which-pack.md){ .md-button .md-button--primary } -[What BIOS files are](what-are-bios-files.md) explains what they do, and which -consoles need them. +## What is a BIOS? -## Installation +[What is a BIOS?](what-are-bios-files.md) tells you which consoles need one. -[Get your files](../which-pack.md) gives the command and the packs for every -setup. [First install, step by step](first-install.md) shows what each step of -the command prints. +## Install -## BIOS directory by platform +[Get your files](../which-pack.md) gives the command and the ZIP for your +setup. [First install, step by step](first-install.md) shows each step. -The page of each setup opens on where its files go. All setups are on -[Get your files](../which-pack.md), grouped by the device they run on. - -## Checking what you installed +## Check your folder [When a game does not start](troubleshooting.md#checking-what-you-installed) -shows how to check the files in your BIOS folder. +shows how to check your BIOS folder. diff --git a/wiki/glossary.md b/wiki/glossary.md index 8cd3b40a..aabab70c 100644 --- a/wiki/glossary.md +++ b/wiki/glossary.md @@ -2,152 +2,47 @@ level: start --- -# Glossary +# Words used on this site -The first part holds the words anyone meets when setting up BIOS files. The -second part holds the words this project uses in its reference pages. - -## Words you will meet - -**BIOS, or firmware** -: A small program built into a console. Many emulators need a copy of it to - start games. [What BIOS files are](what-are-bios-files.md) says more. +**BIOS**, or **firmware** +: A small program inside a console. It starts the games. **Emulator** -: A program that lets a computer, a phone or a handheld act like an older - console. +: An app that acts like a console. **Core** -: One emulator inside RetroArch. RetroArch loads a different core for each +: One emulator inside RetroArch. RetroArch uses a different core for each console. -**Frontend** -: The menu you pick and start games from, such as EmulationStation, MinUI or - the RetroArch menu. It starts the emulator for you. - -**Setup, or platform** -: The software you play with: RetroArch, Batocera, EmuDeck, KNULLI and the - others. Each one keeps its BIOS files in a folder of its own. +**Setup** +: What you play with: RetroArch, Batocera, EmuDeck and others. **BIOS folder** -: The folder where your setup looks for BIOS files. Its place depends on the - setup, and the page of each setup gives it. +: The folder where your setup looks for BIOS files. -**Pack** -: One ZIP file with every BIOS file a setup uses. A pack over 2 GB comes in - several parts, and every part is needed. +**ZIP** +: One file that holds many files. You unzip it to get them out. + +**Terminal** +: A window where you type commands. **Installer** -: A small program started by one line pasted into a terminal. It finds your - BIOS folder and downloads only the files that are missing. +: The small program the command starts. It fills your BIOS folder. -**Terminal, or command line** -: A window where you type commands. On Windows it is PowerShell, on a Mac it - is Terminal, on a Steam Deck it is Konsole. - -**Hash, or checksum (MD5, SHA1)** -: A fingerprint of a file's contents. Two files with the same hash hold the - same bytes. Some setups use it to refuse a file that is not the expected - version. +**Hash** +: A fingerprint of a file's contents. A different file gets a different hash. **Region** -: The market a console was sold in, such as Japan, North America or Europe. - Some emulators pick the BIOS of the same region as the game. +: Where a console was sold: Japan, North America, Europe. -**Required and optional** -: Required: the emulator does not start games without the file. Optional: - games start without it, with small differences such as a missing start-up - screen. - -**HLE** -: High-level emulation: code in the emulator that stands in for the BIOS. - Many games work with it, some glitch or do not start. +**Required** and **optional** +: Required: games do not start without it. Optional: games start anyway. **Game data** -: Files of a game that a game engine reads, such as the data of Doom or - Diablo. They come from your own copy of the game, and their absence does not - affect any console. - -## Project terms - -**Profile** -: A YAML file in `emulators/` that documents what one emulator loads, read - from its source code. - -**Ground truth** -: The emulator's source code, the reference every profile is checked against. - -**System** -: A console or computer being emulated, named by an id such as - `sony-playstation` or `nintendo-gba`. - -**Verification mode** -: How a setup checks a file: by its name only (existence), by MD5 or by SHA1. - Reference: [How each platform checks files](verification-modes.md). - -**Status** -: The outcome of one file check: `ok`, `untested` (present, but not the - expected hash) or `missing`. - -**Severity** -: How much a check result matters: OK, INFO (negligible), WARNING (degraded), - CRITICAL (the emulator cannot start games). - -**Discrepancy** -: A file that passes the setup's own check but fails the emulator's size or - hash check. - -**hle_fallback** -: A flag on a profile entry: the emulator has HLE for this file, so its - absence is reported at INFO. - -**Variant** -: Another version of a BIOS file (a revision, a region or another dump), kept - in `.variants/` in the repository. - -**Cross-reference** -: The comparison of emulator profiles with a setup's own list, which finds the - files the setup's emulators load and its list does not name. - -**Shared group** -: A group of files in `_shared.yml` that several setups include, with the - folder an emulator expects. - -**Data directory** -: A whole folder an emulator needs, such as Dolphin's `Sys/` or the PPSSPP - assets, kept in `data/` and not indexed in the database. - -**Storage tier** -: Where a file comes from: `embedded` (in `bios/`), `external` (downloaded at - build time) or `user_provided`. - -**Target** -: A hardware architecture a setup runs on, such as `switch`, `rpi4` or - `x86_64`. - -**Manifest** -: The JSON list of files per setup in `install/`, which the installer reads. - -**Scraper** -: A script that reads a setup's own BIOS list (System.dat, es_bios.xml and - others) into this project's format. - -**Exporter** -: A script that writes the project's data back in a setup's own format. - -**Truth** -: A setup's file list generated from the emulator profiles, compared with the - list the setup publishes. - -**Dump catalog** -: A preservation project (Redump, No-Intro, TOSEC) that publishes lists of - verified hardware dumps. - -**Provenance** -: The catalogs that list a file, joined to the database by hash. It informs, - it never overrides the emulator code. +: Files from your own copy of a PC game, like Doom. Not a BIOS. !!! tip "Next" - - [What BIOS files are](what-are-bios-files.md) + - [What is a BIOS?](what-are-bios-files.md) - [Get your files](../which-pack.md) + - Reference: [Terms of the reference pages](reference-terms.md) diff --git a/wiki/index.md b/wiki/index.md index dfe00c8a..ce552536 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -12,7 +12,7 @@ toolchain works. To set up BIOS files on a device, start with - **[How each platform checks files](verification-modes.md)** - existence, MD5 and SHA1 modes, severity matrix, resolution chain, reading `verify.py` output - **[What the packs do not hold](../gaps.md)** - files emulators load that no pack can carry yet, by platform and by emulator -- **[Cross-reference](../cross-reference.md)** - files the emulators of a platform load and its own list does not name +- **[Cores by platform](../cross-reference.md)** - every core each platform runs, the systems it emulates, the upstream project it comes from - **[Dump provenance](../provenance.md)** - how the collection lines up against the No-Intro, Redump and TOSEC catalogs - **[Updated platform files](../exports.md)** - each platform's own file, with the changes the emulator source supports - **[Technical questions](technical-faq.md)** - multiple hashes, MAME versions, UNTESTED, dump verification @@ -53,7 +53,8 @@ at run time, so the source is the primary reference: See [contributing](../contributing.md) for submission guidelines. -## Glossary +## Terms -The words these pages use, from BIOS to provenance, are in the -[glossary](glossary.md). +The words these pages use, from profile to provenance, are in +[Terms of the reference pages](reference-terms.md). The everyday words are in +the [glossary](glossary.md). diff --git a/wiki/installer.md b/wiki/installer.md index 27939b74..fd6ce788 100644 --- a/wiki/installer.md +++ b/wiki/installer.md @@ -4,9 +4,19 @@ level: explore # All installer options -Reference for `install.sh`, `install.ps1` and `install.py`: what the one-liner -runs, what it detects, where the bytes come from and what it is allowed to -write. For the BIOS folder of each setup, see +The one-line command finds your setup and its BIOS folder. It downloads only +the files that are missing or wrong, and checks each one against its +fingerprint before writing it. The options below are for anything else: fewer +files (one console, one region, only the cores you have), another folder, an +SD card, a check without download, or a run from a script. + +Example: `--platform retroarch --system sony-playstation --region north-america` +installs only the PlayStation files, and the North American version where +there is one per region. + +How the command protects your machine is under +[What the bootstrap does](#what-the-bootstrap-does) and +[Trust boundary](#trust-boundary). The BIOS folder of each setup is on [Get your files](../which-pack.md). ## One line @@ -50,43 +60,6 @@ from the text and calling it passes the arguments through: --platform retroarch --dest D:\bios ``` -## What the bootstrap does - -Both wrappers do the same four things before any BIOS file is considered. - -They fetch `install.py` over HTTPS only, from `RETROBIOS_INSTALL_URL` or the -pinned raw URL. A URL with any other scheme is refused outright. - -They cap the download at 2 MB and require the SHA-256 to equal the 64 hex -characters embedded in the wrapper, `RETROBIOS_INSTALL_SHA256` when set. A -mismatch aborts with "the downloaded installer does not match its expected -fingerprint" and nothing runs. Right after an update of this project the -wrapper and `install.py` can come from caches a few minutes apart, which reads -the same way: waiting five minutes settles it. Any change to `install.py` -therefore means recomputing that pin in both wrappers. - -They locate a Python 3.8 or newer interpreter (`python3` then `python` on -POSIX, `py -3` then `python3` then `python` on Windows). Without one, -`install.sh` names the command that installs it on the system it runs on -(`pkg install python` in Termux, which it offers to run since Termux installs -without root, `xcode-select --install` on macOS, apt, dnf, pacman, zypper or apk -from `/etc/os-release`), and `install.ps1` at a console offers to run `winget -install Python.Python.3.12`, then reads the new PATH and carries on. -`install.sh` needs `curl` or `wget` (GNU or BusyBox) and `sha256sum` or `shasum`. -Windows PowerShell 5.1 still negotiates SSL 3.0 and TLS 1.0 by default and -GitHub refuses both, so `install.ps1` raises the floor to TLS 1.2 before -downloading, and `install.sh` pins the same floor with `curl --tlsv1.2`. - -They delete the temporary installer on the way out, including on interrupt. -Run from a clone, `install.sh` and `install.ps1` reuse the `install.py` sitting -next to them and skip the download entirely. Piped from stdin, `install.sh` -never treats the working directory as a trusted location for it. - -Piped from `curl`, the shell's standard input is the script itself, so -`install.sh` hands the installer the terminal (`/dev/tty`) when there is one. -Without that, every question the installer can ask was skipped on Linux and -macOS. - ## What the installer does 1. Detects the host OS and the platforms installed on it. @@ -131,6 +104,119 @@ ones, grouped by system with the emulators that read them, and point at the emulator pages. They do not change the exit code of `--check`, since no run of the installer can fetch them. +## Reading a run { #reading-a-run } + +Each line a first install prints, and what it means. The run below is a +Batocera install into an empty folder, from the list as published on +2026-10-21. The figures change with each release. + +### Before the run + +The size depends on the platform and changes with each release. The run prints +it before downloading, and asks first when the download reaches 1 GB. The +installer also measures the free space and refuses before writing anything if +there is not enough. + +`curl` or `wget`, and Python 3.8 or newer, are checked by the command itself: +it stops with a message naming what is absent. PowerShell's execution policy +does not block the Windows line. The policy governs script files, and the +command never writes one. + +### The first lines + +``` +RetroBIOS 8c0e9c762d2f + +Detecting platform... + Found Batocera at /userdata/bios +``` + +The code after RetroBIOS identifies the installer: it is the start of the +SHA-256 the command checked before running it. The path is the folder it will +write to. + +### The file index and the download + +Two phases follow. The first reads what is already in the folder. It takes a +few minutes on a memory card, because every file is hashed: + +``` +Fetching file index for batocera... + 14481 files (9.1 GB) + Not installable: 399 files (79 required, not found: 399). + +Checking existing files... + 0/14481 present (0 verified, 0 wrong hash) +``` + +`Not installable` counts the files of the list that the collection does not +hold. They are named at the end of the run. + +On a first install the present count is zero. On a later run it counts what +is already there, and only the rest is downloaded. A download of 1 GB or more +is confirmed first (`Download 9.1 GB to /userdata/bios? [Y/n]`). + +The second phase downloads what is missing, one line per file: + +``` +Downloading 14481 files (9.1 GB)... + Press Ctrl+C to stop. Running the same command again carries on from here. + [1/14481] 3do_arcade_saot.bin ok + [2/14481] 3dobios.zip ok +``` + +Ctrl+C is safe at any point. Files already installed stay, and running the +command again picks up where it stopped. + +### The end of the run + +``` +Done. 14481 files installed, 0 files already up to date. +Location: /userdata/bios +399 files were not installed: the collection does not hold them, or its copy fails their check. + Needed to start an emulator (79 files): + dark-forces (theforceengine): DARK.GOB, SOUNDS.GOB, SPRITES.GOB, TEXTURES.GOB + dark-forces-ii (openjkdf2): openjkdf2/episode/JK1.gob, openjkdf2/episode/JK1CTF.gob, openjkdf2/episode/JK1MP.gob, openjkdf2/resource/Res1hi.gob, openjkdf2/resource/Res2.gob + diablo (devilutionx): devilutionx/diabdat.mpq + doom3 (dhewm3): base/pak000.pk4, base/pak002.pk4, base/pak003.pk4, base/pak004.pk4 + droidworks (openjkdf2): dwCD.GOB, dwHD.GOB, dwMin.GOB, dwStream.GOB + fallout (fallout1-ce): fallout1-ce/critter.dat, fallout1-ce/master.dat + fallout2 (fallout2-ce): fallout2-ce/critter.dat, fallout2-ce/master.dat + fury (fury): fury.grp + ... and 9 more systems (--check --verbose lists them all) + Each emulator's page says what these files are: https://abdess.github.io/retrobios/emulators/ +To check this later, run the same command with --check. +``` + +The word `Done` and a location are the confirmation. Any other ending +(`Stopped`, `Finished with errors`, `Nothing was installed`) means something +was left undone, and the run says which case it was. + +The lines between `Location:` and the `--check` hint name the files of the +list that the collection does not hold. `Needed to start an emulator` gives, +for each system, the emulators that read it and the files they cannot start +without. On the run above, all 79 belong to games (Dark Forces, Diablo, Doom 3, +Tomb Raider and the others): the engine reads them from a copy of the game +itself. The other 320 are optional, and the emulators reading them start +without them. + +### A later check + +The same command with `--check` reads the folder and writes nothing: + +``` +Checking existing files... + 14481/14481 present (14481 verified, 0 wrong hash) + + All files up to date. +``` + +The files the collection does not hold are listed after it again, as long as +they are absent from the folder. A number before `wrong hash` means a file is +present with contents the platform does not accept. Running the install again +replaces it and keeps the old one, see +[Files you already had](#files-you-already-had). + ## Files you already had A file in the BIOS folder whose contents differ from the expected one (another @@ -338,46 +424,6 @@ whose contents do not match is re-downloaded and replaced by the verified copy, the old one kept beside the BIOS folder. Files the list does not name are never read, moved or deleted. -## Machine-readable events - -With `--events`, standard output carries only JSON objects, one per line, and -the text a person reads goes to standard error. A program driving the installer -(the [handheld port](#on-a-handheld-the-retrobios-port) is one) reads the -objects and never parses the text. Every object has an `event` key naming its -kind, and each kind has exactly the keys below, in this order. The schema is -`EVENT_KEYS` in `install.py`, and a test holds this table to it. - -| Kind | Keys | When | -|------|------|------| -| `start` | `schema`, `installer`, `mode` | First line. `schema` is 1, `installer` the identity `--version` prints, `mode` one of `install`, `check`, `dry-run`, `list` | -| `detect` | `os`, `os_name`, `platforms` | Once the platforms are known. `os_name` comes from `/etc/os-release`. `platforms` lists `{platform, name, bios_dir}`, and is empty when nothing supported was found | -| `error` | `stage`, `reason`, `platform`, `message` | The file list could not be read (`stage` is `manifest`). `reason` is `network`, `not_found` or `invalid` | -| `file` | `platform`, `dest`, `systems`, `size`, `state` | Each file of the list after the check. `state` is `present` (hash verified), `wrong_hash` or `missing` | -| `system` | `platform`, `system`, `name`, `files`, `present`, `missing`, `wrong_hash`, `bytes_needed` | Each system, sorted by id. Files with no system count under `other`. `name` is the platform's own name for it | -| `check` | `platform`, `name`, `bios_dir`, `files`, `present`, `missing`, `wrong_hash`, `bytes_needed`, `data_missing`, `not_installable` | The platform's totals. `data_missing` counts data-tree files to fetch, `not_installable` the files the collection cannot supply | -| `download` | `platform`, `files`, `bytes` | Before the downloads start | -| `progress` | `files_done`, `files_total`, `bytes_done`, `bytes_total` | At most every half second while bytes arrive | -| `downloaded` | `dest`, `ok`, `reason`, `size`, `files_done`, `files_total`, `bytes_done`, `bytes_total` | Each finished file. `reason` says why one failed, and is empty otherwise | -| `summary` | `mode`, `installed`, `up_to_date`, `pending`, `failed`, `skipped`, `left_out`, `kept`, `interrupted` | The outcome. `pending` counts what a next run would still fetch, `failed` lists `{dest, reason}`, `skipped` lists `{reason, label, files, required}` for the files the collection cannot supply, `left_out` lists `{category, label, files, dests}` for what `--no-game-data` left out, and is empty without it. `kept` lists `{folder, files, bytes}`, one per backup folder, for the files in place the run moved aside before replacing them (with `--dry-run`, would move), and is empty when none. Not written for a listing | -| `end` | `code` | Last line, with the exit status the process returns (see [Exit status](#exit-status)) | - -An option refused before the run starts (`install.py: error:`, code 2) writes -no event at all. A program reads a missing `end` as that case. - -## Environment overrides - -| Variable | Effect | -|----------|--------| -| `RETROBIOS_REF` | Branch or tag the file list and the files committed to the repository are read from. Defaults to `main`. Files the manifest marks `release_asset` always come from the `large-files` release, whatever the ref, and that release replaces an asset in place when it is rebuilt. Each release deletes the previous release tags ([release process](release-process.md#cutting-a-release)), so a tag holds a file list until the next release | -| `RETROBIOS_BASE_URL` | Base URL for the manifest and the files it declares. HTTPS only, plain HTTP allowed to loopback for end-to-end tests. Whoever controls this base controls the expected hashes, hence the check. A file list that cannot be fetched names the address, and this variable when it is set | -| `RETROBIOS_OS` | Names the host outright (`linux`, `wsl`, `windows`, `darwin`, `android`) instead of detecting it | -| `EXTERNAL_STORAGE` | Android only, the shared storage root RetroArch builds its default directories from. Defaults to `/storage/emulated/0` | -| `RETROBIOS_INSTALL_URL` | Where the bootstrap fetches `install.py`. HTTPS only | -| `RETROBIOS_INSTALL_SHA256` | The SHA-256 the bootstrap requires of that installer, 64 hex characters. When either of these two is set and the fingerprint does not match, the error names the variable instead of suggesting a retry | -| `RETROBIOS_BOOTSTRAP` | Set by `install.sh` (`sh`) and `install.ps1` (`ps1`) for the temporary installer they run, so that its hints give back the one-line command rather than `python3 install.py` | -| `NO_COLOR` | Plain text, no colour, even at a terminal. `TERM=dumb` also turns the live progress line off | -| `HTTPS_PROXY`, `NO_PROXY` | Honoured for every download through the standard library's default proxy handling: one names the proxy, the other the hosts to reach directly | - ## Platform detection Detection reads what the system itself writes, never a guess from a directory @@ -743,6 +789,43 @@ counts the data files missing, `--dry-run` names each tree it would fetch, and narrowing with `--system`, `--core` or `--target` keeps only the trees of the systems that stay. +## What the bootstrap does + +Both wrappers do the same four things before any BIOS file is considered. + +They fetch `install.py` over HTTPS only, from `RETROBIOS_INSTALL_URL` or the +pinned raw URL. A URL with any other scheme is refused outright. + +They cap the download at 2 MB and require the SHA-256 to equal the 64 hex +characters embedded in the wrapper, `RETROBIOS_INSTALL_SHA256` when set. A +mismatch aborts with "the downloaded installer does not match its expected +fingerprint" and nothing runs. Right after an update of this project the +wrapper and `install.py` can come from caches a few minutes apart, which reads +the same way: waiting five minutes settles it. Any change to `install.py` +therefore means recomputing that pin in both wrappers. + +They locate a Python 3.8 or newer interpreter (`python3` then `python` on +POSIX, `py -3` then `python3` then `python` on Windows). Without one, +`install.sh` names the command that installs it on the system it runs on +(`pkg install python` in Termux, which it offers to run since Termux installs +without root, `xcode-select --install` on macOS, apt, dnf, pacman, zypper or apk +from `/etc/os-release`), and `install.ps1` at a console offers to run `winget +install Python.Python.3.12`, then reads the new PATH and carries on. +`install.sh` needs `curl` or `wget` (GNU or BusyBox) and `sha256sum` or `shasum`. +Windows PowerShell 5.1 still negotiates SSL 3.0 and TLS 1.0 by default and +GitHub refuses both, so `install.ps1` raises the floor to TLS 1.2 before +downloading, and `install.sh` pins the same floor with `curl --tlsv1.2`. + +They delete the temporary installer on the way out, including on interrupt. +Run from a clone, `install.sh` and `install.ps1` reuse the `install.py` sitting +next to them and skip the download entirely. Piped from stdin, `install.sh` +never treats the working directory as a trusted location for it. + +Piped from `curl`, the shell's standard input is the script itself, so +`install.sh` hands the installer the terminal (`/dev/tty`) when there is one. +That keeps the installer's questions working when the script is piped from +`curl`. + ## Trust boundary The manifest is fetched over the network and treated as untrusted input. Every @@ -802,7 +885,134 @@ in the session. Failures that are per-file, such as a download exhausting its three attempts, are counted and reported at the end. +## Machine-readable events + +With `--events`, standard output carries only JSON objects, one per line, and +the text a person reads goes to standard error. A program driving the installer +(the [handheld port](#on-a-handheld-the-retrobios-port) is one) reads the +objects and never parses the text. Every object has an `event` key naming its +kind, and each kind has exactly the keys below, in this order. The schema is +`EVENT_KEYS` in `install.py`, and a test holds this table to it. + +| Kind | Keys | When | +|------|------|------| +| `start` | `schema`, `installer`, `mode` | First line. `schema` is 1, `installer` the identity `--version` prints, `mode` one of `install`, `check`, `dry-run`, `list` | +| `detect` | `os`, `os_name`, `platforms` | Once the platforms are known. `os_name` comes from `/etc/os-release`. `platforms` lists `{platform, name, bios_dir}`, and is empty when nothing supported was found | +| `error` | `stage`, `reason`, `platform`, `message` | The file list could not be read (`stage` is `manifest`). `reason` is `network`, `not_found` or `invalid` | +| `file` | `platform`, `dest`, `systems`, `size`, `state` | Each file of the list after the check. `state` is `present` (hash verified), `wrong_hash` or `missing` | +| `system` | `platform`, `system`, `name`, `files`, `present`, `missing`, `wrong_hash`, `bytes_needed` | Each system, sorted by id. Files with no system count under `other`. `name` is the platform's own name for it | +| `check` | `platform`, `name`, `bios_dir`, `files`, `present`, `missing`, `wrong_hash`, `bytes_needed`, `data_missing`, `not_installable` | The platform's totals. `data_missing` counts data-tree files to fetch, `not_installable` the files the collection cannot supply | +| `download` | `platform`, `files`, `bytes` | Before the downloads start | +| `progress` | `files_done`, `files_total`, `bytes_done`, `bytes_total` | At most every half second while bytes arrive | +| `downloaded` | `dest`, `ok`, `reason`, `size`, `files_done`, `files_total`, `bytes_done`, `bytes_total` | Each finished file. `reason` says why one failed, and is empty otherwise | +| `summary` | `mode`, `installed`, `up_to_date`, `pending`, `failed`, `skipped`, `left_out`, `kept`, `interrupted` | The outcome. `pending` counts what a next run would still fetch, `failed` lists `{dest, reason}`, `skipped` lists `{reason, label, files, required}` for the files the collection cannot supply, `left_out` lists `{category, label, files, dests}` for what `--no-game-data` left out, and is empty without it. `kept` lists `{folder, files, bytes}`, one per backup folder, for the files in place the run moved aside before replacing them (with `--dry-run`, would move), and is empty when none. Not written for a listing | +| `end` | `code` | Last line, with the exit status the process returns (see [Exit status](#exit-status)) | + +An option refused before the run starts (`install.py: error:`, code 2) writes +no event at all. A program reads a missing `end` as that case. + +## Environment overrides + +| Variable | Effect | +|----------|--------| +| `RETROBIOS_REF` | Branch or tag the file list and the files committed to the repository are read from. Defaults to `main`. Files the manifest marks `release_asset` always come from the `large-files` release, whatever the ref, and that release replaces an asset in place when it is rebuilt. Each release deletes the previous release tags ([release process](release-process.md#cutting-a-release)), so a tag holds a file list until the next release | +| `RETROBIOS_BASE_URL` | Base URL for the manifest and the files it declares. HTTPS only, plain HTTP allowed to loopback for end-to-end tests. Whoever controls this base controls the expected hashes, hence the check. A file list that cannot be fetched names the address, and this variable when it is set | +| `RETROBIOS_OS` | Names the host outright (`linux`, `wsl`, `windows`, `darwin`, `android`) instead of detecting it | +| `EXTERNAL_STORAGE` | Android only, the shared storage root RetroArch builds its default directories from. Defaults to `/storage/emulated/0` | +| `RETROBIOS_INSTALL_URL` | Where the bootstrap fetches `install.py`. HTTPS only | +| `RETROBIOS_INSTALL_SHA256` | The SHA-256 the bootstrap requires of that installer, 64 hex characters. When either of these two is set and the fingerprint does not match, the error names the variable instead of suggesting a retry | +| `RETROBIOS_BOOTSTRAP` | Set by `install.sh` (`sh`) and `install.ps1` (`ps1`) for the temporary installer they run, so that its hints give back the one-line command rather than `python3 install.py` | +| `NO_COLOR` | Plain text, no colour, even at a terminal. `TERM=dumb` also turns the live progress line off | +| `HTTPS_PROXY`, `NO_PROXY` | Honoured for every download through the standard library's default proxy handling: one names the proxy, the other the hosts to reach directly | + ## When something goes wrong -See [Troubleshooting](troubleshooting.md#installation-script-fails) for network, -permission and detection failures. +The common cases, with the short answer, are on +[When a game does not start](troubleshooting.md). The details follow. + +### Network + +The installer fetches its manifest, then downloads only the repository or +`large-files` release objects it needs. When a download fails, check that +`https://github.com` opens. Behind a proxy, set `HTTPS_PROXY` in the +environment. A file that fails three attempts is counted and reported at the +end, and the next run fetches only what is still missing. + +### Permission denied + +The installer needs to write to the BIOS folder. On Linux and macOS, check who +owns it (`ls -la`). On Batocera, Recalbox and ROCKNIX the folder belongs to +root, so the same command with `sudo` in front of it succeeds. On Windows, run +PowerShell as Administrator only for a protected folder. + +A file system mounted read-only refuses root too, so the installer does not +suggest `sudo` for it. Linux remounts a card or drive read-only after an error, +an unclean removal for one: check it (`fsck`, or `chkdsk` on Windows) and +mount it read-write again. + +A file already in the BIOS folder with the read-only attribute is replaced +like any other: on Windows the installer clears the attribute first. +`file in place is read-only` appears only when Windows refuses that too, which +a file owned by another account can cause. + +### Platform not detected + +The installer finds a platform from the config files it knows. When it finds +none, name the platform and its BIOS folder: + +```bash +curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform retroarch --dest ~/.config/retroarch/system +curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform batocera --dest /userdata/bios +``` + +```powershell +&([scriptblock]::Create((irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1))) ` + --platform retroarch --dest C:\RetroArch-Win64\system +``` + +`--list-platforms` lists the platforms the installer knows and the ones it +finds here. `--help` lists the options with examples. + +### No file list + +The installer reads its manifests from the revision the bootstrap checked it +against. A platform name is checked before anything is downloaded, so a +"No file list" message comes from the revision: `RETROBIOS_REF` naming a tag +that does not exist, or one older than the platform. Unset it, or pick a tag +from the releases page ([Environment overrides](#environment-overrides)). + +### Upper and lower case + +Linux file systems tell `Bios.ROM` and `bios.rom` apart. Windows and macOS do +not, by default. + +The RetroArch pack is built for every system RetroArch runs on, so it holds +one name per file: `BIOS.col`, never `bios.col` beside it, since unzipping both +on Windows or macOS would overwrite one with the other. On Linux, Mesen-CE +opens `bios.col` and NP2kai opens the lowercase `np2kai/2608_*.wav`, which the +pack only holds as `BIOS.col` and `2608_*.WAV`. The installer checks whether +the BIOS folder keeps case and, when it does, also fetches these names. On +Linux, use the installer rather than unzipping the pack by hand. + +The reverse happens on Batocera and Recalbox, whose lists hold both spellings +because their systems keep case: NP2kai reads `np2kai/bios.rom` and +`np2kai/BIOS.ROM` as two files. Written to a folder that folds case (the share +opened from Windows, an exFAT card), one overwrites the other. The installer +checks the folder first, keeps one file per name there and lists the ones it +left out. + +### Packs and their parts + +`scripts/download.py --list` and `scripts/download.sh --list` print the packs of +a release. A pack's name lists every platform it serves. Releases up to +v2026.10.10 ship RetroArch and Lakka as one `RetroArch_Lakka` pack, and the +next ones give Lakka its own, since Lakka runs on Linux where two names that +differ by case are two files. The installer and the download scripts find the +right pack either way. + +Parts named `Pack.part1of2.zip` are ordinary ZIPs: unzip each one into the same +folder, and check that all of them were downloaded. The name says how many +there are. `scripts/download.sh` and `scripts/download.py` download every part +and check it against the release's checksums before unzipping. Both stage +inside the destination folder, never in `/tmp`, which is a RAM disk on most of +these systems. diff --git a/wiki/legal.md b/wiki/legal.md new file mode 100644 index 00000000..b751783f --- /dev/null +++ b/wiki/legal.md @@ -0,0 +1,45 @@ +--- +level: explore +--- + +# The project's legal position + +Redistributing firmware is not settled law. Below is the reasoning the project acts on, with the strength of each argument, so anyone can weigh it. None of it is legal advice, and none of it has been tested in court. + +## Emulation and BIOS redistribution + +- **Writing an emulator is lawful.** *Sony v. Connectix* (2000) and *Sega v. Accolade* (1992) held that copying a work during reverse engineering, to reach the functional elements needed for interoperability, is fair use. Both concern the act of copying while developing, and *Connectix* specifically involved copying Sony's BIOS in the course of that work. Neither decision addresses distributing firmware to third parties, so they support the legality of emulation itself rather than of this collection. +- **Fair use (US, 17 USC 107) is a defence, argued case by case.** The project's position is that non-commercial redistribution for personal backup and archival weighs favourably on purpose and on market effect, since these files are not sold separately. It weighs poorly on the amount used: each file is copied whole. Manufacturers do still monetise some of this firmware indirectly, through re-releases, subscription services and mini consoles, so the market factor is contestable rather than clear. +- **Interoperability in the EU.** The Software Directive (2009/24/EC, Art. 5-6) permits decompilation and use to achieve interoperability, subject to conditions, and Art. 6 restricts passing the resulting information to others. It supports the emulation use case, but it is not a general redistribution permission. +- **Discontinued hardware.** Most firmware here is for hardware no longer sold or supported. This bears on the fair-use market factor and on the practical likelihood of complaint. "Abandonware" is not a legal doctrine in any jurisdiction, and nothing here rests on it. + +## Encryption keys (Switch prod.keys, 3DS AES keys, Wii U keys) + +This is the most contested area, and the weakest part of the project's position. + +- **Keys are unlikely to be copyrightable.** A bare numeric value has no author's creative expression, so copyright is a poor fit. This says nothing about the DMCA, which is the provision that actually applies. +- **DMCA 1201(f) is narrow.** Section 1201(f) allows a person who has lawfully obtained the right to use a copy to circumvent for the sole purpose of achieving interoperability with an independently created program, and 1201(f)(3) allows the means to be made available to others only for that same purpose. Publishing keys openly is not obviously within that allowance, and the trafficking prohibitions in 1201(a)(2) and 1201(b) remain in play. This is the clearest legal risk the project carries. +- **Library of Congress exemptions do not cover this repository.** The triennial exemptions at 37 CFR 201.40 let an eligible library, archive or museum circumvent access controls to preserve a lawfully acquired video game whose server support has ended, with access limited to the institution's premises. The Ninth Triennial Proceeding (2024 cycle) renewed them and declined to extend them to off-premises remote access. A public repository is neither an eligible institution nor an on-premises reading room, so these exemptions show which way preservation policy is moving without authorising what is done here. +- **Keys come from consumer hardware.** They are extracted from retail devices their owners bought. *Chamberlain v. Skylink* (2004) held that a 1201 claim requires some nexus to copyright infringement, in a dispute over a garage-door opener. Later cases have narrowed its reach, and it is a weak foundation for a general post-sale right. +- **Trade secret is not a live issue.** Keys embedded in millions of shipped devices and widely republished are not kept secret by reasonable measures. + +## Recent firmware (Switch 19.0.0, PS3UPDAT, PSVUPDAT) + +- **Free download is not a licence.** Manufacturers publish these updates on open CDNs without authentication, which makes them easy to obtain but does not grant a right to redistribute them, and PlayStation firmware ships under an end-user licence agreement. The practical argument is that mirroring a file the manufacturer already gives away for free causes no identifiable loss, not that doing so is expressly permitted. +- **Functional necessity.** Emulators cannot run the software these systems were built for without this firmware. That is why the files are collected. It is a statement of purpose, not a legal defence in itself. +- **Yuzu context.** The Yuzu settlement (2024) concerned the emulator and its alleged facilitation of piracy. Yuzu settled without admitting liability, so it set no precedent either way on firmware or key redistribution. It does show that rights holders in this space litigate. + +## Summary + +This project preserves BIOS files, firmware and keys for personal use, archival and interoperability. The strongest ground is the older, discontinued firmware, where fair use and the absence of a market both point the same way. The weakest is the encryption keys, where DMCA 1201 applies and the interoperability exemption is narrower than the use made of it here. The project accepts that risk deliberately, in the belief that these files are worth preserving while they can still be obtained. + +This reflects the project's good-faith understanding, not legal advice. + +## Asking for a file to be removed + +A rights holder, or anyone acting for one, can [open an issue](https://github.com/Abdess/retrobios/issues) identifying the file by path or hash. A file whose removal is requested by its rights holder is removed from the repository and from the next release. No formal notice is required and none will be demanded. State the file and the basis of the claim. The same channel is open to anyone else with a concern about a specific file. + +!!! tip "Next" + + - [FAQ](faq.md) + - [What is a BIOS?](what-are-bios-files.md) diff --git a/wiki/profiling.md b/wiki/profiling.md index f5a7cba4..8d60f38f 100644 --- a/wiki/profiling.md +++ b/wiki/profiling.md @@ -144,11 +144,13 @@ degraded functionality (missing boot animation, different fonts, reduced audio in menus), it is optional. This keeps the classification objective and consistent across all profiles. -When a core has HLE (high-level emulation), the real BIOS typically -gives better accuracy, but the core functions without it. These files -are marked with `hle_fallback: true` and `required: false`. The file -still ships in packs (better experience for the user), but its absence -does not raise alarms during verification. +When a core has HLE (high-level emulation), code of its own that stands +in for the file, mark the entry `hle_fallback: true`. The flag records the +stand-in. It does not decide `required`, which still comes from what the +code does without the file: most of the 778 entries that carry the flag +are optional, and 13 are required, among them Beetle PSX's three region +BIOS (`emulators/beetle_psx.yml`). The file still ships in packs, and +verification reports its absence at INFO whatever `required` says. MAME is read the same way. A missing ROM is fatal only when it is neither `ROM_OPTIONAL` nor `NO_DUMP` (`src/emu/romload.cpp`), so a set made only of diff --git a/wiki/reference-terms.md b/wiki/reference-terms.md new file mode 100644 index 00000000..7f4d07b9 --- /dev/null +++ b/wiki/reference-terms.md @@ -0,0 +1,153 @@ +--- +level: reference +--- + +# Terms of the reference pages + +The words the reference and contributor pages use, each with a real case. The +everyday words (BIOS, emulator, core, hash) are in the [glossary](glossary.md). + +**Frontend** +: The menu you pick and start games from, such as EmulationStation, MinUI or + the RetroArch menu. It starts the emulator. + +**Pack** +: One ZIP file with every BIOS file a setup uses. A pack over 2 GB comes in + several parts, and every part is needed. + +**Profile and source_ref** +: A profile records which files one emulator opens, read from its code. + Every entry has a `source_ref`, the line of code that proves it, at the + revision named by `source_commit` (the pin). Example: + `emulators/duckstation.yml` lists `scph5501.bin` at + `src/core/bios.cpp:47`, commit `4122fed9`. + +**Alias** +: Another name the same emulator accepts for a file, listed under `aliases:` + in its profile, never as a second file. + +**Region and priority** +: `region:` is the territory for which the code picks a file. `priority:` is + the order the code tries files that fill the same role, lowest first. + Example: DuckStation gives `scph5501.bin` priority 5 and `scph1001.bin` + priority 10, so with both present it loads `scph5501.bin`. + +**Slot** +: One role a file fills for one system and region, such as the North + American PlayStation BIOS. `scph5501.bin`, `scph1001.bin` and `scph7001.bin` + fill the same slot. `--one-per-slot` keeps the one the emulator prefers. + +**Category: firmware, game data, unread** +: Each entry of a platform's list is sorted by what its emulators do with + it: firmware they load, game data, or a file none of them opens. Only + missing firmware counts as a fault. Example: AmberELEC lists + `hardwareconfig.xml` in each blueMSX machine folder, and blueMSX never + opens it, so it is `unread`. + +**Romset and recipe** +: An arcade BIOS is a ZIP of several chips, and each MAME or FBNeo version + expects a slightly different ZIP. A recipe is the list of chips (name and + CRC32) one version expects, enough to rebuild that exact ZIP from chips + already held. Example: `spec128.zip` (MD5 `1a524bfa...`) was rebuilt from + the FBNeo recipe. + +**Profile** +: A YAML file in `emulators/` that documents what one emulator loads, read + from its source code. + +**Ground truth** +: The emulator's source code, the reference every profile is checked against. + +**System** +: A console or computer being emulated, named by an id such as + `sony-playstation` or `nintendo-gba`. + +**Verification mode** +: How a setup checks a file: by its name only (existence), by MD5 or by SHA1. + Reference: [How each platform checks files](verification-modes.md). + +**Status** +: The outcome of one file check: `ok`, `untested` (present, but not the + expected hash) or `missing`. Batocera's own checker prints the same + `UNTESTED` for a file whose MD5 is not on its list (`batocera-systems`, + `checkBios`). + +**Severity** +: How much a check result matters: OK, INFO (negligible), WARNING (degraded), + CRITICAL (the emulator cannot start games). + +**Discrepancy** +: A file that passes the setup's own check but fails the emulator's size or + hash check. Example: RetroBat accepts `d64_1.rom`, while XRoar checks its + CRC32 and expects `84f68bf9`, not `60a4634c`. `verify.py` prints it as + `DISCREPANCY`, and the gaps page lists it under that name, with the check + that failed (size or hash) in the row. + +**HLE** +: High-level emulation: code inside the emulator that does the BIOS's job, + so some games start without the file. Some games glitch or do not start + with it, so the real file still ships. + +**hle_fallback** +: A flag on a profile entry: the emulator has HLE for this file, so its + absence is reported at INFO. It does not make the file optional: 13 of + the 778 entries that carry it are `required: true`, among them Beetle + PSX's `scph5501.bin`. + +**Variant** +: Another version of a BIOS file (a revision, a region or another dump), kept + in `.variants/` in the repository. + +**Cross-reference** +: The comparison of emulator profiles with a setup's own list, which finds the + files the setup's emulators load and its list does not name. The packs add + those files, and [What the packs do not hold](../gaps.md) counts them. The + page [Cores by platform](../cross-reference.md) lists the cores this + comparison reads. + +**Shared group** +: A group of files in `_shared.yml` that several setups include, with the + folder an emulator expects. + +**Data directory** +: A whole folder an emulator needs, such as Dolphin's `Sys/` or the PPSSPP + assets, kept in `data/` and not indexed in the database. + +**Storage tier** +: Where a file comes from: `embedded` (in `bios/`), `external` (downloaded at + build time) or `user_provided`. + +**Target** +: A hardware architecture a setup runs on, such as `switch`, `rpi4` or + `x86_64`. + +**Manifest** +: The JSON list of files per setup in `install/`, which the installer reads. + +**Scraper** +: A script that reads a setup's own BIOS list (System.dat, es_bios.xml and + others) into this project's format. + +**Exporter** +: A script that writes the project's data back in a setup's own format. + +**Truth** +: A setup's file list generated from the emulator profiles, compared with the + list the setup publishes. + +**Dump catalog** +: A preservation project (Redump, No-Intro, TOSEC) that publishes lists of + verified hardware dumps. + +**Provenance** +: Whether a file is byte for byte a dump that Redump, No-Intro or TOSEC + catalogues, joined to the database by hash. Example: `scph5501.bin` is + Redump's `ps-30a.bin`. It says where a dump is known from, not whether an + emulator accepts it, and it never overrides the emulator code. Where a pack + takes a file from (the collection, a data folder, a release asset) is its + source, not its provenance. + +!!! tip "Next" + + - [How each platform checks files](verification-modes.md) + - [Glossary](glossary.md) diff --git a/wiki/technical-faq.md b/wiki/technical-faq.md index 10e7dff9..4b4c0306 100644 --- a/wiki/technical-faq.md +++ b/wiki/technical-faq.md @@ -4,8 +4,32 @@ level: reference # Technical questions -Questions about how the files are identified and checked. The questions asked -before installing are in the [FAQ](faq.md). +Answers to "how do you know?" questions: how a file is identified, why one BIOS +has several fingerprints, why some files are not BIOS, and which MAME version an +arcade set belongs to. Each answer gives a real file as its example, so you can +check it on the system pages. Questions asked before installing are in the +[FAQ](faq.md). + +## What's a hash/checksum? + +A hash is a fixed-length fingerprint computed from a file's contents. If even one byte differs, the hash changes completely. Every file in the database carries these: + +| Type | Length | Example | Used for | +|------|--------|---------|----------| +| SHA1 | 40 hex chars | `10155d8d6e6e832d8ea1571511e40dfb15fede05` | database primary key, BizHawk, installer downloads | +| MD5 | 32 hex chars | `924e392ed05558ffdb115408c263dccf` | most platform verification | +| SHA256 | 64 hex chars | `9a1c...` | emulator profiles whose upstream publishes SHA256 | +| CRC32 | 8 hex chars | `2F468B96` | ROM-set matching, arcade DATs | + +Emulator profiles add Adler-32 where the code checks it, which is how Dolphin +checks its DSP ROMs (`dsp_rom.bin` and `dsp_coef.bin`, hashed byte-swapped). +`IPL.bin` is not one of them: Dolphin loads it without a hash check. + +Verification uses whichever one the platform itself uses: MD5 for Batocera, +RetroBat, Recalbox, EmuDeck, RetroDECK, RomM, ROCKNIX, KNULLI, AmberELEC, EmuELEC +and MiSTer FPGA, SHA1 for +BizHawk, and nothing at all for RetroArch, Lakka and RetroPie, which only check +that the file exists. ## How is a missing file ranked? @@ -29,19 +53,52 @@ Platforms that verify by MD5 accept specific hashes. If yours doesn't match any A filename is never allowed to stand in for a declared hash. When a file entry carries a hash, resolution matches on content and reports `hash_mismatch` if the only same-named copy disagrees. What happens next depends on the platform: an -MD5 or SHA1 platform would reject the file, so the pack omits it and says why, +MD5 or SHA1 platform would report the file as not the expected one, so the pack +omits it and says why, while an existence platform only checks presence, so the file ships and the divergence is reported instead. ## Why are there files that aren't BIOS? -A file earns its place when an emulator loads it from disk and does not bundle it. That covers actual BIOS ROMs, console firmware, arcade BIOS sets, and also game data like `prboom.wad` for the PrBoom core or soundfonts for EasyRPG. +A file earns its place when an emulator loads it from disk and does not bundle it. That covers actual BIOS ROMs, console firmware, arcade BIOS sets, and also game data, such as `doom.wad` for the PrBoom core (`category: game_data` in `emulators/prboom.yml`) or soundfonts for EasyRPG. -This follows the platforms themselves: Batocera lists `prboom.wad` in its BIOS checker, and libretro declares these files in the `firmware` field of its core info. The word "BIOS" is their vocabulary for "external file the emulator needs". Files are tagged by category (`bios`, `game_data`, `bios_zip`) in the emulator profiles, so the distinction stays visible. +This follows the platforms themselves: EmuELEC lists Diablo's `diabdat.mpq` in its BIOS list (`platforms/emuelec.yml`, destination `ports/diablo/diabdat.mpq`), and libretro declares such files in the `firmware` field of its core info. The word "BIOS" is their vocabulary for "external file the emulator needs". Files are tagged by category (`bios`, `game_data`, `bios_zip`) in the emulator profiles, so the distinction stays visible. + +A file the emulator carries inside itself is not one of them. PrBoom compiles its engine data `prboom.wad` into the core from `src/prboom_wad_data.h` and never opens it on disk (`emulators/prboom.yml`, `src/d_main.c:1449-1465`), so no PrBoom pack needs it. + +## Why is a file the emulator reads not in the pack? + +A pack brings what the user has to obtain. A file the emulator installs itself +(a font, a shader, a game database, a free replacement BIOS) is already on the +machine once the emulator is installed, so the pack leaves it out. This is +decided per build and per platform: the PPSSPP standalone release ships its +`assets/` folder, the PPSSPP libretro core does not, so RetroArch packs carry +those files and a standalone PPSSPP pack does not. When a platform's own +package installs data beside a core (Batocera with blueMSX), that platform's +pack leaves it out too. The emulator's page marks these files as installed with +the emulator. + +## What's HLE? + +HLE (high-level emulation) is code inside an emulator that does the BIOS's job, +so some games start without the file. Beetle PSX, for one, carries OpenBIOS for +that (`emulators/beetle_psx.yml`, notes). The real file still ships, because +the stand-in is not perfect: some games glitch or do not start. + +A profile entry records it as `hle_fallback: true`. That flag does not decide +`required`: 13 of the 778 entries that carry it are required, among them Beetle +PSX's three region BIOS. Verification reports a missing `hle_fallback` file at +INFO either way. + +## How often are packs updated? + +The platforms' own lists (libretro's System.dat, batocera-systems and the +others) are read again by hand, and the difference is reviewed before it lands. +A release follows when files are added or profiles change. ## Are these files verified against original hardware dumps? -No, and that is a deliberate boundary. The hashes here document what +No. The hashes here document what emulator code loads and accepts, read from the source. When several revisions or regions pass, all of them are kept (`.variants/` in the repository, per-emulator pages list each one), and the pack ships the @@ -75,27 +132,6 @@ compiles two MAME versions of its own, 0.258 and 0.278, profiled as ships the sets matching the core you run, and a generic unversioned arcade pack cannot do that. -## What's a hash/checksum? - -A hash is a fixed-length fingerprint computed from a file's contents. If even one byte differs, the hash changes completely. Every file in the database carries these: - -| Type | Length | Example | Used for | -|------|--------|---------|----------| -| SHA1 | 40 hex chars | `10155d8d6e6e832d8ea1571511e40dfb15fede05` | database primary key, BizHawk, installer downloads | -| MD5 | 32 hex chars | `924e392ed05558ffdb115408c263dccf` | most platform verification | -| SHA256 | 64 hex chars | `9a1c...` | emulator profiles whose upstream publishes SHA256 | -| CRC32 | 8 hex chars | `2F468B96` | ROM-set matching, arcade DATs | - -Emulator profiles add Adler-32 where the code checks it, which is how Dolphin -checks its DSP ROMs (`dsp_rom.bin` and `dsp_coef.bin`, hashed byte-swapped). -`IPL.bin` is not one of them: Dolphin loads it without a hash check. - -Verification uses whichever one the platform itself uses: MD5 for Batocera, -RetroBat, Recalbox, EmuDeck, RetroDECK, RomM, ROCKNIX, KNULLI, AmberELEC, EmuELEC -and MiSTer FPGA, SHA1 for -BizHawk, and nothing at all for RetroArch, Lakka and RetroPie, which only check -that the file exists. - ## Why does my verification report say UNTESTED? `untested` means the file exists on disk but its hash does not match the expected value. This happens on MD5 and SHA1 platforms (Batocera, Recalbox, BizHawk, ROCKNIX, MiSTer FPGA, and the rest) when the file is present but contains different data than what the platform declares. diff --git a/wiki/testing-guide.md b/wiki/testing-guide.md index f3ee044c..43bf98bb 100644 --- a/wiki/testing-guide.md +++ b/wiki/testing-guide.md @@ -51,7 +51,7 @@ Every module in `tests/`, by the area it guards. | Handheld port | stubbed PortMaster dialogs, fake installer | `test_port` | | Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` | | Scrapers, upstream and provenance | inline sources, synthetic | `test_check_freshness`, `test_dedup`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_profile_sync`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_scraper_contract`, `test_upstream`, `test_upstream_index` | -| Site, docs and repository contracts | rendered pages, sources | `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_doc_pages`, `test_doc_surfaces`, `test_optional_dependencies`, `test_prose_style`, `test_site_availability`, `test_site_cores`, `test_site_emulator_page`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_platform_hashes`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` | +| Site, docs and repository contracts | rendered pages, sources | `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_doc_pages`, `test_doc_surfaces`, `test_optional_dependencies`, `test_prose_style`, `test_site_availability`, `test_site_copy`, `test_site_cores`, `test_site_emulator_page`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_platform_hashes`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_workflow_paths` | ## Test architecture @@ -325,6 +325,40 @@ what it is not. A reply on an issue or a pull request thanks once, gives the facts, describes the script or the file rather than the person, and leaves the door open. +### Pages for newcomers + +The Start here pages and the guide part of each setup page are read by someone +who knows nothing about BIOS files, often on a phone. Each word costs that +reader attention, so `tests/test_site_copy.py` holds them to numbers measured +on the generated pages in `docs/`, code blocks left out: + +| Budget | Start here | Setup guide | +|--------|-----------:|------------:| +| Longest sentence | 20 words | 25 words | +| Median sentence | 12 words | 14 words | +| Longest paragraph, list item or definition | 35 words | 40 words | +| Words before the first button or command | 50 (home page 40) | 90 | +| Items in one list | 5 | 6 | +| Heading | 8 words | 8 words | + +Each Start here page has its own ceiling (home 220 words, FAQ 450), a setup +guide 220 and the setup index 300. Of a group of tabs only the first counts +toward a page's words, since a reader opens one. A folded block counts toward +the words, not toward what comes before the first command. + +The same test refuses filler ("simply", "just", "in order to", "this page", +"what follows") and words a newcomer does not know yet, unless the sentence +explains them: "Termux, a terminal app", "SSH, a terminal over the network", +"its hash, a fingerprint of its contents". A word that needs more than that +(manifest, libretro, revision, standalone) belongs one level down, behind a +link labelled with that level ("Details:", "Reference:"). On Start here pages +a pack is a ZIP and a platform is a setup. + +The reasons: a reader holds about four things at once, scans more than reads, +and reads less on a phone. Facts a newcomer does not act on (legal analysis, +source lines, annotated runs) are kept, on the page one level down. To see +the numbers of every page, run `python -m tests.test_site_copy --report docs`. + ## Landing a change A commit subject is English, 50 characters at most, `type: description` with a diff --git a/wiki/troubleshooting.md b/wiki/troubleshooting.md index fabf451b..13f4b16c 100644 --- a/wiki/troubleshooting.md +++ b/wiki/troubleshooting.md @@ -4,259 +4,80 @@ level: start # When a game does not start -Start here when a game does not start, or says that a BIOS file is missing. -Each section is one check, in the order that finds the cause soonest. +Most of the time, a BIOS file is missing or is the wrong version. Three checks +find it. -## Game won't start / black screen +## 1. Check your BIOS folder { #checking-what-you-installed } -Most launch failures are caused by a missing or incorrect BIOS file. + -### Check what is in your BIOS folder +Run the command with `--check`. It changes nothing. -```bash +```sh curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --check ``` -It hashes what is there and writes nothing. Anything counted as needing -downloading is absent, and a required file that is absent means the core cannot -start games for that system at all. `--system` narrows the check to one -system (`--list-systems` names them): - -```bash -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --check --system sony-playstation --verbose -``` - -### Check if the hash matches - -The count `--check` prints says how many files are present with contents the -platform does not accept: +Read the count line: ``` -Checking existing files... 1/109 present (0 verified, 1 wrong hash) ``` -With `--verbose` the list under it names each one (`wrong hash: scph5501.bin`). -The check hashes every file, on RetroArch too, where the frontend itself only -looks for the name. Common causes: +- **`wrong hash`**: a file is there, but its contents are wrong. It may be for + another region or model, or damaged. +- **Fewer present than the total**: some files are missing. +- **`0 wrong hash` and all present**: your files are fine. Go to step 3. -- Wrong region (a PAL BIOS instead of NTSC, or vice versa) -- Wrong hardware revision (e.g. SCPH-5501 vs SCPH-1001 for PlayStation) -- Corrupted download +## 2. Fix the files -Running the same command without `--check` replaces each file with a wrong -hash by a verified copy, and keeps yours in `retrobios-backup` beside the BIOS -folder ([putting it back](installer.md#files-you-already-had)). Each system page on the site also lists the expected -hashes, to compare against your file's MD5 or SHA1 by hand. +Run the same command without `--check`. It adds the missing files and replaces +the wrong ones. It keeps the old ones in `retrobios-backup`, beside your BIOS +folder. -### Wrong region BIOS +## 3. Check the folder and the names -Some cores require region-specific BIOS files. A Japanese BIOS won't boot -North American games on cores that enforce region matching. Check the emulator -profile for your core to see which regions are supported and which files -correspond to each. +- **Right folder?** Your emulator must read the folder the command used. + [Your setup's page](../which-pack.md) names it. +- **Is the name exact?** `scph5501.bin` and `SCPH-5501.BIN` are two different + names. On Linux, capital letters count too. +- **Unzipped by hand on Linux?** Use the command. Some emulators there need a + lower-case name the ZIP lacks. +- **A ZIP in parts will not open?** Download every part, like + `Pack.part1of2.zip` and `Pack.part2of2.zip`. Unzip each one into the same + folder. -## Checking what you installed +## One console only -`--check` reads the BIOS folder, hashes what is there and writes nothing: +Add `--system` and its name: -```bash -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --check +```sh +curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --check --system sony-playstation --verbose ``` -It ends on a count: +`--verbose` names each missing or wrong file. `--list-systems` shows every +console name. -``` -Checking existing files... - 5865/5865 present (5865 verified, 0 wrong hash) +## My setup says UNTESTED - All files up to date. -``` +Batocera shows `UNTESTED` for a file whose contents it does not expect. Run +the command without `--check` to replace it. -`0 wrong hash` is the line that matters. A number there means a file is present -whose contents the platform will reject, which is how a wrong revision or a bad -dump shows up. +## If the command fails { #installation-script-fails } -The installer's `--check` hashes every file against the list on every -platform, so it catches a wrong revision even where the frontend would not: -RetroArch, Lakka, RetroPie, MinUI, NextUI and MyMinUI only check that a file -exists, while Batocera, RetroBat, Recalbox, EmuDeck, RetroDECK, RomM, ROCKNIX, -KNULLI, AmberELEC, EmuELEC and MiSTer FPGA compare -MD5 and BizHawk compares SHA1. +- **Network error:** check that github.com opens, then try again. +- **Permission denied:** on Batocera, Recalbox and ROCKNIX, put `sudo` and a + space before the command. It then runs as the administrator. +- **No setup found:** name it yourself, with your BIOS folder: -For a single system, or the files a single emulator core reads (`--list-systems` -and `--list-cores` give the names): - -```bash -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --check --system sony-playstation -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --check --emulator beetle_psx -``` - -`--verbose` names every file still to fetch, wrong hashes included. -Details: [All installer options](installer.md). - -## BIOS not found by emulator - -The file exists on disk, but the emulator reports it as missing. - -### Wrong directory - -Each platform expects BIOS files in a specific base directory: - -- RetroArch: `system/` inside the RetroArch directory -- Lakka: `/storage/system/` -- Batocera: `/userdata/bios/` -- Recalbox: `/recalbox/share/bios/` -- RetroPie: `~/RetroPie/BIOS/` -- ROCKNIX: `/storage/roms/bios/` -- KNULLI: `/userdata/bios/` -- AmberELEC and EmuELEC: `/storage/roms/` (the pack holds `bios/` and the other folders their lists name) -- MiSTer FPGA: `/media/fat/games/` -- BizHawk: `Firmware/` inside the BizHawk directory -- EmuDeck: `~/Emulation/bios/` -- RetroDECK: `~/retrodeck/bios/` -- RetroBat: `bios/` inside the RetroBat directory -- MinUI, NextUI, MyMinUI: `Bios//` at the root of the SD card, one folder - per tag in parentheses of the Roms folders (`Bios/PS/` for - `Roms/Sony PlayStation (PS)/`) - -Some cores expect files in subdirectories (e.g. `dc/` for Dreamcast, `pcsx2/bios/` -for PlayStation 2). Check the `path:` field in the emulator profile for the exact -expected location relative to the base directory. - -### Wrong filename - -Cores match BIOS files by exact filename. If a core expects `scph5501.bin` and your -file is named `SCPH-5501.BIN`, it won't be found on platforms that do exact name matching. - -Check the emulator profile for the expected filename and any aliases listed under -`aliases:`. Aliases are alternative names that the core also accepts. - -### Case sensitivity - -Linux filesystems are case-sensitive. A file named `Bios.ROM` won't match a lookup -for `bios.rom`. Windows and macOS are case-insensitive by default, so the same -file works there but fails on Linux. - -Batocera's verification uses `casefold()` for case-insensitive matching, but -the actual emulator may still require exact case. When in doubt, use the exact -filename from the emulator profile. - -The RetroArch pack is built for every system RetroArch runs on, so it holds one -name per file: `BIOS.col`, never `bios.col` beside it, since extracting both on -Windows or macOS would overwrite one with the other. On Linux, Mesen-CE opens -`bios.col` and NP2kai opens the lowercase `np2kai/2608_*.wav`, which the pack -only holds as `BIOS.col` and `2608_*.WAV`. The installer checks whether the -BIOS directory keeps case and, if it does, also fetches these names. On Linux, use the installer -rather than extracting the ZIP by hand. - -The reverse happens on Batocera and Recalbox, whose lists hold both spellings -because their systems keep case: NP2kai reads `np2kai/bios.rom` and -`np2kai/BIOS.ROM` as two different files. Written to a folder that folds case -(the share opened from Windows, an exFAT card), one overwrites the other. The -installer checks the folder first, keeps one file per name there and lists -the ones it left out. - -## Installation script fails - -Problems with `install.py`, `install.sh`, or `download.sh`. - -### Network issues - -The installer fetches its manifest, then downloads only the repository or -`large-files` release objects it needs. If a download fails: - -- Check your internet connection -- Verify that `https://github.com` is reachable -- If behind a proxy, set `HTTPS_PROXY` in your environment -- Try again later if GitHub is experiencing issues - -### Permission denied - -The installer needs write access to the target directory. - -- On Linux/macOS: check directory ownership (`ls -la`) and run with appropriate - permissions. Avoid running as root unless the target directory requires it. -- On Windows: run PowerShell as Administrator if installing to a protected directory. -- A file system mounted read-only refuses root too, so the installer does not - suggest sudo for it. Linux remounts a card or drive read-only after an error - (an unclean removal, for one): check it (`fsck`, or `chkdsk` on Windows) and - mount it read-write again. - -A file already in the BIOS folder with the read-only attribute is replaced -like any other: on Windows the installer clears the attribute before putting -the verified copy in place. `file in place is read-only` only appears when -Windows refuses that too, which a file owned by another account can cause. - -### Platform not detected - -The installer finds your platform from the config files it knows. If it finds -none, name the platform and its BIOS folder. - -Linux, macOS, Android (Termux): - -```bash +```sh curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform retroarch --dest ~/.config/retroarch/system -curl -fsSL https://raw.githubusercontent.com/Abdess/retrobios/main/install.sh | sh -s -- --platform batocera --dest /userdata/bios ``` -Windows (PowerShell): - -```powershell -&([scriptblock]::Create((irm https://raw.githubusercontent.com/Abdess/retrobios/main/install.ps1))) ` - --platform retroarch --dest C:\RetroArch-Win64\system -``` - -`--list-platforms` lists the platforms the installer knows and the ones it -finds here, and `--help` lists the options with examples. - -### Manifest or file not found - -The one-line bootstrap verifies the downloaded installer before running it, and -the installer reads its manifests from the same revision. A platform name is -checked before anything is downloaded, so a "No file list" message comes from -the revision: `RETROBIOS_REF` naming a tag that does not exist, or one older -than the platform. Unset it, or pick a tag from the releases page. What a tag -holds and for how long is under -Details: [environment overrides](installer.md#environment-overrides). - -### Manual pack not found in release - -If the installer reports that no pack exists for your platform, check available -releases: - -```bash -python scripts/download.py --list -# or -bash scripts/download.sh --list -``` - -A pack's name lists every platform it serves. Releases up to v2026.10.10 -ship RetroArch and Lakka as one `RetroArch_Lakka` pack, and the next ones give -Lakka its own, since Lakka runs on Linux where two names that differ by case -are two files. The installer and the download scripts find the right pack -either way. - -### A pack in several parts will not extract - -Parts named `Pack.part1of2.zip` are ordinary ZIPs: extract each one into the -same folder, and check that all of them were downloaded, the name says how -many there are. - -`scripts/download.sh` and `scripts/download.py` download every part and check -it against the release's checksums before extracting. Both stage inside the -destination directory, never in `/tmp`, which is a RAM disk on most of these -systems. -## Checks of the collection itself +Other messages: Details: [When something goes wrong](installer.md#when-something-goes-wrong). -`scripts/verify.py` checks this project's own collection against each -platform, from a clone of the repository. It never reads your BIOS folder. -Reference: [Reading verify.py output](verification-modes.md#hash-mismatch-untested). - !!! tip "Next" - - The page of your setup, from [Get your files](../which-pack.md): its folder - and how to check it - - [FAQ](faq.md), for the questions people ask before installing + - [Your setup's page](../which-pack.md): its folder and how it checks files + - [FAQ](faq.md) diff --git a/wiki/what-are-bios-files.md b/wiki/what-are-bios-files.md index 465df5df..837751dc 100644 --- a/wiki/what-are-bios-files.md +++ b/wiki/what-are-bios-files.md @@ -2,75 +2,62 @@ level: start --- -# What BIOS files are +# What is a BIOS? -Many consoles have a small program built into them. It runs when the console -is switched on, sets up the hardware and then starts the game. That program is -called the BIOS, or the firmware. +A console has a small program inside it. It starts first, then starts the game. +This program is the **BIOS**, also called **firmware**. -An emulator recreates a console in software. To start the games of many -consoles, it needs a copy of that built-in program. The copy is a file, and -that file is what people call a BIOS file. +[Get your files](../which-pack.md){ .md-button .md-button--primary } + +An **emulator** is an app that acts like a console. For many consoles, it needs +a copy of the BIOS. That copy is a file: a **BIOS file**. -## How the pieces fit together +## How it works -1. You pick a game in the **frontend**, the menu you play from (EmulationStation, - the RetroArch menu, MinUI and others). -2. The frontend starts an **emulator** for that console. -3. The emulator reads the **BIOS file** from the **BIOS folder** of your setup. +1. You pick a game in your game menu. +2. The menu starts the emulator. +3. The emulator opens the BIOS file in your **BIOS folder**. 4. The game starts. -When the file is not in the folder, or is not the version the emulator expects, -step 3 fails. The emulator then says that a BIOS is missing, or the game stays -on a black screen. +If step 3 fails, you see "BIOS missing" or a black screen. -## Do I need them? +## Do I need one? -It depends on the console. +- **Yes:** PlayStation 1 and 2, Saturn, Dreamcast, Neo Geo, PC Engine CD and + many home computers. +- **Usually not:** NES, Super Nintendo, Mega Drive, Game Boy and Game Boy + Advance. +- **Add-ons need their own:** the Famicom Disk System and the Mega-CD. -- **Usually yes** for consoles that start every game through their own - program: PlayStation, PlayStation 2, Saturn, Mega-CD, PC Engine CD, Neo Geo, - Dreamcast, and many home computers. -- **Usually not** for cartridge consoles such as the NES, the Super Nintendo, - the Mega Drive, the Game Boy and the Game Boy Advance. Their emulators start - the games without one. Some can use one to show the original start-up screen. -- **Add-ons need their own**, even on those consoles: the Famicom Disk System - and the Mega-CD each have a BIOS. +Details: [Every console, one by one](../systems/index.md) -Some emulators can stand in for the BIOS with code of their own. This is called -HLE, for high-level emulation. It works for many games, but some glitch or do -not start. The real file gives the best results, so the packs include it. +## The name must match -Details: [the files each console needs](../systems/index.md), with the -emulators that read them. +An emulator looks for one exact name. A PlayStation emulator looks for +`scph5501.bin`. A file with another name is not found. -## Where they go +## The version must match -Each setup keeps its BIOS files in a folder of its own: `system` for RetroArch, -`/userdata/bios` on Batocera, `Emulation/bios` with EmuDeck. The page of your -setup gives the exact folder. +Two files can have the same name and different contents. One may be for Japan, +another for Europe. -The name matters. An emulator looks for an exact name, such as `scph5501.bin` -for a PlayStation, and a file with another name is not found. +Some setups check each file's **hash**, a fingerprint of its contents. +Batocera, Recalbox and EmuDeck report a version they do not expect. -## Why the right version matters +Our files are the versions each setup expects. -Two files with the same name can hold different versions: one per region, or -a newer revision of the same chip. Some setups, among them Batocera, Recalbox -and EmuDeck, compare a fingerprint of the file's contents, called a hash, and -refuse a version they do not list. RetroBIOS gives each setup the versions it -expects. +## Can a game start without it? + +Sometimes. Some emulators carry their own code in place of the BIOS. Some games +then glitch or do not start. The real file works best, so we include it. ## Is it legal? -Laws on copying console firmware differ by country, and the question is not -settled. The [project's position](faq.md#is-this-legal) explains where it -stands, where its arguments are weakest, and how to ask for a file to be -removed. +Laws differ by country. [Read the project's position](faq.md#is-this-legal). !!! tip "Next" - - [Get your files](../which-pack.md), for the setup you play on - - [Glossary](glossary.md), every word these pages use + - [Get your files](../which-pack.md) + - [Words used on this site](glossary.md)