mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-10-23 20:29:06 -05:00
feat: guard split packs and install offline
This commit is contained in:
Abdessamad Derraz committed
commit
651813f10a
18 files changed
+1694
-63
No files matched your search
@@ -48,22 +48,22 @@ The Files column counts what a pack holds once extracted, the figure a file mana
|
||||
| Platform | Files | Extracted size | Extract to | Download |
|
||||
|----------|------:|---------------:|-----------|----------|
|
||||
| AmberELEC | - | - | `/storage/roms/` | Not in v2026.10.10: [use the installer](https://abdess.github.io/retrobios/wiki/installer/) |
|
||||
| Batocera | 12,810 | 7.5 GB | `/userdata/bios/` | Download: [part 1 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part3of3.zip) |
|
||||
| Batocera | 12,810 | 7.5 GB | `/userdata/bios/` | Download, 3 files, all needed: [part 1 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/Batocera_43.1_BIOS_Pack.part3of3.zip) |
|
||||
| BizHawk | 950 | 2.4 GB | `Firmware/` | [Download](../../releases/download/v2026.10.10/BizHawk_2.11.1_BIOS_Pack.zip) |
|
||||
| EmuDeck | 620 | 3.2 GB | `~/Emulation/bios/` | [Download](../../releases/download/v2026.10.10/EmuDeck_2.7EA_BIOS_Pack.zip) |
|
||||
| EmuELEC | - | - | `/storage/roms/` | Not in v2026.10.10: [use the installer](https://abdess.github.io/retrobios/wiki/installer/) |
|
||||
| KNULLI | - | - | `/userdata/bios/` | Not in v2026.10.10: [use the installer](https://abdess.github.io/retrobios/wiki/installer/) |
|
||||
| Lakka | 8,504 | 6.0 GB | `/storage/system/` | Download: [part 1 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part2of2.zip) |
|
||||
| Lakka | 8,504 | 6.0 GB | `/storage/system/` | Download, 2 files, all needed: [part 1 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part2of2.zip) |
|
||||
| MiSTer FPGA | 83 | 29 MB | `/media/fat/games/` | [Download](../../releases/download/v2026.10.10/MiSTer_FPGA_2026-09-29_BIOS_Pack.zip) |
|
||||
| MinUI | - | - | `<SD card>/` | Not in v2026.10.10: [use the installer](https://abdess.github.io/retrobios/wiki/installer/) |
|
||||
| MyMinUI | - | - | `<SD card>/` | Not in v2026.10.10: [use the installer](https://abdess.github.io/retrobios/wiki/installer/) |
|
||||
| NextUI | - | - | `<SD card>/` | Not in v2026.10.10: [use the installer](https://abdess.github.io/retrobios/wiki/installer/) |
|
||||
| ROCKNIX | 5,804 | 6.1 GB | `/storage/roms/bios/` | Download: [part 1 of 2](../../releases/download/v2026.10.10/ROCKNIX_20261001_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/ROCKNIX_20261001_BIOS_Pack.part2of2.zip) |
|
||||
| Recalbox | 3,222 | 3.9 GB | `/recalbox/share/bios/` | Download: [part 1 of 2](../../releases/download/v2026.10.10/Recalbox_10.1.1_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/Recalbox_10.1.1_BIOS_Pack.part2of2.zip) |
|
||||
| RetroArch | 8,504 | 6.0 GB | `system/` | Download: [part 1 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part2of2.zip) |
|
||||
| RetroBat | 12,399 | 7.9 GB | `bios/` | Download: [part 1 of 3](../../releases/download/v2026.10.10/RetroBat_8.2.1_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/RetroBat_8.2.1_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/RetroBat_8.2.1_BIOS_Pack.part3of3.zip) |
|
||||
| RetroDECK | 7,793 | 8.0 GB | `~/retrodeck/` | Download: [part 1 of 3](../../releases/download/v2026.10.10/RetroDECK_0.10.10b_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/RetroDECK_0.10.10b_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/RetroDECK_0.10.10b_BIOS_Pack.part3of3.zip) |
|
||||
| RetroPie * | 12,533 | 8.1 GB | `~/RetroPie/BIOS/` | Download: [part 1 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part3of3.zip) |
|
||||
| ROCKNIX | 5,804 | 6.1 GB | `/storage/roms/bios/` | Download, 2 files, all needed: [part 1 of 2](../../releases/download/v2026.10.10/ROCKNIX_20261001_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/ROCKNIX_20261001_BIOS_Pack.part2of2.zip) |
|
||||
| Recalbox | 3,222 | 3.9 GB | `/recalbox/share/bios/` | Download, 2 files, all needed: [part 1 of 2](../../releases/download/v2026.10.10/Recalbox_10.1.1_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/Recalbox_10.1.1_BIOS_Pack.part2of2.zip) |
|
||||
| RetroArch | 8,504 | 6.0 GB | `system/` | Download, 2 files, all needed: [part 1 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part1of2.zip), [part 2 of 2](../../releases/download/v2026.10.10/RetroArch_Lakka_v1.22.2_BIOS_Pack.part2of2.zip) |
|
||||
| RetroBat | 12,399 | 7.9 GB | `bios/` | Download, 3 files, all needed: [part 1 of 3](../../releases/download/v2026.10.10/RetroBat_8.2.1_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/RetroBat_8.2.1_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/RetroBat_8.2.1_BIOS_Pack.part3of3.zip) |
|
||||
| RetroDECK | 7,793 | 8.0 GB | `~/retrodeck/` | Download, 3 files, all needed: [part 1 of 3](../../releases/download/v2026.10.10/RetroDECK_0.10.10b_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/RetroDECK_0.10.10b_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/RetroDECK_0.10.10b_BIOS_Pack.part3of3.zip) |
|
||||
| RetroPie * | 12,533 | 8.1 GB | `~/RetroPie/BIOS/` | Download, 3 files, all needed: [part 1 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part1of3.zip), [part 2 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part2of3.zip), [part 3 of 3](../../releases/download/v2026.10.10/RetroPie_4.8.12_BIOS_Pack.part3of3.zip) |
|
||||
| RomM | 1,087 | 1.5 GB | `library/bios/` | [Download](../../releases/download/v2026.10.10/RomM_5.3.1_BIOS_Pack.zip) |
|
||||
|
||||
The AmberELEC pack holds its own `bios/` folder. Unzip it into `/storage/roms/`.
|
||||
@@ -209,4 +209,4 @@ The scripts and tooling are released under the [MIT License](LICENSE).
|
||||
The BIOS and firmware files are not covered by that license: they are third-party system software, preserved and provided for personal backup, archival, and interoperability with emulation software. [NOTICE](NOTICE) sets out their status and how to ask for a file to be removed.
|
||||
The reasoning, and where it is weakest, is in [the project's legal position](https://abdess.github.io/retrobios/wiki/legal/).
|
||||
|
||||
*Auto-generated on 2026-10-22T20:58:47Z*
|
||||
*Auto-generated on 2026-10-23T05:39:12Z*
|
||||
+1
-1
@@ -11,7 +11,7 @@
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$defaultInstallUrl = "https://raw.githubusercontent.com/Abdess/retrobios/main/install.py"
|
||||
$defaultInstallSha256 = "ad3b338a8f709be45352a12bd9c6a6250819a8fb9afd19498e24f41e3f062d20"
|
||||
$defaultInstallSha256 = "568d9399c5639db607cfc3944fa46a0dfa8d47741291d9346ed336cd2500eb00"
|
||||
$maximumInstallerBytes = 2MB
|
||||
$installerExit = 0
|
||||
$installer = if ($PSScriptRoot) { Join-Path $PSScriptRoot "install.py" } else { $null }
|
||||
|
||||
+863
-18
File diff suppressed because it is too large.
Load diff
+1
-1
@@ -18,7 +18,7 @@ esac
|
||||
TEMP_INSTALLER=""
|
||||
TEMP_DIRECTORY=""
|
||||
DEFAULT_INSTALL_URL="https://raw.githubusercontent.com/Abdess/retrobios/main/install.py"
|
||||
DEFAULT_INSTALL_SHA256="ad3b338a8f709be45352a12bd9c6a6250819a8fb9afd19498e24f41e3f062d20"
|
||||
DEFAULT_INSTALL_SHA256="568d9399c5639db607cfc3944fa46a0dfa8d47741291d9346ed336cd2500eb00"
|
||||
MAX_INSTALLER_BYTES=2097152
|
||||
|
||||
cleanup() {
|
||||
|
||||
@@ -236,7 +236,9 @@ def download_links(
|
||||
"""Markdown links to the files of a platform's pack in the recorded release.
|
||||
|
||||
A pack published whole gets one link labelled *label*. A pack in parts
|
||||
gets one link per part, since every part is needed. *repo* is the
|
||||
gets one link per part, since every part is needed, after the number of
|
||||
files to download: a reader who took one link for the pack extracted
|
||||
half of it. *repo* is the
|
||||
repository URL, absolute or relative, and *attrs* an attribute list
|
||||
put after each link, or *button* the function that draws each link as a
|
||||
button on the site. Without a recorded release the link is the latest
|
||||
@@ -263,8 +265,8 @@ def download_links(
|
||||
for number, url in enumerate(urls, 1)
|
||||
]
|
||||
if styled:
|
||||
return " ".join(parts)
|
||||
return f"{label}: " + ", ".join(parts)
|
||||
return f"{len(urls)} files to download, all needed: " + " ".join(parts)
|
||||
return f"{label}, {len(urls)} files, all needed: " + ", ".join(parts)
|
||||
|
||||
|
||||
def extract_targets(platforms_dir: str) -> list[tuple[str, str]]:
|
||||
|
||||
@@ -63,7 +63,8 @@ def _links(tag: str, assets: list[str]) -> str:
|
||||
base = f"https://github.com/{REPO}/releases/download/{tag}"
|
||||
if len(assets) == 1:
|
||||
return f"[{assets[0]}]({base}/{assets[0]})"
|
||||
return " + ".join(f"[part {n}]({base}/{asset})" for n, asset in enumerate(assets, 1))
|
||||
parts = " + ".join(f"[part {n}]({base}/{asset})" for n, asset in enumerate(assets, 1))
|
||||
return f"{len(assets)} files, all needed: {parts}"
|
||||
|
||||
|
||||
def render(record: dict, platforms_dir: str, whats_new: str) -> str:
|
||||
|
||||
@@ -103,8 +103,11 @@ def _read_record(dist: Path, tag: str, owners: dict[str, list[str]]) -> dict:
|
||||
members: dict[str, int] = {}
|
||||
for path in paths:
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
# A part's marker names its siblings and is not pack content:
|
||||
# the manifest's pack_files is counted before any split.
|
||||
for info in archive.infolist():
|
||||
members[info.filename] = info.file_size
|
||||
if not split_pack.is_marker(info.filename):
|
||||
members[info.filename] = info.file_size
|
||||
packs[name] = {
|
||||
"files": len(members),
|
||||
"extracted_size": sum(members.values()),
|
||||
|
||||
+123
-16
@@ -6,6 +6,11 @@ as parts, each a complete archive of whole files: any tool opens one, and
|
||||
the parts extracted into the same folder are the pack. Members are copied
|
||||
as stored, never recompressed, so the parts depend on the pack alone.
|
||||
|
||||
Each part also carries a short text file, its marker, at the archive root:
|
||||
the pack it belongs to, its number, and the name and size of every part.
|
||||
Extracted with the files, the markers let `install.py --check` name a part
|
||||
that is missing from the folder.
|
||||
|
||||
Usage:
|
||||
python scripts/split_pack.py dist/
|
||||
python scripts/split_pack.py dist/RetroArch_v1.22.2_BIOS_Pack.zip
|
||||
@@ -20,6 +25,7 @@ import re
|
||||
import struct
|
||||
import sys
|
||||
import zipfile
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
from artifacts import ArtifactLockBusy, artifact_lock
|
||||
@@ -29,6 +35,10 @@ ASSET_LIMIT = 2 * 1024**3
|
||||
|
||||
PACK_SUFFIX = "_BIOS_Pack.zip"
|
||||
_PART_NAME = re.compile(r"^(?P<stem>.+_BIOS_Pack)\.part(?P<index>\d+)of(?P<count>\d+)\.zip$")
|
||||
_MARKER_NAME = re.compile(r"^(?P<stem>.+_BIOS_Pack)\.part(?P<index>\d+)of(?P<count>\d+)\.txt$")
|
||||
# Sizes are written at a fixed width, so a marker's length, and with it the
|
||||
# size of its own part, does not depend on the sizes it records.
|
||||
_SIZE_WIDTH = 13
|
||||
|
||||
_LOCAL = struct.Struct("<4s2B4HL2L2H")
|
||||
_CENTRAL = struct.Struct("<4s4B4HL2L5H2L")
|
||||
@@ -58,6 +68,44 @@ def part_name(pack_name: str, index: int, count: int) -> str:
|
||||
return f"{pack_name[: -len('.zip')]}.part{index}of{count}.zip"
|
||||
|
||||
|
||||
def marker_name(pack_name: str, index: int, count: int) -> str:
|
||||
return f"{pack_name[: -len('.zip')]}.part{index}of{count}.txt"
|
||||
|
||||
|
||||
def is_marker(name: str) -> bool:
|
||||
"""Whether an archive member is the marker a split writes in each part."""
|
||||
return _MARKER_NAME.match(name) is not None
|
||||
|
||||
|
||||
def marker_text(pack_name: str, index: int, sizes: list[int]) -> bytes:
|
||||
"""The marker of part *index* of a pack published in ``len(sizes)`` parts.
|
||||
|
||||
Plain text a person can read, in a form install.py parses back: the
|
||||
lines that matter are `pack:`, `part:` and one `file:` per part.
|
||||
"""
|
||||
count = len(sizes)
|
||||
lines = [
|
||||
"RetroBIOS split pack",
|
||||
f"This folder holds part {index} of {count} of {pack_name}.",
|
||||
"Every part is needed: extract them all into this same folder.",
|
||||
"The installer's --check names a part missing from the folder.",
|
||||
"",
|
||||
f"pack: {pack_name}",
|
||||
f"part: {index} of {count}",
|
||||
]
|
||||
lines += [
|
||||
f"file: {size:>{_SIZE_WIDTH}} {part_name(pack_name, number, count)}"
|
||||
for number, size in enumerate(sizes, 1)
|
||||
]
|
||||
return ("\n".join(lines) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def _marker_weight(pack_name: str, index: int, count: int) -> int:
|
||||
name = len(marker_name(pack_name, index, count).encode("utf-8"))
|
||||
body = len(marker_text(pack_name, index, [0] * count))
|
||||
return _LOCAL.size + name + body + _CENTRAL.size + name
|
||||
|
||||
|
||||
def _name_bytes(info: zipfile.ZipInfo) -> bytes:
|
||||
encoding = "utf-8" if info.flag_bits & _UTF8_NAME else "cp437"
|
||||
return info.orig_filename.encode(encoding)
|
||||
@@ -70,23 +118,24 @@ def _weight(info: zipfile.ZipInfo) -> int:
|
||||
|
||||
|
||||
def plan_parts(
|
||||
members: list[zipfile.ZipInfo], limit: int
|
||||
members: list[zipfile.ZipInfo], limit: int, reserve: int = 0
|
||||
) -> list[list[zipfile.ZipInfo]]:
|
||||
"""Fill each part in archive order, up to the limit."""
|
||||
"""Fill each part in archive order, up to the limit less *reserve*."""
|
||||
parts: list[list[zipfile.ZipInfo]] = [[]]
|
||||
size = _END.size
|
||||
floor = _END.size + reserve
|
||||
size = floor
|
||||
for info in members:
|
||||
weight = _weight(info)
|
||||
if _END.size + weight >= limit:
|
||||
if floor + weight >= limit:
|
||||
raise ValueError(
|
||||
f"{info.filename} needs {weight:,} bytes, more than a part of "
|
||||
f"{limit:,} can hold"
|
||||
)
|
||||
if info.compress_size > _MAX_FIELD or info.file_size > _MAX_FIELD:
|
||||
raise ValueError(f"{info.filename} needs ZIP64, which a part does not use")
|
||||
if size + weight >= limit or len(parts[-1]) >= _MAX_ENTRIES:
|
||||
if size + weight >= limit or len(parts[-1]) >= _MAX_ENTRIES - 1:
|
||||
parts.append([])
|
||||
size = _END.size
|
||||
size = floor
|
||||
parts[-1].append(info)
|
||||
size += weight
|
||||
return parts
|
||||
@@ -109,12 +158,24 @@ def _data_offset(source: io.BufferedReader, info: zipfile.ZipInfo) -> int:
|
||||
return info.header_offset + _LOCAL.size + header[10] + header[11]
|
||||
|
||||
|
||||
def _marker_info(name: str, body: bytes) -> zipfile.ZipInfo:
|
||||
info = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0))
|
||||
info.compress_type = zipfile.ZIP_STORED
|
||||
info.CRC = zlib.crc32(body)
|
||||
info.compress_size = info.file_size = len(body)
|
||||
info.create_system = 3
|
||||
info.external_attr = (0o100000 | 0o644) << 16
|
||||
return info
|
||||
|
||||
|
||||
def _write_part(
|
||||
source: io.BufferedReader, members: list[zipfile.ZipInfo], dest: Path
|
||||
source: io.BufferedReader, members: list[zipfile.ZipInfo], dest: Path,
|
||||
marker: tuple[str, bytes],
|
||||
) -> None:
|
||||
directory = []
|
||||
marker_info = _marker_info(*marker)
|
||||
with open(dest, "wb") as out:
|
||||
for info in members:
|
||||
for info in [marker_info, *members]:
|
||||
name = _name_bytes(info)
|
||||
flags = info.flag_bits & ~_DATA_DESCRIPTOR
|
||||
dos_time, dos_date = _dos_stamp(info)
|
||||
@@ -127,8 +188,12 @@ def _write_part(
|
||||
)
|
||||
)
|
||||
out.write(name)
|
||||
source.seek(_data_offset(source, info))
|
||||
remaining = info.compress_size
|
||||
if info is marker_info:
|
||||
out.write(marker[1])
|
||||
remaining = 0
|
||||
else:
|
||||
source.seek(_data_offset(source, info))
|
||||
remaining = info.compress_size
|
||||
while remaining:
|
||||
chunk = source.read(min(_CHUNK, remaining))
|
||||
if not chunk:
|
||||
@@ -157,7 +222,33 @@ def _write_part(
|
||||
|
||||
|
||||
def _identity(archive: zipfile.ZipFile) -> list[tuple[str, int, int]]:
|
||||
return [(info.filename, info.CRC, info.file_size) for info in archive.infolist()]
|
||||
return [
|
||||
(info.filename, info.CRC, info.file_size)
|
||||
for info in archive.infolist()
|
||||
if not is_marker(info.filename)
|
||||
]
|
||||
|
||||
|
||||
def marker_weight(pack_name: str, count: int) -> int:
|
||||
"""Bytes the largest marker of a pack in *count* parts adds to its part."""
|
||||
return max(_marker_weight(pack_name, index, count) for index in (1, count))
|
||||
|
||||
|
||||
def _plan_with_markers(
|
||||
pack_name: str, members: list[zipfile.ZipInfo], limit: int
|
||||
) -> list[list[zipfile.ZipInfo]]:
|
||||
"""Plan the parts with room for the marker each one carries.
|
||||
|
||||
The marker lists every part, so its length grows with their count: the
|
||||
plan is made again until the count it assumed is the count it gives.
|
||||
"""
|
||||
count = 1
|
||||
while True:
|
||||
reserve = marker_weight(pack_name, count)
|
||||
plan = plan_parts(members, limit, reserve)
|
||||
if len(plan) <= count:
|
||||
return plan
|
||||
count = len(plan)
|
||||
|
||||
|
||||
def split_pack(zip_path: Path, limit: int = ASSET_LIMIT) -> list[Path]:
|
||||
@@ -174,15 +265,31 @@ def split_pack(zip_path: Path, limit: int = ASSET_LIMIT) -> list[Path]:
|
||||
with zipfile.ZipFile(zip_path) as archive:
|
||||
members = archive.infolist()
|
||||
expected = _identity(archive)
|
||||
plan = plan_parts(members, limit)
|
||||
taken = [info.filename for info in members if is_marker(info.filename)]
|
||||
if taken:
|
||||
raise ValueError(f"{zip_path.name} already holds {taken[0]}, a part marker name")
|
||||
plan = _plan_with_markers(zip_path.name, members, limit)
|
||||
count = len(plan)
|
||||
parts = [
|
||||
zip_path.with_name(part_name(zip_path.name, index, len(plan)))
|
||||
for index in range(1, len(plan) + 1)
|
||||
zip_path.with_name(part_name(zip_path.name, index, count))
|
||||
for index in range(1, count + 1)
|
||||
]
|
||||
sizes = [
|
||||
_END.size + _marker_weight(zip_path.name, index, count)
|
||||
+ sum(_weight(info) for info in chosen)
|
||||
for index, chosen in enumerate(plan, 1)
|
||||
]
|
||||
try:
|
||||
with open(zip_path, "rb") as source:
|
||||
for dest, chosen in zip(parts, plan):
|
||||
_write_part(source, chosen, dest)
|
||||
for index, (dest, chosen) in enumerate(zip(parts, plan), 1):
|
||||
marker = (marker_name(zip_path.name, index, count),
|
||||
marker_text(zip_path.name, index, sizes))
|
||||
_write_part(source, chosen, dest, marker)
|
||||
if dest.stat().st_size != sizes[index - 1]:
|
||||
raise ValueError(
|
||||
f"{dest.name} is {dest.stat().st_size:,} bytes, its marker "
|
||||
f"says {sizes[index - 1]:,}"
|
||||
)
|
||||
found: list[tuple[str, int, int]] = []
|
||||
for dest in parts:
|
||||
with zipfile.ZipFile(dest) as archive:
|
||||
|
||||
@@ -111,6 +111,15 @@ class DownloadLinksNameTheFile(unittest.TestCase):
|
||||
)
|
||||
self.assertIn("part 1 of 2", cell)
|
||||
|
||||
def test_a_split_pack_says_how_many_files_to_download(self):
|
||||
"""One link taken for the pack left half of it out of the folder."""
|
||||
self.assertIn(
|
||||
"Download, 2 files, all needed: ",
|
||||
download_links("batocera", SPLIT_RECORD, "../..", "Download"),
|
||||
)
|
||||
buttons = download_links("batocera", SPLIT_RECORD, REPO, "Pack", " { .x }")
|
||||
self.assertTrue(buttons.startswith("2 files to download, all needed: "), buttons)
|
||||
|
||||
def test_buttons_carry_their_attributes_on_each_part(self):
|
||||
cell = download_links("batocera", SPLIT_RECORD, REPO, "Pack", " { .x }")
|
||||
self.assertEqual(cell.count("{ .x }"), 2)
|
||||
|
||||
+10
-1
@@ -446,8 +446,11 @@ class DownloaderCase(unittest.TestCase):
|
||||
"pack staged in the system temp directory",
|
||||
)
|
||||
searched = list(self.dest.parent.iterdir()) + list(self.dest.rglob("*"))
|
||||
# A part's marker is extracted on purpose: install.py --check reads it.
|
||||
leftovers = sorted(
|
||||
p.name for p in searched if "_BIOS_Pack" in p.name or p.name.startswith(".")
|
||||
p.name for p in searched
|
||||
if ("_BIOS_Pack" in p.name or p.name.startswith("."))
|
||||
and not split_pack.is_marker(p.name)
|
||||
)
|
||||
self.assertEqual(leftovers, [], "download staging left behind")
|
||||
|
||||
@@ -573,6 +576,12 @@ class ZipPartsCase(DownloaderCase):
|
||||
"""A pack published as parts that are each an archive."""
|
||||
|
||||
zip_parts = True
|
||||
# Bytes deflate cannot shrink: each member outweighs the marker a part
|
||||
# carries, so the pack splits in two.
|
||||
payload = {
|
||||
"bios/scph5501.bin": b"".join(hashlib.sha256(b"us%d" % n).digest() for n in range(64)),
|
||||
"bios/dc/dc_boot.bin": b"".join(hashlib.sha256(b"dc%d" % n).digest() for n in range(64)),
|
||||
}
|
||||
|
||||
|
||||
class ZipPartsTests:
|
||||
|
||||
@@ -0,0 +1,502 @@
|
||||
"""What --check says about a refused file, and an install with no network.
|
||||
|
||||
A user with the BIOS one folder too deep, still in its ZIP or under another
|
||||
name read "missing: scph5501.bin" and nothing else. --check now names the
|
||||
reason and where each core opens the file. A pack extracted from some of
|
||||
its parts is named by the part it lacks. And a pack downloaded once
|
||||
installs again from disk, --from-pack, every file checked by its hash.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||
sys.path.insert(0, str(REPO_ROOT / "tests"))
|
||||
|
||||
import install # noqa: E402
|
||||
import split_pack # noqa: E402
|
||||
from test_install_selection import FakeRepository # noqa: E402
|
||||
|
||||
US = b"us bios " * 64
|
||||
JP = b"jp bios " * 64
|
||||
EU = b"eu bios " * 64
|
||||
GBA = b"gba bios" * 32
|
||||
EXTRA = b"pcsx extra"
|
||||
|
||||
|
||||
def _sha1(data: bytes) -> str:
|
||||
return hashlib.sha1(data).hexdigest()
|
||||
|
||||
|
||||
def _entry(dest: str, data: bytes, systems: list[str], **fields) -> dict:
|
||||
return {
|
||||
"dest": dest, "sha1": _sha1(data), "size": len(data),
|
||||
"repo_path": f"bios/{dest.replace('/', '_')}", "cores": None,
|
||||
"systems": systems, **fields,
|
||||
}
|
||||
|
||||
|
||||
def manifest() -> dict:
|
||||
"""A small RetroArch list: two regions of one BIOS, a core file with a
|
||||
second revision, a file in a subfolder."""
|
||||
return {
|
||||
"manifest_version": 2,
|
||||
"platform": "retroarch",
|
||||
"files": [
|
||||
_entry("scph5501.bin", US, ["sony-playstation"], regions=["north-america"]),
|
||||
_entry("scph5500.bin", JP, ["sony-playstation"], regions=["japan"]),
|
||||
_entry("gba_bios.bin", GBA, ["nintendo-gba"]),
|
||||
_entry("pcsx/extra.bin", EXTRA, ["sony-playstation"], cores=["pcsx_rearmed"],
|
||||
alternatives=[{"sha1": _sha1(EU), "size": len(EU),
|
||||
"repo_path": "bios/eu", "cores": ["beetle_psx"]},
|
||||
{"sha1": _sha1(EXTRA), "size": len(EXTRA),
|
||||
"repo_path": "bios/x", "cores": ["pcsx_rearmed"]}]),
|
||||
],
|
||||
"system_cores": {"sony-playstation": ["beetle_psx", "pcsx_rearmed"],
|
||||
"nintendo-gba": ["mgba"]},
|
||||
}
|
||||
|
||||
|
||||
PAYLOAD = {"scph5501.bin": US, "scph5500.bin": JP, "gba_bios.bin": GBA,
|
||||
"pcsx/extra.bin": EXTRA}
|
||||
|
||||
|
||||
def _closed_port() -> int:
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
return sock.getsockname()[1]
|
||||
|
||||
|
||||
def _offline_env() -> dict:
|
||||
"""An environment where any request the installer makes is refused."""
|
||||
env = dict(os.environ)
|
||||
env["RETROBIOS_BASE_URL"] = f"http://127.0.0.1:{_closed_port()}"
|
||||
env["HTTPS_PROXY"] = env["HTTP_PROXY"] = f"http://127.0.0.1:{_closed_port()}"
|
||||
return env
|
||||
|
||||
|
||||
class Scratch(unittest.TestCase):
|
||||
def setUp(self):
|
||||
scratch = REPO_ROOT / "tmp"
|
||||
scratch.mkdir(exist_ok=True)
|
||||
self.root = Path(tempfile.mkdtemp(dir=scratch))
|
||||
self.addCleanup(shutil.rmtree, self.root, True)
|
||||
self.dest = self.root / "system"
|
||||
self.list = self.root / "retroarch.json"
|
||||
self.list.write_text(json.dumps(manifest()))
|
||||
|
||||
def run_installer(self, *options: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(REPO_ROOT / "install.py"), "--platform", "retroarch",
|
||||
"--dest", str(self.dest), *options],
|
||||
env=_offline_env(), capture_output=True, text=True, timeout=120,
|
||||
check=False, stdin=subprocess.DEVNULL,
|
||||
)
|
||||
|
||||
def place(self, relative: str, data: bytes) -> Path:
|
||||
path = self.dest / relative
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
return path
|
||||
|
||||
def check(self) -> str:
|
||||
proc = self.run_installer("--check", "--manifest", str(self.list), "--verbose")
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr)
|
||||
return proc.stdout
|
||||
|
||||
def complete(self, *skip: str) -> None:
|
||||
for name, data in PAYLOAD.items():
|
||||
if name not in skip:
|
||||
self.place(name, data)
|
||||
|
||||
|
||||
class CheckNamesWhyAFileIsRefused(Scratch):
|
||||
def test_another_region_is_named(self):
|
||||
self.complete("scph5501.bin")
|
||||
self.place("scph5501.bin", JP)
|
||||
out = self.check()
|
||||
self.assertIn("wrong hash: scph5501.bin", out)
|
||||
self.assertIn("another region: the contents of scph5500.bin (japan)", out)
|
||||
|
||||
def test_another_revision_names_the_core_that_reads_it(self):
|
||||
self.complete("pcsx/extra.bin")
|
||||
self.place("pcsx/extra.bin", EU)
|
||||
out = self.check()
|
||||
self.assertIn("another revision: the copy beetle_psx reads", out)
|
||||
|
||||
def test_a_wrong_size_is_measured(self):
|
||||
self.complete("gba_bios.bin")
|
||||
self.place("gba_bios.bin", b"short")
|
||||
self.assertIn(f"wrong size: 5 bytes, the list expects {len(GBA):,}", self.check())
|
||||
|
||||
def test_a_zip_under_the_file_name_is_still_zipped(self):
|
||||
self.complete("gba_bios.bin")
|
||||
buffer = io.BytesIO()
|
||||
with zipfile.ZipFile(buffer, "w") as archive:
|
||||
archive.writestr("gba_bios.bin", GBA)
|
||||
self.place("gba_bios.bin", buffer.getvalue())
|
||||
self.assertIn("still zipped: this file is a ZIP archive", self.check())
|
||||
|
||||
def test_a_file_inside_an_archive_is_still_zipped(self):
|
||||
self.complete("gba_bios.bin")
|
||||
self.dest.mkdir(exist_ok=True)
|
||||
with zipfile.ZipFile(self.dest / "GBA BIOS.zip", "w") as archive:
|
||||
archive.writestr("bios/gba_bios.bin", GBA)
|
||||
self.assertIn("still zipped: it is inside GBA BIOS.zip", self.check())
|
||||
|
||||
def test_a_case_difference_is_named(self):
|
||||
probe = self.root / "CaseProbe"
|
||||
probe.write_text("x")
|
||||
if (self.root / "caseprobe").exists():
|
||||
self.skipTest("this filesystem does not keep case")
|
||||
self.complete("gba_bios.bin")
|
||||
self.place("GBA_BIOS.BIN", GBA)
|
||||
self.assertIn("case of the name: the folder holds GBA_BIOS.BIN", self.check())
|
||||
|
||||
def test_a_nested_bios_folder_is_named(self):
|
||||
for name, data in PAYLOAD.items():
|
||||
self.place(f"system/{name}", data)
|
||||
out = self.check()
|
||||
self.assertIn("nested folder: it is at system/system/scph5501.bin", out)
|
||||
self.assertIn(f"4 files sit too deep, in {self.dest.resolve() / 'system'}", out)
|
||||
|
||||
def test_a_wrong_folder_is_named(self):
|
||||
self.complete("pcsx/extra.bin")
|
||||
self.place("beetle/extra.bin", EXTRA)
|
||||
self.assertIn("wrong folder: it is at beetle/extra.bin", self.check())
|
||||
|
||||
def test_a_folder_too_deep_is_named(self):
|
||||
self.complete("gba_bios.bin")
|
||||
self.place("gba/roms/gba_bios.bin", GBA)
|
||||
self.assertIn("2 folders too deep: it is at gba/roms/gba_bios.bin", self.check())
|
||||
|
||||
def test_a_folder_too_high_is_named(self):
|
||||
self.complete("gba_bios.bin")
|
||||
(self.root / "gba_bios.bin").write_bytes(GBA)
|
||||
self.assertIn(f"one folder too high: it is at {self.root.resolve() / 'gba_bios.bin'}",
|
||||
self.check())
|
||||
|
||||
def test_a_name_no_core_opens_is_named(self):
|
||||
self.complete("scph5501.bin")
|
||||
self.place("PSX BIOS (USA).bin", US)
|
||||
self.assertIn("a name no core opens: its contents are at PSX BIOS (USA).bin, "
|
||||
"rename it scph5501.bin", self.check())
|
||||
|
||||
def test_a_save_directory_file_left_in_the_bios_folder_is_in_the_wrong_folder(self):
|
||||
"""A pack carries the save directory's files under saves/: extracted
|
||||
into the BIOS folder they sit beside it, not too deep in it."""
|
||||
self.place("saves/azahar/keys.txt", b"keys")
|
||||
saves = self.root / "saves"
|
||||
saves.mkdir()
|
||||
install._SAVE_DIRS[self.dest] = install.SavePlace(saves, False)
|
||||
self.addCleanup(install._SAVE_DIRS.pop, self.dest, None)
|
||||
entry = _entry("azahar/keys.txt", b"keys", ["nintendo-3ds"], root="save")
|
||||
found = install.diagnose([entry], [], [entry], self.dest)
|
||||
self.assertEqual(found.reasons[id(entry)],
|
||||
"wrong folder: it is at saves/azahar/keys.txt")
|
||||
self.assertEqual(found.deep, {})
|
||||
|
||||
def test_each_flagged_file_says_where_its_cores_open_it(self):
|
||||
self.complete("scph5501.bin", "pcsx/extra.bin")
|
||||
out = self.check()
|
||||
here = self.dest.resolve()
|
||||
self.assertIn(f"opened at {here / 'scph5501.bin'} by beetle_psx, pcsx_rearmed", out)
|
||||
self.assertIn(f"opened at {here / 'pcsx' / 'extra.bin'} by beetle_psx, pcsx_rearmed",
|
||||
out)
|
||||
|
||||
def test_a_complete_folder_says_nothing_more(self):
|
||||
self.complete()
|
||||
proc = self.run_installer("--check", "--manifest", str(self.list))
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertNotIn("opened at", proc.stdout)
|
||||
|
||||
|
||||
def _pack(directory: Path, name: str, members: dict[str, bytes]) -> Path:
|
||||
"""A pack as generate_pack writes one: the files and a manifest.json."""
|
||||
listed = [{"path": path, "sha1": _sha1(data), "md5": hashlib.md5(data).hexdigest(),
|
||||
"size": len(data), "status": "verified", "name": path.rsplit("/", 1)[-1]}
|
||||
for path, data in members.items()]
|
||||
path = directory / name
|
||||
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as archive:
|
||||
for member, data in members.items():
|
||||
archive.writestr(member, data)
|
||||
archive.writestr("README.txt", b"read me\n")
|
||||
archive.writestr("manifest.json", json.dumps({"files": listed}))
|
||||
return path
|
||||
|
||||
|
||||
def _noise(seed: str, size: int) -> bytes:
|
||||
"""Bytes deflate cannot shrink."""
|
||||
return b"".join(hashlib.sha256(f"{seed}{n}".encode()).digest()
|
||||
for n in range(size // 32))
|
||||
|
||||
|
||||
BIG = {f"dir{n}/file{n}.bin": _noise(str(n), 2048) for n in range(4)}
|
||||
|
||||
|
||||
class SplitParts(Scratch):
|
||||
def split(self) -> list[Path]:
|
||||
downloads = self.root / "downloads"
|
||||
downloads.mkdir()
|
||||
pack = _pack(downloads, "RetroArch_v1_BIOS_Pack.zip", BIG)
|
||||
parts = split_pack.split_pack(pack, limit=pack.stat().st_size // 3)
|
||||
self.assertGreater(len(parts), 2)
|
||||
return parts
|
||||
|
||||
def extract(self, parts: list[Path], into: Path | None = None) -> None:
|
||||
for part in parts:
|
||||
with zipfile.ZipFile(part) as archive:
|
||||
archive.extractall(into or self.dest)
|
||||
|
||||
def list_of_pack(self) -> None:
|
||||
self.list.write_text(json.dumps({
|
||||
"manifest_version": 2, "platform": "retroarch",
|
||||
"files": [_entry(path, data, ["arcade"]) for path, data in BIG.items()],
|
||||
}))
|
||||
|
||||
def test_check_names_the_part_missing_from_the_folder(self):
|
||||
parts = self.split()
|
||||
self.list_of_pack()
|
||||
self.extract(parts[:1] + parts[2:])
|
||||
out = self.check()
|
||||
count = len(parts)
|
||||
self.assertIn(f"RetroArch_v1_BIOS_Pack.zip comes in {count} parts, all needed.", out)
|
||||
self.assertIn(f"missing: part 2, RetroArch_v1_BIOS_Pack.part2of{count}.zip", out)
|
||||
self.assertNotIn("missing: part 1,", out)
|
||||
|
||||
def test_check_names_a_part_extracted_into_a_folder_of_its_own(self):
|
||||
parts = self.split()
|
||||
self.list_of_pack()
|
||||
self.extract(parts[1:])
|
||||
own = self.dest / parts[0].stem
|
||||
self.extract(parts[:1], own)
|
||||
out = self.check()
|
||||
self.assertIn(f"Part 1 went into {own}, a folder of its own", out)
|
||||
self.assertNotIn("missing: part 1,", out)
|
||||
|
||||
def test_a_complete_set_of_parts_says_nothing(self):
|
||||
parts = self.split()
|
||||
self.list_of_pack()
|
||||
self.extract(parts)
|
||||
proc = self.run_installer("--check", "--manifest", str(self.list))
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertNotIn("comes in", proc.stdout)
|
||||
|
||||
def test_from_pack_finds_the_other_parts_beside_one(self):
|
||||
parts = self.split()
|
||||
proc = self.run_installer("--from-pack", str(parts[1]))
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
for path, data in BIG.items():
|
||||
self.assertEqual((self.dest / path).read_bytes(), data)
|
||||
|
||||
def test_from_pack_names_a_part_that_is_not_there(self):
|
||||
parts = self.split()
|
||||
parts[1].unlink()
|
||||
proc = self.run_installer("--from-pack", str(parts[0]))
|
||||
self.assertEqual(proc.returncode, 1)
|
||||
self.assertIn(f"not in {parts[0].parent}: {parts[1].name}", proc.stderr)
|
||||
self.assertFalse(self.dest.exists())
|
||||
|
||||
|
||||
class FromPackInstallsOffline(Scratch):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.downloads = self.root / "downloads"
|
||||
self.downloads.mkdir()
|
||||
self.pack = _pack(self.downloads, "RetroArch_v1_BIOS_Pack.zip", PAYLOAD)
|
||||
|
||||
def installed(self) -> dict[str, bytes]:
|
||||
return {p.relative_to(self.dest).as_posix(): p.read_bytes()
|
||||
for p in self.dest.rglob("*") if p.is_file()}
|
||||
|
||||
def test_the_pack_installs_with_no_network(self):
|
||||
proc = self.run_installer("--from-pack", str(self.pack))
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertEqual(self.installed(), PAYLOAD)
|
||||
self.assertIn("Copying 4 files", proc.stdout)
|
||||
self.assertNotIn("Fetching", proc.stdout)
|
||||
|
||||
def test_the_folder_holding_the_pack_is_enough(self):
|
||||
proc = self.run_installer("--from-pack", str(self.downloads))
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertEqual(self.installed(), PAYLOAD)
|
||||
|
||||
def test_the_platform_is_read_from_the_pack_name(self):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(REPO_ROOT / "install.py"), "--dest", str(self.dest),
|
||||
"--from-pack", str(self.pack), "--no-input"],
|
||||
env=_offline_env(), capture_output=True, text=True, timeout=120,
|
||||
check=False, stdin=subprocess.DEVNULL,
|
||||
)
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertEqual(self.installed(), PAYLOAD)
|
||||
|
||||
def test_a_pack_for_another_platform_is_refused(self):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(REPO_ROOT / "install.py"), "--platform", "batocera",
|
||||
"--dest", str(self.dest), "--from-pack", str(self.pack)],
|
||||
env=_offline_env(), capture_output=True, text=True, timeout=120,
|
||||
check=False, stdin=subprocess.DEVNULL,
|
||||
)
|
||||
self.assertEqual(proc.returncode, 1)
|
||||
self.assertIn("is a pack for RetroArch, not Batocera", proc.stderr)
|
||||
|
||||
def test_a_file_whose_bytes_differ_from_the_list_is_refused(self):
|
||||
"""The hash is checked against the list the run reads, not trusted
|
||||
from the archive that carries both."""
|
||||
listed = manifest()
|
||||
listed["files"][0]["sha1"] = _sha1(b"something else")
|
||||
self.list.write_text(json.dumps(listed))
|
||||
proc = self.run_installer("--from-pack", str(self.pack), "--manifest", str(self.list))
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr)
|
||||
self.assertIn("scph5501.bin", proc.stdout)
|
||||
self.assertFalse((self.dest / "scph5501.bin").exists())
|
||||
self.assertEqual((self.dest / "gba_bios.bin").read_bytes(), GBA)
|
||||
|
||||
def test_with_a_saved_list_the_choice_options_apply(self):
|
||||
proc = self.run_installer("--from-pack", str(self.pack), "--manifest",
|
||||
str(self.list), "--system", "nintendo-gba")
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertEqual(self.installed(), {"gba_bios.bin": GBA})
|
||||
|
||||
def test_a_file_the_pack_does_not_hold_fails_by_name(self):
|
||||
listed = manifest()
|
||||
listed["files"].append(_entry("absent.bin", b"absent", ["nintendo-gba"]))
|
||||
self.list.write_text(json.dumps(listed))
|
||||
proc = self.run_installer("--from-pack", str(self.pack), "--manifest", str(self.list))
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr)
|
||||
self.assertIn("absent.bin (not in the pack)", proc.stdout)
|
||||
|
||||
def test_a_data_tree_is_rebuilt_from_the_pack(self):
|
||||
"""The tree's source archive is never fetched: the pack holds it
|
||||
where it extracts."""
|
||||
tree = {"Sys/GC/font.bin": b"font bytes", "Sys/Wii/setting.txt": b"setting"}
|
||||
self.pack = _pack(self.downloads, "RetroArch_v1_BIOS_Pack.zip",
|
||||
{**PAYLOAD, **{f"dolphin-emu/{k}": v for k, v in tree.items()}})
|
||||
listed = manifest()
|
||||
listed["data_directories"] = [{
|
||||
"ref": "dolphin-sys", "dest": "dolphin-emu", "type": "zip",
|
||||
"url": "https://example.invalid/dolphin.zip", "systems": ["nintendo-gba"],
|
||||
"files": {rel: {"sha1": _sha1(data), "size": len(data)}
|
||||
for rel, data in tree.items()},
|
||||
}]
|
||||
self.list.write_text(json.dumps(listed))
|
||||
proc = self.run_installer("--from-pack", str(self.pack), "--manifest", str(self.list))
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertIn("dolphin-sys: 2 installed", proc.stdout)
|
||||
for rel, data in tree.items():
|
||||
self.assertEqual((self.dest / "dolphin-emu" / rel).read_bytes(), data)
|
||||
|
||||
def test_a_cut_download_is_named(self):
|
||||
self.pack.write_bytes(self.pack.read_bytes()[:100])
|
||||
proc = self.run_installer("--from-pack", str(self.pack))
|
||||
self.assertEqual(proc.returncode, 1)
|
||||
self.assertIn("cannot be read as a ZIP", proc.stderr)
|
||||
|
||||
def test_target_is_refused_with_from_pack(self):
|
||||
proc = self.run_installer("--from-pack", str(self.pack), "--target", "switch")
|
||||
self.assertEqual(proc.returncode, 2)
|
||||
self.assertIn("--target", proc.stderr)
|
||||
|
||||
def test_no_request_leaves_the_process(self):
|
||||
"""The guard holds in the code, not only in the test's dead proxy."""
|
||||
install.LOCAL.reset()
|
||||
install.LOCAL.pack = install.PackSource([self.pack])
|
||||
self.addCleanup(install.LOCAL.reset)
|
||||
with self.assertRaises(OSError):
|
||||
install._fetch_json("http://example.invalid/x.json", 10, "manifest")
|
||||
|
||||
|
||||
class ManifestFromDisk(unittest.TestCase):
|
||||
def test_a_saved_list_installs_without_fetching_the_list(self):
|
||||
with FakeRepository() as repo:
|
||||
saved = repo.root / "saved.json"
|
||||
served = repo.root / "install" / "retroarch.json"
|
||||
saved.write_bytes(served.read_bytes())
|
||||
served.unlink()
|
||||
proc = repo.run("--manifest", str(saved))
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertEqual(len(repo.installed()), 5)
|
||||
|
||||
def test_a_list_for_another_platform_is_refused(self):
|
||||
with FakeRepository() as repo:
|
||||
saved = repo.root / "saved.json"
|
||||
data = json.loads((repo.root / "install" / "retroarch.json").read_text())
|
||||
data["platform"] = "batocera"
|
||||
saved.write_text(json.dumps(data))
|
||||
proc = repo.run("--manifest", str(saved))
|
||||
self.assertEqual(proc.returncode, 1)
|
||||
self.assertIn("not valid for retroarch", proc.stderr)
|
||||
|
||||
|
||||
def _sized(dest: str, size: int, systems: list[str], cores: list[str] | None = None) -> dict:
|
||||
return {"dest": dest, "size": size, "sha1": "0" * 40, "systems": systems,
|
||||
"cores": cores, "repo_path": f"bios/{dest}"}
|
||||
|
||||
|
||||
MB = 1024 * 1024
|
||||
|
||||
|
||||
class TheSummaryNamesTheBulk(unittest.TestCase):
|
||||
def test_a_core_pulling_most_of_a_system_is_named_with_the_saving(self):
|
||||
files = [
|
||||
_sized("opera.bin", 12 * MB, ["3do"]),
|
||||
_sized("mame/3do.zip", 300 * MB, ["3do"], ["mame"]),
|
||||
_sized("mame/3do_m2.zip", 37 * MB, ["3do"], ["mame"]),
|
||||
]
|
||||
lines = install.bulk_hint(files, {"system_cores": {"3do": ["mame", "opera"]}},
|
||||
systems_chosen=True, cores_chosen=False)
|
||||
self.assertEqual(lines, [
|
||||
" mame pulls 337.0 MB of this 349.0 MB download for 3do. --core opera "
|
||||
"keeps 12.0 MB of it and saves 337.0 MB (96%)."
|
||||
])
|
||||
|
||||
def test_a_system_pulling_most_of_the_download_is_named(self):
|
||||
files = [_sized("a.zip", 400 * MB, ["arcade"]), _sized("b.bin", 50 * MB, ["nes"]),
|
||||
_sized("c.bin", 10 * MB, ["gba"])]
|
||||
lines = install.bulk_hint(files, {}, systems_chosen=False, cores_chosen=False)
|
||||
self.assertEqual(len(lines), 1)
|
||||
self.assertIn("arcade pulls 400.0 MB of this 460.0 MB download", lines[0])
|
||||
self.assertIn("saves 400.0 MB (86%)", lines[0])
|
||||
|
||||
def test_nothing_is_said_under_half(self):
|
||||
files = [_sized("a.zip", 100 * MB, ["arcade"]), _sized("b.bin", 100 * MB, ["nes"])]
|
||||
self.assertEqual(install.bulk_hint(files, {}, False, False), [])
|
||||
|
||||
def test_nothing_is_said_for_a_small_download(self):
|
||||
files = [_sized("a.zip", 40 * MB, ["arcade"]), _sized("b.bin", 1 * MB, ["nes"])]
|
||||
self.assertEqual(install.bulk_hint(files, {}, False, False), [])
|
||||
|
||||
def test_a_chosen_core_is_not_second_guessed(self):
|
||||
files = [_sized("opera.bin", 12 * MB, ["3do"]),
|
||||
_sized("mame/3do.zip", 300 * MB, ["3do"], ["mame"])]
|
||||
self.assertEqual(install.bulk_hint(
|
||||
files, {"system_cores": {"3do": ["mame", "opera"]}}, True, True), [])
|
||||
|
||||
def test_the_end_summary_prints_the_line(self):
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out):
|
||||
install._report_outcome([Path("/x")], 3, 0, 0, 0, False,
|
||||
hints=[" arcade pulls most of it."])
|
||||
text = out.getvalue()
|
||||
self.assertIn(" arcade pulls most of it.", text)
|
||||
self.assertLess(text.index("Done."), text.index("arcade pulls"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -341,6 +341,11 @@ class TheTableDescribesTheRelease(unittest.TestCase):
|
||||
self.assertTrue(f"| RetroArch | {files:,} |" in readme)
|
||||
|
||||
|
||||
def _incompressible(seed: int) -> bytes:
|
||||
"""960 bytes deflate cannot shrink, so a member outweighs a part marker."""
|
||||
return b"".join(hashlib.sha256(bytes([seed, n])).digest() for n in range(30))
|
||||
|
||||
|
||||
class ReleaseRecord(unittest.TestCase):
|
||||
OWNERS = {
|
||||
"Whole_1.0_BIOS_Pack.zip": ["whole"],
|
||||
@@ -356,7 +361,7 @@ class ReleaseRecord(unittest.TestCase):
|
||||
self._pack("Whole_1.0_BIOS_Pack.zip", {"a.bin": b"a" * 10, "README.txt": b"r"})
|
||||
big = self._pack(
|
||||
"Demo_2.0_BIOS_Pack.zip",
|
||||
{f"dir/file{n}.bin": bytes([n]) * 700 + bytes(range(256)) for n in range(6)},
|
||||
{f"dir/file{n}.bin": _incompressible(n) for n in range(6)},
|
||||
)
|
||||
self.parts = split_pack.split_pack(big, limit=big.stat().st_size - 1)
|
||||
|
||||
@@ -375,7 +380,7 @@ class ReleaseRecord(unittest.TestCase):
|
||||
self.assertEqual(record["tag"], "v1")
|
||||
demo = record["packs"]["Demo_2.0_BIOS_Pack.zip"]
|
||||
self.assertEqual(demo["files"], 6)
|
||||
self.assertEqual(demo["extracted_size"], 6 * (700 + 256))
|
||||
self.assertEqual(demo["extracted_size"], 6 * 960)
|
||||
self.assertEqual(demo["assets"], [part.name for part in self.parts])
|
||||
self.assertEqual(
|
||||
demo["download_size"], sum(part.stat().st_size for part in self.parts)
|
||||
@@ -399,7 +404,7 @@ class ReleaseRecord(unittest.TestCase):
|
||||
pack, leaves both: summed, the download size doubled."""
|
||||
self._pack(
|
||||
"Demo_2.0_BIOS_Pack.zip",
|
||||
{f"dir/file{n}.bin": bytes([n]) * 700 + bytes(range(256)) for n in range(6)},
|
||||
{f"dir/file{n}.bin": _incompressible(n) for n in range(6)},
|
||||
)
|
||||
with self.assertRaises(release_record.UnfinishedSplitError):
|
||||
release_record.build_record(self.dist, "v1", owners=self.OWNERS)
|
||||
|
||||
@@ -67,6 +67,8 @@ class SplitFixture(unittest.TestCase):
|
||||
with zipfile.ZipFile(part) as archive:
|
||||
self.assertIsNone(archive.testzip())
|
||||
for name in archive.namelist():
|
||||
if split_pack.is_marker(name):
|
||||
continue
|
||||
self.assertNotIn(name, found, "a file sits in two parts")
|
||||
found[name] = archive.read(name)
|
||||
return found
|
||||
@@ -137,6 +139,61 @@ class PartsAreArchives(SplitFixture):
|
||||
self.assertTrue(modes["bin/engine"] & stat.S_IXUSR)
|
||||
|
||||
|
||||
class EachPartNamesItsSiblings(SplitFixture):
|
||||
"""A part extracted alone looked like a complete pack. Each part now
|
||||
carries a marker naming the pack, its number and every part with its
|
||||
size, so a missing one can be named from the extracted folder."""
|
||||
|
||||
def _markers(self, parts: list[Path]) -> dict[str, str]:
|
||||
found = {}
|
||||
for part in parts:
|
||||
with zipfile.ZipFile(part) as archive:
|
||||
names = [n for n in archive.namelist() if split_pack.is_marker(n)]
|
||||
self.assertEqual(len(names), 1, part.name)
|
||||
found[part.name] = archive.read(names[0]).decode("utf-8")
|
||||
return found
|
||||
|
||||
def test_every_part_carries_one_marker(self):
|
||||
parts = split_pack.split_pack(self.pack, limit=LIMIT)
|
||||
count = len(parts)
|
||||
for index, part in enumerate(parts, 1):
|
||||
with zipfile.ZipFile(part) as archive:
|
||||
self.assertIn(f"Demo_1.0_BIOS_Pack.part{index}of{count}.txt",
|
||||
archive.namelist())
|
||||
|
||||
def test_the_marker_names_the_pack_the_part_and_every_size(self):
|
||||
parts = split_pack.split_pack(self.pack, limit=LIMIT)
|
||||
count = len(parts)
|
||||
for index, (name, text) in enumerate(self._markers(parts).items(), 1):
|
||||
with self.subTest(part=name):
|
||||
lines = text.splitlines()
|
||||
self.assertIn("pack: Demo_1.0_BIOS_Pack.zip", lines)
|
||||
self.assertIn(f"part: {index} of {count}", lines)
|
||||
listed = [line.split()[1:] for line in lines if line.startswith("file:")]
|
||||
self.assertEqual(
|
||||
listed,
|
||||
[[str(p.stat().st_size), p.name] for p in parts],
|
||||
)
|
||||
|
||||
def test_the_parts_still_fit_with_their_markers(self):
|
||||
for part in split_pack.split_pack(self.pack, limit=LIMIT):
|
||||
self.assertLess(part.stat().st_size, LIMIT)
|
||||
|
||||
def test_a_pack_holding_a_marker_name_is_refused(self):
|
||||
members = dict(self.members)
|
||||
members["Demo_1.0_BIOS_Pack.part1of2.txt"] = b"x"
|
||||
self.pack.unlink()
|
||||
self._build(self.pack, members)
|
||||
with self.assertRaises(ValueError):
|
||||
split_pack.split_pack(self.pack, limit=LIMIT)
|
||||
self.assertTrue(self.pack.is_file())
|
||||
|
||||
def test_a_marker_is_not_a_part(self):
|
||||
self.assertTrue(split_pack.is_marker("Demo_BIOS_Pack.part2of3.txt"))
|
||||
self.assertFalse(split_pack.is_part("Demo_BIOS_Pack.part2of3.txt"))
|
||||
self.assertFalse(split_pack.is_marker("Demo_BIOS_Pack.part2of3.zip"))
|
||||
|
||||
|
||||
class SplitHoldsTheDirectory(SplitFixture):
|
||||
"""split_pack read and deleted packs in a directory a build was writing:
|
||||
the build could truncate the pack under the split, or the split delete
|
||||
|
||||
+93
-5
@@ -223,6 +223,16 @@ Location: /userdata/bios
|
||||
To check this later, run the same command with --check.
|
||||
```
|
||||
|
||||
When most of a download of 100 MB or more comes from one system, or from
|
||||
one core of a single system, the end of the run says so and what narrowing
|
||||
would save, once that saving is over half:
|
||||
|
||||
```
|
||||
mame pulls 336.5 MB of this 349.0 MB download for 3do. --core opera keeps 12.5 MB of it and saves 336.5 MB (96%).
|
||||
```
|
||||
|
||||
`--dry-run` prints the same line under its plan.
|
||||
|
||||
The word `Done` and a location are the confirmation. Any other ending
|
||||
(`Stopped`, `Finished with errors`, `Nothing was installed`) means something
|
||||
was left undone, and the run says which case it was.
|
||||
@@ -252,6 +262,52 @@ present with contents the platform does not accept. Running the install again
|
||||
replaces it and keeps the old one, see
|
||||
[Files you already had](#files-you-already-had).
|
||||
|
||||
### Why a file is refused { #why-a-file-is-refused }
|
||||
|
||||
Each file `--check` flags comes with the reason it found and the place each
|
||||
core opens it:
|
||||
|
||||
```
|
||||
3 files to download:
|
||||
wrong hash: scph5501.bin
|
||||
another region: the contents of scph5500.bin (japan)
|
||||
opened at /home/me/.config/retroarch/system/scph5501.bin by beetle_psx, pcsx_rearmed
|
||||
missing: gba_bios.bin
|
||||
nested folder: it is at system/system/gba_bios.bin, a system folder inside system
|
||||
opened at /home/me/.config/retroarch/system/gba_bios.bin by mgba
|
||||
missing: pcsx/extra.bin
|
||||
still zipped: it is inside PSX BIOS.zip
|
||||
opened at /home/me/.config/retroarch/system/pcsx/extra.bin by pcsx_rearmed
|
||||
```
|
||||
|
||||
| Reason | What the folder holds |
|
||||
|--------|-----------------------|
|
||||
| `nested folder` | The file inside a folder named like the BIOS folder itself (`bios/bios/`, `system/system/`) |
|
||||
| `one folder too deep`, `2 folders too deep` | The file below its place, as a ZIP extracted into a folder of its own leaves it |
|
||||
| `one folder too high` | The file in the folder above the BIOS folder |
|
||||
| `wrong folder` | The file under its name in another folder |
|
||||
| `case of the name` | A name that differs only by capital letters, on a system where that makes two names |
|
||||
| `still zipped` | The file inside a ZIP in the folder, a `.7z` or `.rar` of the same name, or a ZIP saved under the file's own name |
|
||||
| `wrong size` | A file of another size than any copy the list accepts |
|
||||
| `another region`, `another revision`, `another file` | The contents of another file of the list: another region's BIOS, the copy another core reads, or the one the platform's own list declares |
|
||||
| `a name no core opens` | The right contents under a name no core of the platform reads |
|
||||
| `same size, other contents` | Another dump, or a damaged copy |
|
||||
|
||||
The reasons are read from the folder and the file list, nothing else. When
|
||||
three files or more sit too deep in the same folder, a line before the list
|
||||
names that folder once. Files with a reason are always listed. The plain
|
||||
missing ones stop at ten unless `--verbose` is given.
|
||||
|
||||
A pack published in parts leaves a marker beside the files of each part
|
||||
(`RetroArch_v1.22.2_BIOS_Pack.part1of3.txt`). `--check` reads them and names
|
||||
a part that was not extracted, or one extracted into a folder of its own:
|
||||
|
||||
```
|
||||
RetroArch_v1.22.2_BIOS_Pack.zip comes in 3 parts, all needed.
|
||||
missing: part 2, RetroArch_v1.22.2_BIOS_Pack.part2of3.zip (1.9 GB)
|
||||
Download it and extract into /home/me/.config/retroarch/system, beside the other parts.
|
||||
```
|
||||
|
||||
## Files you already had
|
||||
|
||||
A file in the BIOS folder whose contents differ from the expected one (another
|
||||
@@ -448,6 +504,8 @@ what the options name, everything by default.
|
||||
| `--yes`, `-y` | Do not ask before a download of 1 GB or more |
|
||||
| `--quiet`, `-q` | Print only failures and the final summary. Implies `--no-input` |
|
||||
| `--events` | Write the run as JSON events on standard output, one object per line, and the usual text on standard error. Implies `--no-input`. See [Machine-readable events](#machine-readable-events) |
|
||||
| `--from-pack PATH` | Install from a pack already downloaded: its ZIP, one of its parts (the others are found in the same folder), or the folder holding them. Makes no network request and checks every file's hash. See [Without a network](#without-a-network) |
|
||||
| `--manifest FILE` | Read the file list from FILE, a copy of `install/<platform>.json`, instead of downloading it. See [Without a network](#without-a-network) |
|
||||
| `--restore FOLDER` | Put back the files a run replaced and kept in FOLDER, a dated folder inside `retrobios-backup`. Takes no other option but `--dry-run` and `--no-input`. See [Files you already had](#files-you-already-had) |
|
||||
| `--version` | Print the installer's identity, the first 12 hex digits of the SHA-256 the bootstraps pin |
|
||||
| `--list-platforms` | Print the supported platforms and what was detected here |
|
||||
@@ -459,8 +517,9 @@ what the options name, everything by default.
|
||||
|
||||
`--check` reads the same file list, hashes the destination and exits, with
|
||||
code 1 when a file is missing or does not match and 0 when every one does. Files
|
||||
with a wrong hash are listed first and in full, then the first ten missing ones
|
||||
(`--verbose` lists them all). `--dry-run` counts the data trees in its total. Without
|
||||
with a wrong hash are listed first and in full, then the missing ones whose
|
||||
reason it found, then the first ten other missing ones (`--verbose` lists them
|
||||
all). See [Why a file is refused](#why-a-file-is-refused). `--dry-run` counts the data trees in its total. Without
|
||||
it, a file already present with the expected hash is left untouched, and one
|
||||
whose contents do not match is re-downloaded and replaced by the verified copy,
|
||||
the old one kept beside the BIOS folder.
|
||||
@@ -806,6 +865,34 @@ The launcher (`ports/RetroBIOS/RetroBIOS.sh`) and its screens
|
||||
(`retrobios/screens.txt`) read the installer through
|
||||
[Machine-readable events](#machine-readable-events) only.
|
||||
|
||||
## Without a network { #without-a-network }
|
||||
|
||||
A pack downloaded once installs again with no network, on this machine or on
|
||||
one that has none:
|
||||
|
||||
```bash
|
||||
python install.py --from-pack ~/Downloads/RetroArch_v1.22.2_BIOS_Pack.zip
|
||||
python install.py --from-pack ~/Downloads/RetroArch_v1.22.2_BIOS_Pack.part1of3.zip
|
||||
python install.py --from-pack ~/Downloads --dest /media/card/retroarch/system
|
||||
```
|
||||
|
||||
One part is enough to name a split pack: the others are taken from the same
|
||||
folder, and a part that is not there is named before anything is written.
|
||||
The platform is read from the pack's name when `--platform` is not given. The
|
||||
file list is the one the pack carries (`manifest.json`), with each file's
|
||||
SHA-1 and size, and every file is checked against it before it takes its
|
||||
place. A data tree (Dolphin's `Sys/`, PPSSPP's assets) comes from the pack
|
||||
too, not from its source archive. `--target` reads a core list from the
|
||||
network and is refused with `--from-pack`. A board found on the device is not
|
||||
applied either: every board's files are installed.
|
||||
|
||||
`--manifest FILE` reads the file list from a copy of `install/<platform>.json`
|
||||
instead of downloading it. Alone, the files still come from the repository.
|
||||
With `--from-pack`, the run reads that list and takes every file from the
|
||||
pack: `--system`, `--core`, `--region` and `--no-game-data` then apply as
|
||||
they do online, and a file of the list the pack does not hold fails as
|
||||
`not in the pack`. The list's platform has to be the one installed for.
|
||||
|
||||
## Where the files come from
|
||||
|
||||
Each manifest entry names either `repo_path`, served from the repository at the
|
||||
@@ -911,8 +998,8 @@ fields, the installer refuses to narrow rather than install everything.
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | everything asked for is installed or already up to date |
|
||||
| `1` | a failure: the one-liner could not download or verify `install.py`, or found no Python 3.8 (nothing was run), an unusable base URL, an unknown platform or target, `--installed-cores` with no core found, a manifest that fails validation, a fetch that never succeeded, no platform to install for in a non-interactive run, some files failed while others installed, `--check` found files to fetch, or `--restore` could not put a file back |
|
||||
| `2` | nothing was installed: every file the run tried failed (a BIOS folder that cannot be written, for one), the drive has too little free space for the download (checked before any folder is created), or the question about a `--dest` folder that does not exist was answered no. Also an option the installer refused before any download: an unknown option, `--jobs` outside 1 to 32, `--target` with `--every-board`. The message then starts with `install.py: error:`. With `--restore`, a folder that is not a dated folder inside `retrobios-backup` |
|
||||
| `1` | a failure: the one-liner could not download or verify `install.py`, or found no Python 3.8 (nothing was run), an unusable base URL, an unknown platform or target, `--installed-cores` with no core found, a manifest that fails validation, a fetch that never succeeded, no platform to install for in a non-interactive run, some files failed while others installed, `--check` found files to fetch, a `--from-pack` pack that cannot be read, lacks a part or is for another platform, or `--restore` could not put a file back |
|
||||
| `2` | nothing was installed: every file the run tried failed (a BIOS folder that cannot be written, for one), the drive has too little free space for the download (checked before any folder is created), or the question about a `--dest` folder that does not exist was answered no. Also an option the installer refused before any download: an unknown option, `--jobs` outside 1 to 32, `--target` with `--every-board` or `--from-pack`. The message then starts with `install.py: error:`. With `--restore`, a folder that is not a dated folder inside `retrobios-backup` |
|
||||
| `130` | stopped with Ctrl+C, or with Ctrl+C or Ctrl+D at a question. Run the same command again to carry on |
|
||||
| `141` | the output was piped to a reader that closed early (`--list-platforms \| head`) |
|
||||
|
||||
@@ -1054,7 +1141,8 @@ right pack either way.
|
||||
|
||||
Parts named `Pack.part1of2.zip` are ordinary ZIPs: unzip each one into the same
|
||||
folder, and check that all of them were downloaded. The name says how many
|
||||
there are. `scripts/download.sh` and `scripts/download.py` download every part
|
||||
there are, and each part leaves a marker file naming every part, which
|
||||
`--check` reads to name a part that is missing. `scripts/download.sh` and `scripts/download.py` download every part
|
||||
and check it against the release's checksums before unzipping. Both stage
|
||||
inside the destination folder, never in `/tmp`, which is a RAM disk on most of
|
||||
these systems.
|
||||
@@ -213,7 +213,9 @@ python scripts/pipeline.py --include-archived
|
||||
# 2. A release file must be under 2 GiB. A larger pack becomes parts that are
|
||||
# each a ZIP of whole files (Pack.part1of2.zip): any tool opens one, and
|
||||
# the parts extracted into one folder are the pack. Each part is read back
|
||||
# and the set compared with the pack before the pack is removed.
|
||||
# and the set compared with the pack before the pack is removed. Each part
|
||||
# also holds a marker (Pack.part1of2.txt) naming every part and its size,
|
||||
# which install.py --check reads to name a part missing from a folder.
|
||||
python scripts/split_pack.py dist/
|
||||
|
||||
# 2b. The handheld port, RetroBIOS.zip, built from ports/RetroBIOS with the
|
||||
|
||||
@@ -51,7 +51,7 @@ Every module in `tests/`, by the area it guards.
|
||||
| Packs and manifests | synthetic + real packs | `test_artifact_lock`, `test_atomic_artifacts`, `test_batocera_cores`, `test_deterministic_zip`, `test_emulator_pack_counts`, `test_finalize_pack_names`, `test_large_file_cache`, `test_libretro_layout`, `test_manifest_entry`, `test_minui_platform`, `test_no_game_data`, `test_pack_conformance_dirs`, `test_pack_core_extras`, `test_pack_counts`, `test_pack_destinations`, `test_pack_failures`, `test_pack_file_names`, `test_pack_grouping`, `test_pack_integrity`, `test_pack_member_identity`, `test_pack_readme`, `test_platform_inheritance`, `test_platform_keys`, `test_required_only_archives`, `test_required_only_twins`, `test_save_root`, `test_split_extras`, `test_split_names`, `test_split_pack`, `test_system_pack_extras`, `test_system_pack_parity`, `test_torrentzip`, `test_which_pack` |
|
||||
| Slots and regions | synthetic + profiles | `test_region`, `test_region_agnostic_scan`, `test_region_groups`, `test_region_held`, `test_region_platform_index`, `test_region_required_only`, `test_slot`, `test_slots`, `test_slots_shipped_file` |
|
||||
| Hardware targets and systems | synthetic + target files | `test_extract_targets`, `test_lakka_targets`, `test_libretro_slugs`, `test_system_ids`, `test_system_selection`, `test_target_names`, `test_target_scrapers`, `test_target_systems`, `test_target_verify` |
|
||||
| Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_selection` |
|
||||
| Installer and downloads | loopback server | `test_check_release`, `test_download`, `test_install`, `test_install_cores`, `test_install_experience`, `test_install_offline`, `test_install_selection` |
|
||||
| Handheld port | stubbed PortMaster dialogs, fake installer | `test_port` |
|
||||
| Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` |
|
||||
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_dedup`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_profile_sync`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_scraper_contract`, `test_upstream`, `test_upstream_index` |
|
||||
|
||||
+1
-1
@@ -688,7 +688,7 @@ python scripts/refresh_stale.py --only platforms,targets --jobs 6
|
||||
| `list_platforms.py` | List active platforms (`--include-archived` adds the archived ones, used by CI) |
|
||||
| `download.py` | Download a pack from GitHub releases, every part checked (Python, stdlib only) |
|
||||
| `download.sh` | Same, as a shell one-liner (`curl` + `unzip`) |
|
||||
| `split_pack.py` | Cut a pack over the release asset limit into parts that are each a ZIP of whole files, read back before the pack is removed |
|
||||
| `split_pack.py` | Cut a pack over the release asset limit into parts that are each a ZIP of whole files, each with a marker naming every part, read back before the pack is removed |
|
||||
| `provenance_report.py` | Dump-catalog coverage and acquisition targets (see above) |
|
||||
| `find_pinned_revision.py` | Walk the upstream commits that touched a file and return the revision whose bytes match a platform's pinned md5 (`--write` saves them) |
|
||||
| `generate_readme.py` | Generate README.md and CONTRIBUTING.md from database |
|
||||
|
||||
@@ -28,6 +28,7 @@ Read the count line:
|
||||
- **`wrong hash`**: a file is there with the wrong contents: another region
|
||||
or model, or damaged.
|
||||
- **Fewer present than the total**: some files are missing.
|
||||
- **A line under a file**, like `still zipped`: it says what to fix.
|
||||
- **`0 wrong hash` and all present**: your files are fine. Go to step 3.
|
||||
|
||||
## 2. Fix the files
|
||||
@@ -47,7 +48,7 @@ the wrong ones, keeping the old ones.
|
||||
differ only by capital letters, which some ZIPs leave out.
|
||||
- **A ZIP in parts will not open?** Download every part, like
|
||||
`Pack.part1of2.zip` and `Pack.part2of2.zip`. Unzip each one into the same
|
||||
folder.
|
||||
folder. `--check` names a part you missed.
|
||||
|
||||
## One console only
|
||||
|
||||
|
||||
Reference in new issue
Block a user