diff --git a/wiki/adding-a-platform.md b/wiki/adding-a-platform.md index ab5aabc6c..1c03b9592 100644 --- a/wiki/adding-a-platform.md +++ b/wiki/adding-a-platform.md @@ -629,7 +629,7 @@ They live in `scripts/scraper/targets/` and are auto-discovered by filename ```python from scripts.scraper.targets import BaseTargetScraper -PLATFORM_NAME = "myplatform_targets" +PLATFORM_NAME = "myplatform" # the registry key, as the scraper module names it class Scraper(BaseTargetScraper): def fetch_targets(self) -> dict: @@ -650,13 +650,16 @@ class Scraper(BaseTargetScraper): } ``` -Add `target_scraper` and `target_source` to the platform's registry entry. +Add `target_scraper: myplatform_targets` (the module name without `_scraper`) +and `target_source` to the platform's registry entry. ### Overrides -Hardware-specific overrides go in `platforms/targets/_overrides.yml`. This file -defines aliases (e.g. `arm64` maps to `aarch64`) and per-platform core -additions/removals that the scraper cannot determine automatically. +Hardware-specific overrides go in `platforms/targets/_overrides.yml`, which no +scraper writes. Each target can list `aliases` (Batocera's `bcm2711` also +answers to `rpi4`), `add_cores` and `remove_cores`, and `_default` applies to +every target of the platform. `apply_target_overrides()` in `common.py` applies +them, for the packs and for the published target lists alike. ### Single-target platforms @@ -698,18 +701,35 @@ After all pieces are in place, run the full pipeline: python scripts/pipeline.py --offline ``` -This executes in sequence: +This executes in sequence, numbered as the run prints them (`STEP_LABELS` in +`scripts/pipeline.py`): -0. `validate_schemas.py --source-only` - the schema check PR validation runs -1. `generate_db.py` - rebuild `database.json` from `bios/` -1b. `provenance_report.py` - dump-catalog coverage from `provenance/` -2. `refresh_data_dirs.py` - update data directories (skipped with `--offline`) -3. `verify.py --all` - verify all platforms including the new one -4. `generate_pack.py --all` - build ZIP packs + install and target manifests -5. Consistency check: verify and the pack agree on counts and on the paths each archive holds -6. Pack integrity - extract ZIPs and verify hashes per platform mode -7. `generate_readme.py` - regenerate README and CONTRIBUTING -8. `generate_site.py` - regenerate the documentation site and its navigation +- 0 `validate_schemas.py --source-only`, the schema check PR validation runs +- 1 `generate_db.py`, rebuild `database.json` from `bios/` +- 1b `provenance_report.py`, dump-catalog coverage from `provenance/` +- 1c `romset_recipes.py`, which version each arcade archive is +- 2 `refresh_data_dirs.py`, update data directories +- 2a `mame_hash_scraper`, MAME BIOS hashes +- 2a2 `fbneo_hash_scraper`, FBNeo BIOS hashes +- 2b `check_buildbot_system.py`, with `--check-buildbot` only +- 2b2 `check_release_assets.py`, release assets against gitignored files +- 2c `generate_truth.py`, with `--with-truth` or `--with-export` +- 2d `diff_truth.py`, truth against the scraped platforms +- 2e `export_native.py`, with `--with-export` +- 3 `verify.py --all`, verify all platforms including the new one +- 3a `identity.py --strict`, one content under two identities +- 3b `slots.py`, destinations both layers claim +- 4 `generate_pack.py --all`, build the ZIP packs +- 4b `generate_pack.py --manifest`, install manifests +- 4c `generate_pack.py --manifest-targets`, target manifests +- 4d `bundled_only.py --strict`, copies only an emulator install provides +- 5 Consistency check: verify and the pack agree on counts and on the paths each archive holds +- 6 Pack integrity: extract the ZIPs and check hashes per platform mode +- 7 `generate_readme.py`, regenerate README and CONTRIBUTING +- 8 `generate_site.py`, regenerate the site pages and navigation +- 8b `npm run build` in `website/`, then `validate_site.py` + +`--offline` skips steps 2 to 2b2. A new platform is archived-by-default nowhere: `status: active` puts it in `list_platforms.py`, the CI matrix, and the scrapers. Set diff --git a/wiki/adding-a-scraper.md b/wiki/adding-a-scraper.md index ece1c1372..9f97bb166 100644 --- a/wiki/adding-a-scraper.md +++ b/wiki/adding-a-scraper.md @@ -44,6 +44,7 @@ for finder, name, ispkg in pkgutil.iter_modules([package_dir]): ``` BaseScraper __init__(url: str) + lists_files: bool = True # False: the platform lists no file of its own _fetch_raw() -> str # HTTP GET, cached, 50 MB limit fetch_requirements() -> list # abstract: parse upstream data validate_format(raw_data) -> bool # abstract: detect format changes @@ -51,10 +52,11 @@ BaseScraper test_connection() -> bool # reachability check ``` -`_fetch_raw()` handles HTTP with `urllib.request`, sets a `retrobios-scraper/1.0` -user-agent, enforces a 30-second timeout, and reads the response in 64 KB chunks -with a 50 MB hard limit to prevent memory exhaustion. The result is cached on the -instance after the first call. +`_fetch_raw()` goes through `fetch_bytes()`: `urllib.request` with a +`retrobios-scraper/1.0` user-agent and a 30-second timeout, the response read in +64 KB chunks with a 50 MB hard limit. A 429, 502, 503 or 504 answer is retried +up to five attempts, after the server's `Retry-After` or a doubling wait. The +result is cached on the instance after the first call. ### BiosRequirement @@ -73,8 +75,16 @@ class BiosRequirement: required: bool = True zipped_file: str | None = None # ROM name inside a ZIP native_id: str | None = None # original system name + sha256: str | None = None + alt_md5: str | None = None # a second accepted MD5 + native_path: str | None = None # the reference as the platform writes it + mirror_of: str | None = None # the native system this entry repeats + native: dict[str, object] = field(default_factory=dict) # fields written back unchanged ``` +[BiosRequirement fields](adding-a-platform.md#biosrequirement-fields) says what +each one carries. + ### ChangeSet Returned by `compare_with_config()`. Contains: @@ -95,10 +105,12 @@ Returned by `compare_with_config()`. Contains: | `--json` | Output all requirements as JSON | | `--output FILE` | Write platform YAML to FILE | -When `--output` targets an existing file, the CLI preserves keys not generated by -the scraper (e.g. `data_directories`, manual additions). Only the `systems` section -is replaced. If the scraper defines `generate_platform_yaml()`, that method is used -instead of the generic YAML builder. +When `--output` targets an existing file, every top-level key the scraper writes +is replaced and the others are kept, as is a system's `data_directories` the new +scrape does not carry. If the scraper defines `generate_platform_yaml()`, that +method is used instead of the generic YAML builder. A scrape that returns no entry, +or a YAML that lists no file, stops with an error instead of writing, unless the +scraper sets `lists_files = False`. ### Helper functions @@ -247,6 +259,7 @@ pattern (`*_targets_scraper.py`). BaseTargetScraper __init__(url: str) fetch_targets() -> dict # abstract: return target data + pinned_revision(platform) # the upstream revision platforms/.yml was read at write_output(data, path) # write YAML to disk ``` @@ -280,19 +293,21 @@ pipeline to work correctly. `platforms/targets/_overrides.yml` provides post-scrape adjustments: -- **aliases**: map alternate target names to canonical ones (e.g. `arm64` -> `aarch64`) +- **aliases**: other names a target answers to (Batocera's `bcm2711` is also `rpi4`) - **add_cores**: cores present on a target but not detected by the scraper -- **remove_cores**: cores detected by the scraper but not actually functional +- **remove_cores**: names the scraper reads as cores that emulate nothing +- **_default**: an entry applied to every target of the platform -Overrides are applied by `load_target_config()` in `common.py` after loading -the scraped data. The scraper itself does not need to handle overrides. +`apply_target_overrides()` in `common.py` applies them, read by +`load_target_config()` and by the target manifests. The scraper itself does not +handle overrides. ### Module contract ```python from scripts.scraper.targets import BaseTargetScraper -PLATFORM_NAME = "myplatform_targets" +PLATFORM_NAME = "myplatform" # the platform's registry key class Scraper(BaseTargetScraper): def __init__(self): @@ -319,6 +334,9 @@ Register the target scraper in `_registry.yml`: | `batocera_targets` | Config.in + es_systems.yml | Cross-reference kernel config with system definitions | | `emudeck_targets` | EmuScripts + RetroArch cores | GitHub API for script availability per OS | | `retropie_targets` | scriptmodules + rp_module_flags | Parse Bash scriptmodules for platform flags | +| `lakka_targets` | Lakka-LibreELEC `package.mk` and project options | Core list minus each project and device's exclusions | +| `recalbox_targets` | board defconfigs + `system.ini` | Buildroot packages each image enables, cores per system | +| `rocknix_targets` | emulators `package.mk` + device options | Cores per SoC block and 32-bit support per device | ## Parser modules @@ -360,7 +378,7 @@ since neither can be fetched without a browser). Both write a snapshot to Parses MAME C source files to extract BIOS root sets. Handles: -- Machine declaration macros: `GAME`, `SYST`, `COMP`, `CONS` +- Machine declaration macros: `GAME`, `GAMEL`, `SYST`, `COMP`, `CONS` - `MACHINE_IS_BIOS_ROOT` flag detection - `ROM_START`/`ROM_END` blocks - `ROM_LOAD` variants and `ROM_REGION` declarations @@ -442,6 +460,9 @@ be missing from the round trip. # Scrape upstream -> platform YAML python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml +# Build the truth model the export reads (refused when out of date) +python scripts/generate_truth.py --platform myplatform --output-dir dist/truth + # Rewrite the platform's own file with the changes applied python scripts/export_native.py --platform myplatform --fetch \ --output-dir tmp/native @@ -461,10 +482,10 @@ the fields listed in [BiosRequirement fields](adding-a-platform.md#biosrequireme | Symptom | Cause | Fix | |---------|-------|-----| | Unmapped slug warnings | New system added upstream | Add mapping to `SLUG_MAP` | -| Empty requirements list | Upstream format changed | Check `validate_format()`, update parser | +| `upstream returned no BIOS entry` | Upstream format changed | Check `validate_format()`, update parser | | Hash mismatch in verify | Upstream updated hashes | Re-scrape and regenerate platform YAML | | Scraper hangs | URL unreachable, no timeout | `_fetch_raw()` has 30s timeout, check URL | -| `Response exceeds 50 MB` | Upstream file grew | Investigate, may need chunked parsing | +| `Response exceeds 52428800 byte limit` | Upstream file grew | Investigate, may need chunked parsing | | `validate_format` fails | Upstream restructured | Update both `validate_format` and `fetch_requirements` | ### E2E tests