docs: scraper and target contracts from the code

wiki/adding-a-platform.md

@@ -629,7 +629,7 @@ They live in `scripts/scraper/targets/` and are auto-discovered by filename
 ```python
 from scripts.scraper.targets import BaseTargetScraper
 
-PLATFORM_NAME = "myplatform_targets"
+PLATFORM_NAME = "myplatform"  # the registry key, as the scraper module names it
 
 class Scraper(BaseTargetScraper):
     def fetch_targets(self) -> dict:
@@ -650,13 +650,16 @@ class Scraper(BaseTargetScraper):
         }
 ```
 
-Add `target_scraper` and `target_source` to the platform's registry entry.
+Add `target_scraper: myplatform_targets` (the module name without `_scraper`)
+and `target_source` to the platform's registry entry.
 
 ### Overrides
 
-Hardware-specific overrides go in `platforms/targets/_overrides.yml`. This file
-defines aliases (e.g. `arm64` maps to `aarch64`) and per-platform core
-additions/removals that the scraper cannot determine automatically.
+Hardware-specific overrides go in `platforms/targets/_overrides.yml`, which no
+scraper writes. Each target can list `aliases` (Batocera's `bcm2711` also
+answers to `rpi4`), `add_cores` and `remove_cores`, and `_default` applies to
+every target of the platform. `apply_target_overrides()` in `common.py` applies
+them, for the packs and for the published target lists alike.
 
 ### Single-target platforms
 
@@ -698,18 +701,35 @@ After all pieces are in place, run the full pipeline:
 python scripts/pipeline.py --offline
 ```
 
-This executes in sequence:
+This executes in sequence, numbered as the run prints them (`STEP_LABELS` in
+`scripts/pipeline.py`):
 
-0. `validate_schemas.py --source-only` - the schema check PR validation runs
-1. `generate_db.py` - rebuild `database.json` from `bios/`
-1b. `provenance_report.py` - dump-catalog coverage from `provenance/`
-2. `refresh_data_dirs.py` - update data directories (skipped with `--offline`)
-3. `verify.py --all` - verify all platforms including the new one
-4. `generate_pack.py --all` - build ZIP packs + install and target manifests
-5. Consistency check: verify and the pack agree on counts and on the paths each archive holds
-6. Pack integrity - extract ZIPs and verify hashes per platform mode
-7. `generate_readme.py` - regenerate README and CONTRIBUTING
-8. `generate_site.py` - regenerate the documentation site and its navigation
+- 0 `validate_schemas.py --source-only`, the schema check PR validation runs
+- 1 `generate_db.py`, rebuild `database.json` from `bios/`
+- 1b `provenance_report.py`, dump-catalog coverage from `provenance/`
+- 1c `romset_recipes.py`, which version each arcade archive is
+- 2 `refresh_data_dirs.py`, update data directories
+- 2a `mame_hash_scraper`, MAME BIOS hashes
+- 2a2 `fbneo_hash_scraper`, FBNeo BIOS hashes
+- 2b `check_buildbot_system.py`, with `--check-buildbot` only
+- 2b2 `check_release_assets.py`, release assets against gitignored files
+- 2c `generate_truth.py`, with `--with-truth` or `--with-export`
+- 2d `diff_truth.py`, truth against the scraped platforms
+- 2e `export_native.py`, with `--with-export`
+- 3 `verify.py --all`, verify all platforms including the new one
+- 3a `identity.py --strict`, one content under two identities
+- 3b `slots.py`, destinations both layers claim
+- 4 `generate_pack.py --all`, build the ZIP packs
+- 4b `generate_pack.py --manifest`, install manifests
+- 4c `generate_pack.py --manifest-targets`, target manifests
+- 4d `bundled_only.py --strict`, copies only an emulator install provides
+- 5 Consistency check: verify and the pack agree on counts and on the paths each archive holds
+- 6 Pack integrity: extract the ZIPs and check hashes per platform mode
+- 7 `generate_readme.py`, regenerate README and CONTRIBUTING
+- 8 `generate_site.py`, regenerate the site pages and navigation
+- 8b `npm run build` in `website/`, then `validate_site.py`
+
+`--offline` skips steps 2 to 2b2.
 
 A new platform is archived-by-default nowhere: `status: active` puts it in
 `list_platforms.py`, the CI matrix, and the scrapers. Set

wiki/adding-a-scraper.md

@@ -44,6 +44,7 @@ for finder, name, ispkg in pkgutil.iter_modules([package_dir]):
 ```
 BaseScraper
   __init__(url: str)
+  lists_files: bool = True          # False: the platform lists no file of its own
   _fetch_raw() -> str               # HTTP GET, cached, 50 MB limit
   fetch_requirements() -> list       # abstract: parse upstream data
   validate_format(raw_data) -> bool  # abstract: detect format changes
@@ -51,10 +52,11 @@ BaseScraper
   test_connection() -> bool          # reachability check
 ```
 
-`_fetch_raw()` handles HTTP with `urllib.request`, sets a `retrobios-scraper/1.0`
-user-agent, enforces a 30-second timeout, and reads the response in 64 KB chunks
-with a 50 MB hard limit to prevent memory exhaustion. The result is cached on the
-instance after the first call.
+`_fetch_raw()` goes through `fetch_bytes()`: `urllib.request` with a
+`retrobios-scraper/1.0` user-agent and a 30-second timeout, the response read in
+64 KB chunks with a 50 MB hard limit. A 429, 502, 503 or 504 answer is retried
+up to five attempts, after the server's `Retry-After` or a doubling wait. The
+result is cached on the instance after the first call.
 
 ### BiosRequirement
 
@@ -73,8 +75,16 @@ class BiosRequirement:
     required: bool = True
     zipped_file: str | None = None   # ROM name inside a ZIP
     native_id: str | None = None     # original system name
+    sha256: str | None = None
+    alt_md5: str | None = None       # a second accepted MD5
+    native_path: str | None = None   # the reference as the platform writes it
+    mirror_of: str | None = None     # the native system this entry repeats
+    native: dict[str, object] = field(default_factory=dict)  # fields written back unchanged
 ```
 
+[BiosRequirement fields](adding-a-platform.md#biosrequirement-fields) says what
+each one carries.
+
 ### ChangeSet
 
 Returned by `compare_with_config()`. Contains:
@@ -95,10 +105,12 @@ Returned by `compare_with_config()`. Contains:
 | `--json` | Output all requirements as JSON |
 | `--output FILE` | Write platform YAML to FILE |
 
-When `--output` targets an existing file, the CLI preserves keys not generated by
-the scraper (e.g. `data_directories`, manual additions). Only the `systems` section
-is replaced. If the scraper defines `generate_platform_yaml()`, that method is used
-instead of the generic YAML builder.
+When `--output` targets an existing file, every top-level key the scraper writes
+is replaced and the others are kept, as is a system's `data_directories` the new
+scrape does not carry. If the scraper defines `generate_platform_yaml()`, that
+method is used instead of the generic YAML builder. A scrape that returns no entry,
+or a YAML that lists no file, stops with an error instead of writing, unless the
+scraper sets `lists_files = False`.
 
 ### Helper functions
 
@@ -247,6 +259,7 @@ pattern (`*_targets_scraper.py`).
 BaseTargetScraper
   __init__(url: str)
   fetch_targets() -> dict    # abstract: return target data
+  pinned_revision(platform)  # the upstream revision platforms/<platform>.yml was read at
   write_output(data, path)   # write YAML to disk
 ```
 
@@ -280,19 +293,21 @@ pipeline to work correctly.
 
 `platforms/targets/_overrides.yml` provides post-scrape adjustments:
 
-- **aliases**: map alternate target names to canonical ones (e.g. `arm64` -> `aarch64`)
+- **aliases**: other names a target answers to (Batocera's `bcm2711` is also `rpi4`)
 - **add_cores**: cores present on a target but not detected by the scraper
-- **remove_cores**: cores detected by the scraper but not actually functional
+- **remove_cores**: names the scraper reads as cores that emulate nothing
+- **_default**: an entry applied to every target of the platform
 
-Overrides are applied by `load_target_config()` in `common.py` after loading
-the scraped data. The scraper itself does not need to handle overrides.
+`apply_target_overrides()` in `common.py` applies them, read by
+`load_target_config()` and by the target manifests. The scraper itself does not
+handle overrides.
 
 ### Module contract
 
 ```python
 from scripts.scraper.targets import BaseTargetScraper
 
-PLATFORM_NAME = "myplatform_targets"
+PLATFORM_NAME = "myplatform"  # the platform's registry key
 
 class Scraper(BaseTargetScraper):
     def __init__(self):
@@ -319,6 +334,9 @@ Register the target scraper in `_registry.yml`:
 | `batocera_targets` | Config.in + es_systems.yml | Cross-reference kernel config with system definitions |
 | `emudeck_targets` | EmuScripts + RetroArch cores | GitHub API for script availability per OS |
 | `retropie_targets` | scriptmodules + rp_module_flags | Parse Bash scriptmodules for platform flags |
+| `lakka_targets` | Lakka-LibreELEC `package.mk` and project options | Core list minus each project and device's exclusions |
+| `recalbox_targets` | board defconfigs + `system.ini` | Buildroot packages each image enables, cores per system |
+| `rocknix_targets` | emulators `package.mk` + device options | Cores per SoC block and 32-bit support per device |
 
 ## Parser modules
 
@@ -360,7 +378,7 @@ since neither can be fetched without a browser). Both write a snapshot to
 
 Parses MAME C source files to extract BIOS root sets. Handles:
 
-- Machine declaration macros: `GAME`, `SYST`, `COMP`, `CONS`
+- Machine declaration macros: `GAME`, `GAMEL`, `SYST`, `COMP`, `CONS`
 - `MACHINE_IS_BIOS_ROOT` flag detection
 - `ROM_START`/`ROM_END` blocks
 - `ROM_LOAD` variants and `ROM_REGION` declarations
@@ -442,6 +460,9 @@ be missing from the round trip.
 # Scrape upstream -> platform YAML
 python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
 
+# Build the truth model the export reads (refused when out of date)
+python scripts/generate_truth.py --platform myplatform --output-dir dist/truth
+
 # Rewrite the platform's own file with the changes applied
 python scripts/export_native.py --platform myplatform --fetch \
     --output-dir tmp/native
@@ -461,10 +482,10 @@ the fields listed in [BiosRequirement fields](adding-a-platform.md#biosrequireme
 | Symptom | Cause | Fix |
 |---------|-------|-----|
 | Unmapped slug warnings | New system added upstream | Add mapping to `SLUG_MAP` |
-| Empty requirements list | Upstream format changed | Check `validate_format()`, update parser |
+| `upstream returned no BIOS entry` | Upstream format changed | Check `validate_format()`, update parser |
 | Hash mismatch in verify | Upstream updated hashes | Re-scrape and regenerate platform YAML |
 | Scraper hangs | URL unreachable, no timeout | `_fetch_raw()` has 30s timeout, check URL |
-| `Response exceeds 50 MB` | Upstream file grew | Investigate, may need chunked parsing |
+| `Response exceeds 52428800 byte limit` | Upstream file grew | Investigate, may need chunked parsing |
 | `validate_format` fails | Upstream restructured | Update both `validate_format` and `fetch_requirements` |
 
 ### E2E tests