docs: scraper and target contracts from the code

This commit is contained in:
Abdessamad Derraz committed 2026-11-03 10:09:32 +01:00
1 parent 8609810d8f
commit b2e313d5f1
2 files changed
+73 -32

No files matched your search

+37 -16
View File
@@ -44,6 +44,7 @@ for finder, name, ispkg in pkgutil.iter_modules([package_dir]):
```
BaseScraper
__init__(url: str)
lists_files: bool = True # False: the platform lists no file of its own
_fetch_raw() -> str # HTTP GET, cached, 50 MB limit
fetch_requirements() -> list # abstract: parse upstream data
validate_format(raw_data) -> bool # abstract: detect format changes
@@ -51,10 +52,11 @@ BaseScraper
test_connection() -> bool # reachability check
```
`_fetch_raw()` handles HTTP with `urllib.request`, sets a `retrobios-scraper/1.0`
user-agent, enforces a 30-second timeout, and reads the response in 64 KB chunks
with a 50 MB hard limit to prevent memory exhaustion. The result is cached on the
instance after the first call.
`_fetch_raw()` goes through `fetch_bytes()`: `urllib.request` with a
`retrobios-scraper/1.0` user-agent and a 30-second timeout, the response read in
64 KB chunks with a 50 MB hard limit. A 429, 502, 503 or 504 answer is retried
up to five attempts, after the server's `Retry-After` or a doubling wait. The
result is cached on the instance after the first call.
### BiosRequirement
@@ -73,8 +75,16 @@ class BiosRequirement:
required: bool = True
zipped_file: str | None = None # ROM name inside a ZIP
native_id: str | None = None # original system name
sha256: str | None = None
alt_md5: str | None = None # a second accepted MD5
native_path: str | None = None # the reference as the platform writes it
mirror_of: str | None = None # the native system this entry repeats
native: dict[str, object] = field(default_factory=dict) # fields written back unchanged
```
[BiosRequirement fields](adding-a-platform.md#biosrequirement-fields) says what
each one carries.
### ChangeSet
Returned by `compare_with_config()`. Contains:
@@ -95,10 +105,12 @@ Returned by `compare_with_config()`. Contains:
| `--json` | Output all requirements as JSON |
| `--output FILE` | Write platform YAML to FILE |
When `--output` targets an existing file, the CLI preserves keys not generated by
the scraper (e.g. `data_directories`, manual additions). Only the `systems` section
is replaced. If the scraper defines `generate_platform_yaml()`, that method is used
instead of the generic YAML builder.
When `--output` targets an existing file, every top-level key the scraper writes
is replaced and the others are kept, as is a system's `data_directories` the new
scrape does not carry. If the scraper defines `generate_platform_yaml()`, that
method is used instead of the generic YAML builder. A scrape that returns no entry,
or a YAML that lists no file, stops with an error instead of writing, unless the
scraper sets `lists_files = False`.
### Helper functions
@@ -247,6 +259,7 @@ pattern (`*_targets_scraper.py`).
BaseTargetScraper
__init__(url: str)
fetch_targets() -> dict # abstract: return target data
pinned_revision(platform) # the upstream revision platforms/<platform>.yml was read at
write_output(data, path) # write YAML to disk
```
@@ -280,19 +293,21 @@ pipeline to work correctly.
`platforms/targets/_overrides.yml` provides post-scrape adjustments:
- **aliases**: map alternate target names to canonical ones (e.g. `arm64` -> `aarch64`)
- **aliases**: other names a target answers to (Batocera's `bcm2711` is also `rpi4`)
- **add_cores**: cores present on a target but not detected by the scraper
- **remove_cores**: cores detected by the scraper but not actually functional
- **remove_cores**: names the scraper reads as cores that emulate nothing
- **_default**: an entry applied to every target of the platform
Overrides are applied by `load_target_config()` in `common.py` after loading
the scraped data. The scraper itself does not need to handle overrides.
`apply_target_overrides()` in `common.py` applies them, read by
`load_target_config()` and by the target manifests. The scraper itself does not
handle overrides.
### Module contract
```python
from scripts.scraper.targets import BaseTargetScraper
PLATFORM_NAME = "myplatform_targets"
PLATFORM_NAME = "myplatform" # the platform's registry key
class Scraper(BaseTargetScraper):
def __init__(self):
@@ -319,6 +334,9 @@ Register the target scraper in `_registry.yml`:
| `batocera_targets` | Config.in + es_systems.yml | Cross-reference kernel config with system definitions |
| `emudeck_targets` | EmuScripts + RetroArch cores | GitHub API for script availability per OS |
| `retropie_targets` | scriptmodules + rp_module_flags | Parse Bash scriptmodules for platform flags |
| `lakka_targets` | Lakka-LibreELEC `package.mk` and project options | Core list minus each project and device's exclusions |
| `recalbox_targets` | board defconfigs + `system.ini` | Buildroot packages each image enables, cores per system |
| `rocknix_targets` | emulators `package.mk` + device options | Cores per SoC block and 32-bit support per device |
## Parser modules
@@ -360,7 +378,7 @@ since neither can be fetched without a browser). Both write a snapshot to
Parses MAME C source files to extract BIOS root sets. Handles:
- Machine declaration macros: `GAME`, `SYST`, `COMP`, `CONS`
- Machine declaration macros: `GAME`, `GAMEL`, `SYST`, `COMP`, `CONS`
- `MACHINE_IS_BIOS_ROOT` flag detection
- `ROM_START`/`ROM_END` blocks
- `ROM_LOAD` variants and `ROM_REGION` declarations
@@ -442,6 +460,9 @@ be missing from the round trip.
# Scrape upstream -> platform YAML
python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
# Build the truth model the export reads (refused when out of date)
python scripts/generate_truth.py --platform myplatform --output-dir dist/truth
# Rewrite the platform's own file with the changes applied
python scripts/export_native.py --platform myplatform --fetch \
--output-dir tmp/native
@@ -461,10 +482,10 @@ the fields listed in [BiosRequirement fields](adding-a-platform.md#biosrequireme
| Symptom | Cause | Fix |
|---------|-------|-----|
| Unmapped slug warnings | New system added upstream | Add mapping to `SLUG_MAP` |
| Empty requirements list | Upstream format changed | Check `validate_format()`, update parser |
| `upstream returned no BIOS entry` | Upstream format changed | Check `validate_format()`, update parser |
| Hash mismatch in verify | Upstream updated hashes | Re-scrape and regenerate platform YAML |
| Scraper hangs | URL unreachable, no timeout | `_fetch_raw()` has 30s timeout, check URL |
| `Response exceeds 50 MB` | Upstream file grew | Investigate, may need chunked parsing |
| `Response exceeds 52428800 byte limit` | Upstream file grew | Investigate, may need chunked parsing |
| `validate_format` fails | Upstream restructured | Update both `validate_format` and `fetch_requirements` |
### E2E tests