mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-11-17 07:07:52 -06:00
docs: scraper and target contracts from the code
This commit is contained in:
1 parent
8609810d8f
commit
b2e313d5f1
2 files changed
+73
-32
No files matched your search
+37
-16
@@ -44,6 +44,7 @@ for finder, name, ispkg in pkgutil.iter_modules([package_dir]):
|
||||
```
|
||||
BaseScraper
|
||||
__init__(url: str)
|
||||
lists_files: bool = True # False: the platform lists no file of its own
|
||||
_fetch_raw() -> str # HTTP GET, cached, 50 MB limit
|
||||
fetch_requirements() -> list # abstract: parse upstream data
|
||||
validate_format(raw_data) -> bool # abstract: detect format changes
|
||||
@@ -51,10 +52,11 @@ BaseScraper
|
||||
test_connection() -> bool # reachability check
|
||||
```
|
||||
|
||||
`_fetch_raw()` handles HTTP with `urllib.request`, sets a `retrobios-scraper/1.0`
|
||||
user-agent, enforces a 30-second timeout, and reads the response in 64 KB chunks
|
||||
with a 50 MB hard limit to prevent memory exhaustion. The result is cached on the
|
||||
instance after the first call.
|
||||
`_fetch_raw()` goes through `fetch_bytes()`: `urllib.request` with a
|
||||
`retrobios-scraper/1.0` user-agent and a 30-second timeout, the response read in
|
||||
64 KB chunks with a 50 MB hard limit. A 429, 502, 503 or 504 answer is retried
|
||||
up to five attempts, after the server's `Retry-After` or a doubling wait. The
|
||||
result is cached on the instance after the first call.
|
||||
|
||||
### BiosRequirement
|
||||
|
||||
@@ -73,8 +75,16 @@ class BiosRequirement:
|
||||
required: bool = True
|
||||
zipped_file: str | None = None # ROM name inside a ZIP
|
||||
native_id: str | None = None # original system name
|
||||
sha256: str | None = None
|
||||
alt_md5: str | None = None # a second accepted MD5
|
||||
native_path: str | None = None # the reference as the platform writes it
|
||||
mirror_of: str | None = None # the native system this entry repeats
|
||||
native: dict[str, object] = field(default_factory=dict) # fields written back unchanged
|
||||
```
|
||||
|
||||
[BiosRequirement fields](adding-a-platform.md#biosrequirement-fields) says what
|
||||
each one carries.
|
||||
|
||||
### ChangeSet
|
||||
|
||||
Returned by `compare_with_config()`. Contains:
|
||||
@@ -95,10 +105,12 @@ Returned by `compare_with_config()`. Contains:
|
||||
| `--json` | Output all requirements as JSON |
|
||||
| `--output FILE` | Write platform YAML to FILE |
|
||||
|
||||
When `--output` targets an existing file, the CLI preserves keys not generated by
|
||||
the scraper (e.g. `data_directories`, manual additions). Only the `systems` section
|
||||
is replaced. If the scraper defines `generate_platform_yaml()`, that method is used
|
||||
instead of the generic YAML builder.
|
||||
When `--output` targets an existing file, every top-level key the scraper writes
|
||||
is replaced and the others are kept, as is a system's `data_directories` the new
|
||||
scrape does not carry. If the scraper defines `generate_platform_yaml()`, that
|
||||
method is used instead of the generic YAML builder. A scrape that returns no entry,
|
||||
or a YAML that lists no file, stops with an error instead of writing, unless the
|
||||
scraper sets `lists_files = False`.
|
||||
|
||||
### Helper functions
|
||||
|
||||
@@ -247,6 +259,7 @@ pattern (`*_targets_scraper.py`).
|
||||
BaseTargetScraper
|
||||
__init__(url: str)
|
||||
fetch_targets() -> dict # abstract: return target data
|
||||
pinned_revision(platform) # the upstream revision platforms/<platform>.yml was read at
|
||||
write_output(data, path) # write YAML to disk
|
||||
```
|
||||
|
||||
@@ -280,19 +293,21 @@ pipeline to work correctly.
|
||||
|
||||
`platforms/targets/_overrides.yml` provides post-scrape adjustments:
|
||||
|
||||
- **aliases**: map alternate target names to canonical ones (e.g. `arm64` -> `aarch64`)
|
||||
- **aliases**: other names a target answers to (Batocera's `bcm2711` is also `rpi4`)
|
||||
- **add_cores**: cores present on a target but not detected by the scraper
|
||||
- **remove_cores**: cores detected by the scraper but not actually functional
|
||||
- **remove_cores**: names the scraper reads as cores that emulate nothing
|
||||
- **_default**: an entry applied to every target of the platform
|
||||
|
||||
Overrides are applied by `load_target_config()` in `common.py` after loading
|
||||
the scraped data. The scraper itself does not need to handle overrides.
|
||||
`apply_target_overrides()` in `common.py` applies them, read by
|
||||
`load_target_config()` and by the target manifests. The scraper itself does not
|
||||
handle overrides.
|
||||
|
||||
### Module contract
|
||||
|
||||
```python
|
||||
from scripts.scraper.targets import BaseTargetScraper
|
||||
|
||||
PLATFORM_NAME = "myplatform_targets"
|
||||
PLATFORM_NAME = "myplatform" # the platform's registry key
|
||||
|
||||
class Scraper(BaseTargetScraper):
|
||||
def __init__(self):
|
||||
@@ -319,6 +334,9 @@ Register the target scraper in `_registry.yml`:
|
||||
| `batocera_targets` | Config.in + es_systems.yml | Cross-reference kernel config with system definitions |
|
||||
| `emudeck_targets` | EmuScripts + RetroArch cores | GitHub API for script availability per OS |
|
||||
| `retropie_targets` | scriptmodules + rp_module_flags | Parse Bash scriptmodules for platform flags |
|
||||
| `lakka_targets` | Lakka-LibreELEC `package.mk` and project options | Core list minus each project and device's exclusions |
|
||||
| `recalbox_targets` | board defconfigs + `system.ini` | Buildroot packages each image enables, cores per system |
|
||||
| `rocknix_targets` | emulators `package.mk` + device options | Cores per SoC block and 32-bit support per device |
|
||||
|
||||
## Parser modules
|
||||
|
||||
@@ -360,7 +378,7 @@ since neither can be fetched without a browser). Both write a snapshot to
|
||||
|
||||
Parses MAME C source files to extract BIOS root sets. Handles:
|
||||
|
||||
- Machine declaration macros: `GAME`, `SYST`, `COMP`, `CONS`
|
||||
- Machine declaration macros: `GAME`, `GAMEL`, `SYST`, `COMP`, `CONS`
|
||||
- `MACHINE_IS_BIOS_ROOT` flag detection
|
||||
- `ROM_START`/`ROM_END` blocks
|
||||
- `ROM_LOAD` variants and `ROM_REGION` declarations
|
||||
@@ -442,6 +460,9 @@ be missing from the round trip.
|
||||
# Scrape upstream -> platform YAML
|
||||
python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
|
||||
|
||||
# Build the truth model the export reads (refused when out of date)
|
||||
python scripts/generate_truth.py --platform myplatform --output-dir dist/truth
|
||||
|
||||
# Rewrite the platform's own file with the changes applied
|
||||
python scripts/export_native.py --platform myplatform --fetch \
|
||||
--output-dir tmp/native
|
||||
@@ -461,10 +482,10 @@ the fields listed in [BiosRequirement fields](adding-a-platform.md#biosrequireme
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| Unmapped slug warnings | New system added upstream | Add mapping to `SLUG_MAP` |
|
||||
| Empty requirements list | Upstream format changed | Check `validate_format()`, update parser |
|
||||
| `upstream returned no BIOS entry` | Upstream format changed | Check `validate_format()`, update parser |
|
||||
| Hash mismatch in verify | Upstream updated hashes | Re-scrape and regenerate platform YAML |
|
||||
| Scraper hangs | URL unreachable, no timeout | `_fetch_raw()` has 30s timeout, check URL |
|
||||
| `Response exceeds 50 MB` | Upstream file grew | Investigate, may need chunked parsing |
|
||||
| `Response exceeds 52428800 byte limit` | Upstream file grew | Investigate, may need chunked parsing |
|
||||
| `validate_format` fails | Upstream restructured | Update both `validate_format` and `fetch_requirements` |
|
||||
|
||||
### E2E tests
|
||||
|
||||
Reference in new issue
Block a user