mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-10-10 13:33:24 -05:00
feat: make the native export round-trip a platform file
This commit is contained in:
Abdessamad Derraz committed
1 parent
7b93285e1d
commit
691ccbfca7
65 files changed
+18417
-2115
No files matched your search
@@ -276,17 +276,20 @@ class Exporter(BaseExporter):
|
||||
|
||||
### Round-trip validation
|
||||
|
||||
The exporter enables a scrape-export-compare workflow:
|
||||
|
||||
```bash
|
||||
# Scrape upstream
|
||||
python -m scripts.scraper.myplatform_scraper --output /tmp/scraped.yml
|
||||
# Export truth data
|
||||
python scripts/export_native.py --platform myplatform --output /tmp/exported.json
|
||||
# Compare exported file with upstream
|
||||
diff /tmp/scraped.yml /tmp/exported.json
|
||||
python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
|
||||
python scripts/export_native.py --platform myplatform --fetch --output-dir tmp/native
|
||||
diff .cache/upstream-native/myplatform/<their file> \
|
||||
tmp/native/myplatform/<their file>
|
||||
```
|
||||
|
||||
The export must not lose anything the platform already declares. If your
|
||||
format carries code as well as data - a script, a C# database, a manifest
|
||||
whose BIOS list sits beside launch configuration - say so with
|
||||
`needs_original()` and patch their file rather than writing a new one. An
|
||||
exporter that regenerates such a file from BIOS data alone hands the
|
||||
maintainer something that no longer runs.
|
||||
|
||||
## Step 6: Create a target scraper (optional)
|
||||
|
||||
Target scrapers determine which emulator cores are available on each hardware
|
||||
|
||||
@@ -403,20 +403,29 @@ manipulation rather than load-dump to maintain human-readable formatting.
|
||||
|
||||
### Round-trip testing
|
||||
|
||||
If an exporter exists for the platform, validate the scrape-export-compare cycle:
|
||||
The scrape and the export are two directions of the same transcription, so
|
||||
run them against each other: scrape upstream, export it back, and diff the
|
||||
result against the file you started from. Nothing the platform declares may
|
||||
be missing from the round trip.
|
||||
|
||||
```bash
|
||||
# Scrape upstream -> platform YAML
|
||||
python -m scripts.scraper.myplatform_scraper --output /tmp/scraped.yml
|
||||
python -m scripts.scraper.myplatform_scraper --output platforms/myplatform.yml
|
||||
|
||||
# Export truth data -> native format
|
||||
python scripts/export_native.py --platform myplatform --output /tmp/exported.json
|
||||
# Rewrite the platform's own file, corrected
|
||||
python scripts/export_native.py --platform myplatform --fetch \
|
||||
--output-dir tmp/native
|
||||
|
||||
# Compare
|
||||
diff <(python -m scripts.scraper.myplatform_scraper --json | python -m json.tool) \
|
||||
/tmp/exported.json
|
||||
# Compare with the file upstream publishes, cached by --fetch
|
||||
diff .cache/upstream-native/myplatform/<their file> \
|
||||
tmp/native/myplatform/<their file>
|
||||
```
|
||||
|
||||
Every line in that diff should be a correction you can defend. A line that
|
||||
merely reshapes their formatting means the scraper lost something the
|
||||
exporter then had to invent: add it to the requirement instead, through
|
||||
`native_path`, `native_system` or the `native` dict.
|
||||
|
||||
### Common issues
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|
||||
+23
-5
@@ -324,15 +324,33 @@ The diff reports:
|
||||
|
||||
### Export to native formats
|
||||
|
||||
Convert truth data to the native format each platform consumes:
|
||||
Rewrite a platform's own BIOS file, corrected. The output is the file its
|
||||
maintainers keep, not our data in their syntax: hashes the truth can prove
|
||||
are applied, entries the truth says nothing about are left alone, and files
|
||||
the truth knows and the platform lacks are added.
|
||||
|
||||
```bash
|
||||
python scripts/export_native.py --platform batocera # Python dict (batocera-systems)
|
||||
python scripts/export_native.py --platform recalbox # XML (es_bios.xml)
|
||||
python scripts/export_native.py --all --output-dir dist/upstream/
|
||||
python scripts/export_native.py --platform batocera --fetch # batocera-systems
|
||||
python scripts/export_native.py --platform recalbox --fetch # es_bios.xml
|
||||
python scripts/export_native.py --all --fetch --output-dir dist/upstream/
|
||||
```
|
||||
|
||||
This allows submitting corrections upstream in the format maintainers expect.
|
||||
Seven of the formats carry code as well as data - Batocera's and ROCKNIX's
|
||||
scripts, EmuDeck's shell library, BizHawk's C# database, RetroDECK's
|
||||
component manifests, MiSTer's BiosDB, RetroPie's scriptmodules. Those are
|
||||
patched from the platform's own file, which `--fetch` downloads once into
|
||||
`.cache/upstream-native/`. Without it the export fails rather than
|
||||
publishing a fragment.
|
||||
|
||||
RetroPie is the odd one: it publishes no BIOS list at all, and the only
|
||||
declaration is the sentence in each package's `rp_module_help`. The export
|
||||
adds a missing file name to a list a maintainer already wrote, never
|
||||
removes one, and never drafts a sentence where there was none.
|
||||
|
||||
The run reports what it changed, and what it had to leave out: an entry a
|
||||
format cannot express is named, never written half-formed. `es_bios.xsd`
|
||||
makes md5 and core required on every element, and RomM compares the file
|
||||
size before any hash, so an entry missing either could never verify.
|
||||
|
||||
|
||||
## Emulator-Level Verification
|
||||
|
||||
@@ -55,8 +55,8 @@ emulators/*.yml verify.py checks generate_pack.py resolves
|
||||
from code verification packs per platform
|
||||
|
||||
truth.py generates diff_truth.py export_native.py
|
||||
ground truth from compares truth vs exports to native formats
|
||||
emulator profiles scraped platform (DAT, XML, JSON, Bash)
|
||||
ground truth from compares truth vs rewrites each platform's
|
||||
emulator profiles scraped platform own file, corrected
|
||||
```
|
||||
|
||||
Pipeline runs all steps in sequence: DB, provenance report, data dirs,
|
||||
|
||||
+17
-3
@@ -226,13 +226,27 @@ python scripts/diff_truth.py --all # diff all platforms
|
||||
|
||||
### export_native.py
|
||||
|
||||
Export truth data to native platform formats (System.dat, es_bios.xml, checkBIOS.sh, etc.).
|
||||
Rewrite each platform's own BIOS file, corrected: System.dat, es_bios.xml,
|
||||
batocera-systems, checkBIOS.sh, FirmwareDatabase.cs, the RetroDECK manifests,
|
||||
MiSTer's BiosDB, RetroPie's scriptmodules. Every platform in the registry,
|
||||
archived included.
|
||||
|
||||
```bash
|
||||
python scripts/export_native.py --platform batocera
|
||||
python scripts/export_native.py --all --output-dir dist/upstream/
|
||||
python scripts/export_native.py --platform batocera --fetch
|
||||
python scripts/export_native.py --all --fetch --output-dir dist/upstream/
|
||||
```
|
||||
|
||||
`--fetch` downloads the platform's own file once into
|
||||
`.cache/upstream-native/`, at the revision the platform YAML's `source:`
|
||||
names. Formats that carry code are patched from it rather than
|
||||
regenerated, and the export fails without it.
|
||||
|
||||
A list is written in the order the code looks: `priority:` first (lowest
|
||||
wins), then the profile's own declaration order. What a format cannot
|
||||
state is reported rather than written — EmuDeck's arrays are shared
|
||||
between emulators its file does not name, so a hash is corrected in place
|
||||
and never added.
|
||||
|
||||
MAME and FBNeo entries are `.zip` ROM sets: their meaningful hashes belong
|
||||
to the files inside the archive, so the exported DAT lists those entries
|
||||
without a container sha1. Anyone submitting the DAT upstream should mention
|
||||
|
||||
Reference in new issue
Block a user