mirror of
https://github.com/Abdess/retroarch_system.git
synced 2026-11-17 23:27:49 -06:00
chore: merge tooling batch
This commit is contained in:
commit
b90f8bb0b9
116 files changed
+12868
-9741
No files matched your search
@@ -332,6 +332,7 @@ Register the target scraper in `_registry.yml`:
|
||||
|---------|--------|----------|
|
||||
| `retroarch_targets` | libretro buildbot nightly | Scrape directory listings for each target arch |
|
||||
| `batocera_targets` | Config.in + es_systems.yml | Cross-reference kernel config with system definitions |
|
||||
| `knulli_targets` | KNULLI's own Config.in + es_systems.yml | The Batocera scraper's Kconfig resolution against KNULLI's tree |
|
||||
| `emudeck_targets` | EmuScripts + RetroArch cores | GitHub API for script availability per OS |
|
||||
| `retropie_targets` | scriptmodules + rp_module_flags | Parse Bash scriptmodules for platform flags |
|
||||
| `lakka_targets` | Lakka-LibreELEC `package.mk` and project options | Core list minus each project and device's exclusions |
|
||||
|
||||
@@ -64,7 +64,7 @@ Every module in `tests/`, by the area it guards.
|
||||
| Pack probe | the cores a platform runs for a system, staging grown to a fixed point, runs mapped to pack and manifest entries, the line and its schema, the probe lock per run, that no page shows its lines. One class builds a real pack narrowed to PlayStation and lays it out both ways, from the archive and with `install.py --dest`, and skips without the collection | `test_pack_probe` |
|
||||
| Automation runner, disk cleanup, asset sync and bit-flip repair | fake jobs, git repositories with a bare origin and session worktrees, a committed file with one bit flipped, canned CI runs in place of `gh` | `test_autorun` |
|
||||
| Truth and native export | synthetic + cached originals | `test_export_counts`, `test_export_native_layer`, `test_export_report`, `test_exporters`, `test_truth_data_dir_readers`, `test_truth_diff`, `test_truth_layout` |
|
||||
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_availability`, `test_site_bootstrap`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_density_cards`, `test_site_emulator_page`, `test_site_entry_tables`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_picker`, `test_site_platform_density`, `test_site_platform_hashes`, `test_site_r2_components`, `test_site_r2_css`, `test_site_r2_emulators`, `test_site_r2_entry`, `test_site_r2_gate`, `test_site_r2_nav`, `test_site_r2_p0`, `test_site_r2_reference`, `test_site_r2_systems`, `test_site_r2_wiki`, `test_site_r3_collection`, `test_site_r3_components`, `test_site_r3_css`, `test_site_r3_emulators`, `test_site_r3_entry`, `test_site_r3_nav`, `test_site_r3_p0`, `test_site_r3_reference`, `test_site_r3_systems`, `test_site_r3_wiki`, `test_site_r4_components`, `test_site_r4_css`, `test_site_r4_emulators`, `test_site_r4_entry`, `test_site_r4_merge`, `test_site_r4_p0`, `test_site_r4_reference`, `test_site_r4_search`, `test_site_r4_systems`, `test_site_r4_wiki`, `test_site_r5_components`, `test_site_r5_css`, `test_site_r5_emulators`, `test_site_r5_entry`, `test_site_r5_reference`, `test_site_r5_search`, `test_site_r5_systems`, `test_site_r5_wiki`, `test_site_render`, `test_site_system_links`, `test_site_validation`, `test_site_weight`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_version_basis`, `test_workflow_paths` |
|
||||
| Scrapers, upstream and provenance | inline sources, synthetic | `test_batocera_md5_list`, `test_check_freshness`, `test_contributing_text`, `test_coverage_cache`, `test_data_dir_refs`, `test_dataclass_defaults`, `test_dedup`, `test_doc_pages`, `test_doc_surfaces`, `test_elec_scrapers`, `test_fbneo_parser`, `test_find_pinned_revision`, `test_hash_merge`, `test_launchers`, `test_launchers_emudeck`, `test_launchers_recalbox`, `test_launchers_retrobat`, `test_launchers_retrodeck`, `test_launchers_retropie`, `test_launchers_rocknix`, `test_mame_devices`, `test_mame_parser`, `test_mame_ref_audit`, `test_optional_dependencies`, `test_profile_sync`, `test_prose_style`, `test_provenance`, `test_refresh_data_dirs`, `test_refresh_failures`, `test_refresh_stale`, `test_retrodeck_scraper_errors`, `test_romset_recipes`, `test_schema_semantic_gate`, `test_scraper_contract`, `test_site_arcade_boards`, `test_site_availability`, `test_site_bootstrap`, `test_site_built_gate`, `test_site_components`, `test_site_console_folders`, `test_site_content_data`, `test_site_copy`, `test_site_cores`, `test_site_css`, `test_site_density_cards`, `test_site_emulator_page`, `test_site_emulators`, `test_site_entry`, `test_site_entry_tables`, `test_site_exports`, `test_site_funnel`, `test_site_images`, `test_site_integration`, `test_site_nav`, `test_site_picker`, `test_site_platform_density`, `test_site_platform_hashes`, `test_site_reference`, `test_site_render`, `test_site_search`, `test_site_system_links`, `test_site_system_names`, `test_site_system_sections`, `test_site_systems`, `test_site_validation`, `test_site_weight`, `test_site_wiki`, `test_site_write`, `test_source_ref_targets`, `test_upstream`, `test_upstream_index`, `test_validate_pr`, `test_validate_pr_flow`, `test_validate_pr_hashes`, `test_version_basis`, `test_workflow_paths` |
|
||||
| Games on the card | small DATs, listxml and setup sources, a fake fetcher, fixture install lists, and the committed data against its schemas | `test_arcade_parents`, `test_game_catalogs`, `test_game_headers`, `test_game_regions`, `test_games_index`, `test_romfolders_card`, `test_romfolders_es` |
|
||||
|
||||
## Test architecture
|
||||
|
||||
+22
-4
@@ -24,6 +24,10 @@ python scripts/pipeline.py --offline --include-archived # include archived plat
|
||||
python scripts/pipeline.py --check-buildbot # online run + buildbot freshness check
|
||||
```
|
||||
|
||||
`--offline` skips the data directory refresh, so on a fresh clone run
|
||||
`python scripts/refresh_data_dirs.py` once first. A run that builds packs with no
|
||||
data directory cached is refused before step 0.
|
||||
|
||||
Pipeline steps:
|
||||
|
||||
| Step | Description | Skipped by |
|
||||
@@ -671,14 +675,23 @@ profile, `--record-coreinfo <core>` records its blob as reviewed. Data
|
||||
directories are checked against their upstream, dump catalogs and romset
|
||||
snapshots against the catalog they were read from (Redump DAT versions, the
|
||||
No-Intro daily rebuild, the newest TOSEC pack, the newest MAME release, the
|
||||
git blobs of the FBNeo DATs), and the CI toolchain against PyPI and the
|
||||
actions' latest releases.
|
||||
git blobs of the FBNeo DATs), the game catalogs of `games/catalogs/` against
|
||||
the No-Intro pack and the Redump datfiles they were read from, and the CI
|
||||
toolchain against PyPI and the actions' latest releases.
|
||||
|
||||
The `native` area needs no network. It reads the cache `export_native.py`
|
||||
patches from and reports an original that is absent, that came from another
|
||||
URL than the one the platform YAML names, or that was fetched before the
|
||||
platform YAML was last rewritten.
|
||||
|
||||
The `romfolders` area compares each pin of `platforms/romfolders/_sources/`.
|
||||
A setup that is a registered platform must be read at the revision its
|
||||
platform file was scraped at, which needs no network unless one side names a
|
||||
tag and the other a commit. Any other setup is compared with its project's
|
||||
newest release, or with its branch head when the project publishes none. A
|
||||
stale pin is moved by hand, then `python -m scripts.scraper.romfolders --setup
|
||||
NAME` reads the folders again.
|
||||
|
||||
The `profiles` area always checks `website/src/data/source-paths.json`
|
||||
against the profiles, offline: a recorded tree that no profile pins any more
|
||||
is `STALE` until `sitesources.py --fetch` runs again.
|
||||
@@ -778,7 +791,7 @@ until a report without one. Exit codes: 0 for `ok`,
|
||||
|
||||
| Job | Schedule | What it does |
|
||||
|-----|----------|--------------|
|
||||
| `nightly` | 02:00 | `check_freshness.py --json` saved to a file, `refresh_stale.py --findings` on its `data` and `catalogs` rows, the MAME and FBNeo hash scrapers, then `pipeline.py --offline` as the gate. `database.json` and `install/*.json` are computed a second time and compared. Commits the hash profiles, the recipes and the regenerated records |
|
||||
| `nightly` | 02:00 | `check_freshness.py --json` saved to a file, `refresh_stale.py --findings` on its `data` and `catalogs` rows, the MAME and FBNeo hash scrapers, then `pipeline.py --offline` as the gate. `database.json` and `install/*.json` are computed a second time and compared. Commits the hash profiles, the recipes, the game catalogs and the regenerated records |
|
||||
| `assets` | 04:30 | `sync_release_assets.py`, as a dry run |
|
||||
| `bios-integrity` | Sunday 08:30 | `bios_integrity.py --apply --json` in the main checkout, once the host is idle (up to four hours of waiting). Its report is kept in the job's status |
|
||||
| `cleanup` | 05:30 | `cleanup_disk.py --apply --hours 72` in the main checkout |
|
||||
@@ -1351,7 +1364,7 @@ Further options, by script:
|
||||
| `romset_recipes.py` | `--dat-recipes PATH`, a snapshot or a directory of them (default `recipes/`) |
|
||||
| `split_pack.py` | `--max-size 1900M`, the ceiling per part (K, M, G, default 2G) |
|
||||
| `torrentzip.py` | `--check ZIP` reports conformance, `--rebuild ZIP -o OUT` normalizes an archive |
|
||||
| `validate_pr.py` | `--changed` takes the BIOS files the branch changes, `--markdown`, `--json` |
|
||||
| `validate_pr.py` | `--changed` takes the BIOS files the branch changes, a folder named checks every file under it, `--markdown`, `--json` |
|
||||
| `validate_site.py` | `--site-dir` (rendered site, default `website/dist`), `--base` (path the site is served under), `--max-errors N` |
|
||||
| `scraper/coreinfo_scraper.py` | `--compare-db database.json` lists the `.info` firmware the database lacks |
|
||||
| `scraper/mame_hash_scraper.py` | `--release mameNNNN --profile FILE --source-commit SHA --core NAME --built-by PLATFORM` writes the profile of a MAME build a platform compiles. `--devices --listxml FILE` documents in `mame.yml` every device set with dumped ROMs that the release's `-listxml` names (`isdevice="yes"`), with `--dry-run` to preview |
|
||||
@@ -1642,6 +1655,10 @@ keeps, is computed from `install/<platform>.json` with the installer's own
|
||||
core narrowing. Every published file is listed in `install/games/index.json`
|
||||
with its size, its gzip size and its sha256.
|
||||
|
||||
`validate_schemas.py` refuses a `games/arcade/<profile>.json` whose source
|
||||
commit or `core_version` is not the one its profile pins: after a profile's pin
|
||||
moves, rerun `arcade_parent_importer --profile NAME`.
|
||||
|
||||
## Target scrapers
|
||||
|
||||
Located in `scripts/scraper/targets/`. Each inherits `BaseTargetScraper` and implements `fetch_targets()`.
|
||||
@@ -1650,6 +1667,7 @@ Located in `scripts/scraper/targets/`. Each inherits `BaseTargetScraper` and imp
|
||||
|---------|--------|----------------|
|
||||
| `retroarch_targets_scraper` | libretro buildbot nightly listings | buildbot architecture |
|
||||
| `batocera_targets_scraper` | Config.in + es_systems.yml + Buildroot Kconfig | board |
|
||||
| `knulli_targets_scraper` | the same, read from KNULLI's tree and Buildroot fork | board |
|
||||
| `recalbox_targets_scraper` | board defconfigs + system.ini cores | board |
|
||||
| `rocknix_targets_scraper` | emulators package.mk + device options | device |
|
||||
| `lakka_targets_scraper` | libretro_cores package.mk + project and device options | device image |
|
||||
|
||||
Reference in new issue
Block a user