kmoody/libretro
1
0
Fork 0
mirror of https://github.com/Abdess/retroarch_system.git synced 2026-04-13 12:22:33 -05:00
Code Issues Packages Projects Releases Wiki Activity

docs: sync wiki sources with pipeline changes

Browse Source 
Update wiki source files (the single source of truth for the site):
- tools.md: renumber pipeline steps 1-8, add step 6 (pack integrity),
  add missing CLI flags for cross_reference.py and refresh_data_dirs.py
- architecture.md: update mermaid diagram with pack integrity step,
  fix test file count (5 files, 249 tests)
- testing-guide.md: add test_pack_integrity section, add step 5 to
  verification discipline checklist
This commit is contained in:
Abdessamad Derraz
2026-04-01 13:08:19 +02:00
parent 6eca4c416a
commit 0e6db8abdf
3 changed files with 51 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
wiki/architecture.md
View File

@@ -48,7 +48,7 @@ truth.py generates diff_truth.py export_native.py
Pipeline runs all steps in sequence: DB, data dirs, MAME/FBNeo hashes,
verify, packs, install manifests, target manifests, consistency check,
README, site. See [tools](tools.md) for the full pipeline reference.
pack integrity, README, site. See [tools](tools.md) for the full pipeline reference.
```mermaid
graph LR
@@ -59,12 +59,14 @@ graph LR
E --> F[install manifests]
F --> G[target manifests]
G --> H[consistency check]
H --> I[generate_readme]
H --> H2[pack integrity]
H2 --> I[generate_readme]
I --> J[generate_site]
style A fill:#2d333b,stroke:#adbac7,color:#adbac7
style D fill:#2d333b,stroke:#adbac7,color:#adbac7
style E fill:#2d333b,stroke:#adbac7,color:#adbac7
style H2 fill:#2d333b,stroke:#adbac7,color:#adbac7
style J fill:#2d333b,stroke:#adbac7,color:#adbac7
```
@@ -234,14 +236,15 @@ user's platform, filter files by hardware target, and download with SHA1 verific
## Tests
4 test files with synthetic fixtures:
5 test files, 249 tests total:
| File | Coverage |
|------|----------|
| `test_e2e.py` | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters |
| `test_mame_parser.py` | BIOS root set detection, ROM block parsing, macro expansion |
| `test_fbneo_parser.py` | BIOS set detection, ROM info parsing |
| `test_hash_merge.py` | MAME/FBNeo YAML merge, diff detection |
| File | Tests | Coverage |
|------|-------|----------|
| `test_e2e.py` | 186 | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters |
| `test_pack_integrity.py` | 8 | extract ZIP packs to disk, verify paths + hashes per platform's native mode |
| `test_mame_parser.py` | 22 | BIOS root set detection, ROM block parsing, macro expansion |
| `test_fbneo_parser.py` | 16 | BIOS set detection, ROM info parsing |
| `test_hash_merge.py` | 17 | MAME/FBNeo YAML merge, diff detection |
```bash
python -m unittest tests.test_e2e -v

26
wiki/testing-guide.md
View File

@@ -11,6 +11,7 @@ Run a single test module:
```bash
python -m unittest tests.test_e2e -v
python -m unittest tests.test_pack_integrity -v
python -m unittest tests.test_mame_parser -v
python -m unittest tests.test_fbneo_parser -v
python -m unittest tests.test_hash_merge -v
@@ -119,6 +120,30 @@ def test_42_my_new_behavior(self):
self.assertEqual(result[0]["status"], Status.OK)
```
### test_pack_integrity.py
End-to-end pack verification. Extracts each platform ZIP to `tmp/` (in the
repo, not `/tmp` which is tmpfs on WSL) and verifies that every file declared
in the platform YAML:
1. Exists at the correct path on disk after extraction
2. Has the correct hash per the platform's native verification mode
Handles inner ZIP verification for MAME/FBNeo ROM sets (checkInsideZip,
md5_composite, inner ROM MD5) and path collision deduplication.
8 tests (one per active platform): RetroArch, Batocera, BizHawk, EmuDeck,
Recalbox, RetroBat, RetroDECK, RomM.
```bash
python -m unittest tests.test_pack_integrity -v
# or via CLI:
python scripts/generate_pack.py --all --verify-packs --output-dir dist/
```
Integrated as pipeline step 6/8 (runs after consistency check, before
README generation). Requires packs in `dist/` — skip with `--skip-packs`.
## Verification discipline
The test suite is one layer of verification. The full quality gate is:
@@ -127,6 +152,7 @@ The test suite is one layer of verification. The full quality gate is:
2. The full pipeline completes without error (`python scripts/pipeline.py --offline`)
3. No unexpected CRITICAL entries in the verify output
4. Pack file counts match verification file counts (consistency check)
5. Pack integrity passes (every declared file extractable with correct hash)
If a change passes tests but breaks the pipeline, it's worth investigating before merging. Similarly, new CRITICAL entries in the verify output after a change usually indicate something to look into. The pipeline is designed so that all steps agree: if verify reports N files for a platform, the pack should contain exactly N files.

21
wiki/tools.md
View File

@@ -7,7 +7,7 @@ All tools are Python scripts in `scripts/`. Single dependency: `pyyaml`.
Run everything in sequence:
```bash
python scripts/pipeline.py --offline # DB + verify + packs + manifests + readme + site
python scripts/pipeline.py --offline # DB + verify + packs + manifests + integrity + readme + site
python scripts/pipeline.py --offline --skip-packs # DB + verify only
python scripts/pipeline.py --offline --skip-docs # skip readme + site generation
python scripts/pipeline.py --offline --target switch # filter by hardware target
@@ -20,21 +20,22 @@ Pipeline steps:
| Step | Description | Skipped by |
|------|-------------|------------|
| 1/9 | Generate database | - |
| 2/9 | Refresh data directories | `--offline` |
| 1/8 | Generate database | - |
| 2/8 | Refresh data directories | `--offline` |
| 2a | Refresh MAME BIOS hashes | `--offline` |
| 2a2 | Refresh FBNeo BIOS hashes | `--offline` |
| 2b | Check buildbot staleness | only with `--check-buildbot` |
| 2c | Generate truth YAMLs | only with `--with-truth` / `--with-export` |
| 2d | Diff truth vs scraped | only with `--with-truth` / `--with-export` |
| 2e | Export native formats | only with `--with-export` |
| 3/9 | Verify all platforms | - |
| 4/9 | Generate packs | `--skip-packs` |
| 3/8 | Verify all platforms | - |
| 4/8 | Generate packs | `--skip-packs` |
| 4b | Generate install manifests | `--skip-packs` |
| 4c | Generate target manifests | `--skip-packs` |
| 5/9 | Consistency check | if verify or pack skipped |
| 8/9 | Generate README | `--skip-docs` |
| 9/9 | Generate site | `--skip-docs` |
| 5/8 | Consistency check | if verify or pack skipped |
| 6/8 | Pack integrity (extract + hash) | `--skip-packs` |
| 7/8 | Generate README | `--skip-docs` |
| 8/8 | Generate site | `--skip-docs` |
## Individual tools
@@ -132,6 +133,8 @@ Reports files that cores need beyond what platforms declare.
python scripts/cross_reference.py # all
python scripts/cross_reference.py --emulator dolphin # single
python scripts/cross_reference.py --emulator dolphin --json # JSON output
python scripts/cross_reference.py --platform batocera # single platform
python scripts/cross_reference.py --platform retroarch --target switch
```
### truth.py, generate_truth.py, diff_truth.py
@@ -168,6 +171,8 @@ from upstream repositories into `data/`.
```bash
python scripts/refresh_data_dirs.py
python scripts/refresh_data_dirs.py --key dolphin-sys --force
python scripts/refresh_data_dirs.py --dry-run # preview without downloading
python scripts/refresh_data_dirs.py --platform batocera # single platform only
```
### Other tools