kmoody/libretro
1
0 Fork 0
mirror of https://github.com/Abdess/retroarch_system.git synced 2026-10-11 05:53:23 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat: add profile sync against upstream

Browse Source
This commit is contained in:
Abdessamad Derraz committed 2026-08-08 11:17:21 +02:00
1 parent eb2c064518
commit ccb9b3e94f
8 files changed
+3212 -29
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

wiki/architecture.md
+11 -7
View File
@@ -275,19 +275,23 @@ user's platform, filter files by hardware target, and download with SHA1 verific
## Tests
10 test files, 400 tests total:
14 test files, 586 tests total:
| File | Tests | Coverage |
|------|-------|----------|
| `test_e2e.py` | 217 | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters |
| `test_e2e.py` | 218 | file resolution, verification, severity, cross-reference, aliases, inheritance, shared groups, data dirs, storage tiers, HLE, launchers, platform grouping, core resolution, target filtering, truth/diff, exporters |
| `test_profile_sync.py` | 129 | ref anchoring, guarded profile writes, detection, triage |
| `test_install.py` | 70 | `install.py` platform detection, config-file parsing, manifest handling |
| `test_upstream.py` | 38 | forge URL parsing, cache, revision resolution, tree comparison |
| `test_provenance.py` | 29 | Logiqx/Redump parsers, DAT pack import, provenance join, coverage report |
| `test_mame_parser.py` | 22 | BIOS root set detection, ROM block parsing, macro expansion |
| `test_mame_parser.py` | 25 | BIOS root set detection, ROM block parsing, macro expansion |
| `test_hash_merge.py` | 17 | MAME/FBNeo YAML merge, diff detection, formatting preservation |
| `test_fbneo_parser.py` | 16 | BIOS set detection, ROM info parsing |
| `test_profile_refs.py` | 12 | `check_profile_refs` pure functions (no network) |
| `test_deterministic_zip.py` | 12 | streaming rebuild, metadata normalisation, entry ordering, source CRC |
| `test_artifact_lock.py` | 10 | writer/writer and writer/reader exclusion, reader sharing, release on error |
| `test_pack_integrity.py` | 8 | extract ZIP packs to disk, verify paths + hashes per platform's native mode |
| `test_torrentzip.py` | 8 | TorrentZip builder against real MAME romsets |
| `test_large_file_cache.py` | 5 | concurrent downloads, temporary file residue, hash rejection |
| `test_no_case_collisions.py` | 1 | guard against case-colliding paths in `bios/` |
```bash
@@ -295,9 +299,9 @@ python -m unittest discover tests -v # full suite
python -m unittest tests.test_e2e -v # single module
```
`test_e2e.py`, `test_install.py`, `test_provenance.py`, the parser tests and
`test_profile_refs.py` run on synthetic fixtures with no network and no real
BIOS files. `test_pack_integrity.py`, `test_torrentzip.py` and
`test_e2e.py`, `test_install.py`, `test_provenance.py`, the parser tests,
`test_profile_sync.py` and `test_upstream.py` run on synthetic fixtures with no
network and no real BIOS files. `test_pack_integrity.py`, `test_torrentzip.py` and
`test_no_case_collisions.py` read the working tree and skip when the data they
need is absent. See the [testing guide](testing-guide.md) for the fixture
pattern and how to add a test.
wiki/profiling.md
+1 -1
View File
@@ -215,7 +215,7 @@ A few field conventions that protect the toolchain:
python scripts/cross_reference.py --emulator dolphin --json
python scripts/verify.py --emulator dolphin
python scripts/verify.py --emulator dolphin --verbose # per-core checks + source refs
python scripts/check_profile_refs.py --emulator dolphin # do the source_ref lines still hold
python scripts/profile_sync.py --emulator dolphin # do the source_ref lines still hold
```
The profile also has to satisfy `schemas/emulator.schema.json`, which CI checks
wiki/testing-guide.md
+21 -7
View File
@@ -23,7 +23,11 @@ python -m unittest tests.test_provenance -v
python -m unittest tests.test_mame_parser -v
python -m unittest tests.test_hash_merge -v
python -m unittest tests.test_fbneo_parser -v
python -m unittest tests.test_profile_refs -v
python -m unittest tests.test_profile_sync -v
python -m unittest tests.test_upstream -v
python -m unittest tests.test_deterministic_zip -v
python -m unittest tests.test_artifact_lock -v
python -m unittest tests.test_large_file_cache -v
python -m unittest tests.test_pack_integrity -v
python -m unittest tests.test_torrentzip -v
python -m unittest tests.test_no_case_collisions -v
@@ -36,15 +40,19 @@ library `unittest` module.
| Module | Tests | Fixtures | What it covers |
|--------|-------|----------|----------------|
| `test_e2e.py` | 217 | synthetic | resolution, verification, packs, cross-reference, targets, truth |
| `test_e2e.py` | 218 | synthetic | resolution, verification, packs, cross-reference, targets, truth |
| `test_profile_sync.py` | 129 | synthetic | ref anchoring, guarded profile writes, detection, triage |
| `test_install.py` | 70 | synthetic | `install.py` detection, config parsing, manifest handling |
| `test_upstream.py` | 38 | synthetic | forge URL parsing, cache, revision resolution, tree comparison |
| `test_provenance.py` | 29 | synthetic | Logiqx/Redump parsing, DAT import, provenance join, coverage report |
| `test_mame_parser.py` | 22 | inline C | BIOS root sets, ROM blocks, macro expansion |
| `test_mame_parser.py` | 25 | inline C | BIOS root sets, ROM blocks, macro expansion |
| `test_hash_merge.py` | 17 | synthetic | YAML hash merge, diff, formatting preservation |
| `test_fbneo_parser.py` | 16 | inline C | `BDF_BOARDROM` sets, ROM info parsing |
| `test_profile_refs.py` | 12 | synthetic | `check_profile_refs` pure functions, no network |
| `test_deterministic_zip.py` | 12 | synthetic | streaming rebuild, metadata normalisation, entry ordering, source CRC |
| `test_artifact_lock.py` | 10 | synthetic | writer/writer and writer/reader exclusion, reader sharing, release on error |
| `test_pack_integrity.py` | 8 | real packs | extract each ZIP, verify paths and hashes |
| `test_torrentzip.py` | 8 | real romsets | TorrentZip builder byte-for-byte |
| `test_large_file_cache.py` | 5 | synthetic | concurrent downloads, temporary file residue, hash rejection |
| `test_no_case_collisions.py` | 1 | real `bios/` | no case-colliding paths on Windows/macOS clones |
## Test architecture
@@ -123,9 +131,15 @@ detection, each registry detection method (`config_file`, `path_exists`,
`file_exists`), config-file key parsing, manifest loading, target filtering,
and destination resolution.
**test_profile_refs.** Covers the pure functions of `check_profile_refs`
(anchor matching, line-window search, hash extraction). The GitHub fetching
path is not exercised, so the module runs offline.
**test_profile_sync.** Covers ref anchoring end to end: the six statuses,
widening a one-line anchor until it is unique, refusing to rebase an ambiguous
one, following a rename, and the guarded YAML writes. Network access is
replaced by an injected fetch function.
**test_upstream.** Covers forge URL parsing for GitHub, GitLab and Forgejo,
the content-addressed cache and its atomic write, revision and tag resolution,
and tree comparison. The HTTP layer is replaced at module level, so nothing
leaves the machine.
### Tests that read the working tree
wiki/tools.md
+44 -14
View File
@@ -208,25 +208,55 @@ to the files inside the archive, so the exported DAT lists those entries
without a container sha1. Anyone submitting the DAT upstream should mention
this.
### check_profile_refs.py
### profile_sync.py
Audit `source_ref` line references against the profiled upstream. The
commit under audit is the profile's `source_commit` when present, else the
last upstream commit at `profiled_date`. Each referenced file is fetched
at that commit and at HEAD, and the entry's declared hashes (or filename)
are searched around the cited lines.
Confront a profile with its upstream. The pinned commit is the profile's
`source_commit` when present, else the last upstream commit at
`profiled_date`. Each cited line range is extracted at the pin and located
in the HEAD revision of the same file.
```bash
python scripts/check_profile_refs.py --emulator vice
python scripts/check_profile_refs.py --all --json
python scripts/profile_sync.py --emulator vice
python scripts/profile_sync.py --emulator vice --full-diff
python scripts/profile_sync.py --all --triage
python scripts/profile_sync.py --all --changed-only --json
python scripts/profile_sync.py --emulator vice --fetch-plan
```
Per ref and revision: `anchored` (found at the cited lines), `moved`
(found elsewhere in the file), `gone` (absent from the file). `moved` at
HEAD means upstream shifted since profiling; `gone` at pin means the
declared value does not come from the referenced file and the entry
deserves a re-read. Uses `GITHUB_TOKEN` when set; GitHub-hosted upstreams
only.
Per part of a ref: `ANCHORED` (same content, same lines), `SHIFTED` (same
content, moved), `RENAMED` (the source file moved), `CHANGED` (content
edited), `AMBIGUOUS` (several equally good candidates), `GONE` (nothing
left to anchor to). An entry carries the worst status of its parts.
A single cited line is often not distinctive, so the anchor widens by
steps of ±3, ±6 and ±12 lines until it is unique. Anything still ambiguous
is reported, never guessed.
Three shorthand forms appear in the corpus and are resolved rather than
reported missing. A part reduced to a line range continues the previous
part's file (`geo.c:234-243, 273-285`). A profile whose `source` differs
from its `upstream` may cite paths from both, and each path is attributed
to the repository that carries it. A path prefixed with a repository
directory name (`EightyOne/Source/HW_.cpp`) is stripped as a last resort,
only after the path as written has failed everywhere, and the result is
reported as `RENAMED` so `--rebase-refs` cleans the profile.
`--check-version` compares `core_version` with the latest upstream tag and
release. `--detect-new-files` lists filename literals at HEAD the profile
does not declare. `--watch-hashes` lists hash literals added upstream that
match no entry. `--tree-diff` shows added, removed and renamed files in the
directories the refs point at.
Writes are explicit and mechanical only. `--backfill-commits` fills a
missing `source_commit`, `--rebase-refs` recales `SHIFTED` and `RENAMED`
line ranges, `--bump-commit` advances `source_commit` to HEAD only when
nothing needs a re-read. All three refuse to run on a dirty `emulators/`
without `--force`, and every write is verified by reparsing the document.
Uses `GITHUB_TOKEN` when set, which `--all` requires. Responses are cached
under `.cache/upstream/`, addressed by commit sha, so `--offline` replays a
previous run. GitHub, GitLab and Forgejo upstreams; other hosts are
reported as skipped.
### validation.py