feat: read measurements at a pinned commit

measurements-pin.json names the measurements repository and the commit read. sync_measurements.py reads the clone in .cache/measurements at that commit, checks every line against its id and sidecar, and writes the site index and the SQLite databases. --fetch refuses a remote that changed or removed a line the pin holds, --bump moves the pin. validate_schemas checks every line and sidecar against their schemas, check_freshness flags a pin behind the remote head.
This commit is contained in:
Abdessamad Derraz committed 2026-10-27 08:30:11 +01:00
1 parent 6109d7abd5
commit effa9e332f
6 files changed
+662 -10

No files matched your search

+9 -1
View File
@@ -27,6 +27,9 @@ Mapping (stale -> command):
--source mame --fetch
catalogs/fbneo recipes python -m scripts.scraper.romset_dat_importer
--source fbneo --fetch
measurements/records pin
python scripts/sync_measurements.py --fetch --bump,
which refuses a remote that rewrote a pinned line
A platform that inherits from a rescraped one (retropie and lakka from
retroarch) reads the parent's file: its jobs start once the parent's chain
@@ -63,7 +66,7 @@ CHECK_FRESHNESS = REPO_ROOT / "scripts" / "check_freshness.py"
PLATFORMS_REGISTRY = REPO_ROOT / "platforms" / "_registry.yml"
DATA_DIRS_REGISTRY = REPO_ROOT / "platforms" / "_data_dirs.yml"
AUTO_AREAS = ("platforms", "native", "targets", "data", "catalogs")
AUTO_AREAS = ("platforms", "native", "targets", "data", "catalogs", "measurements")
MANUAL_AREAS = ("coreinfo", "ci", "profiles")
JOB_TIMEOUT = 1800 # 30 minutes per refresher, scrapers rarely exceed 10
@@ -407,6 +410,11 @@ def _command_for(
# redump / no-intro / tosec need manual DAT download.
return None
if area == "measurements":
# --fetch refuses a remote that rewrote a pinned line, and --bump
# only rewrites measurements-pin.json: nothing is committed.
return [sys.executable, "scripts/sync_measurements.py", "--fetch", "--bump"]
# coreinfo, ci, profiles: never auto-refresh.
return None