feat: add the autorun job runner

Runs the jobs of automation/jobs.yml under one host lock, with load and disk guards, one idle retry on a host misread, artifacts computed twice, explicit-path commits in .cache/autorun/worktree, an opt-in push (--push) and a CI circuit breaker. First jobs: nightly refresh chain, large-files asset dry run, disk cleanup and check, weekly bios integrity. install-timers writes systemd user units without enabling them.
This commit is contained in:
Abdessamad Derraz committed 2026-10-27 10:40:10 +01:00
1 parent 195a67e03b
commit 343a4e8bd7
5 files changed
+2193 -1

No files matched your search

+92
View File
@@ -0,0 +1,92 @@
# Jobs scripts/autorun.py runs. A change here is a reviewed commit: the
# schedule, the gate and the paths a job may commit are all decided here.
#
# Commands run in the automation worktree (.cache/autorun/worktree) unless
# `worktree: false`, which runs them in the main checkout and forbids
# commits. `python` names the interpreter running autorun.py. A step is a
# command list, or a mapping with `run`, `ok_codes` and `stdout` (a file
# the command's output is written to). `twice` recomputes an artifact the
# gate wrote into `{out}` and compares both, timestamps aside. Each commit
# takes only the changed files its `paths` globs match, and a job that
# leaves any other change behind fails. `report` names a JSON file a step
# writes, kept in the job's status.
#
# Schedules are systemd OnCalendar expressions, written into user timers by
# `python scripts/autorun.py install-timers`.
defaults:
min_free_gb: 30
max_load_wait: 3600
timeout: 21600
jobs:
cleanup:
schedule: "*-*-* 05:30"
worktree: false
min_free_gb: 0
steps:
- [python, scripts/cleanup_disk.py, --apply, --hours, "72"]
disk-check:
schedule: hourly
worktree: false
lock: false
min_free_gb: 0
max_load_wait: 0
steps:
- [python, scripts/cleanup_disk.py, --check]
nightly:
schedule: "*-*-* 02:00"
network: true
min_free_gb: 120
steps:
- run: [python, scripts/check_freshness.py, --json]
ok_codes: [0, 1]
stdout: tmp/autorun/freshness.json
- [python, scripts/refresh_stale.py, --findings, tmp/autorun/freshness.json,
--only, "data,catalogs"]
- [python, -m, scripts.scraper.mame_hash_scraper]
- [python, -m, scripts.scraper.fbneo_hash_scraper]
gate:
- [python, scripts/pipeline.py, --offline]
twice:
- artifact: database.json
run: [python, scripts/generate_db.py, --force, --bios-dir, bios, --output, "{out}"]
- artifact: install
glob: "*.json"
run: [python, scripts/generate_pack.py, --all, --manifest, --offline,
--output-dir, "{out}"]
commits:
- subject: "chore: refresh mame and fbneo hashes"
body: >-
Written by mame_hash_scraper and fbneo_hash_scraper from the
upstream sources, behind a green offline pipeline.
paths: [emulators/mame*.yml, emulators/fbneo*.yml]
- subject: "chore: refresh romset recipes"
body: >-
Snapshots refreshed by refresh_stale from the MAME and FBNeo DATs
check_freshness reported stale, behind a green offline pipeline.
paths: [recipes/*.json]
- subject: "chore: regenerate collection records"
body: >-
Regenerated by the offline pipeline, database.json and the
manifests computed twice with the same result.
paths: [database.json, install/*.json, install/targets/*.json, README.md,
CONTRIBUTING.md, website/src/data/*.json]
scratch: [dist, tmp/autorun]
bios-integrity:
schedule: "Sun *-*-* 08:30"
worktree: false
max_load_wait: 14400
steps:
- run: [python, scripts/bios_integrity.py, --apply, --json]
stdout: tmp/autorun/bios-integrity.json
report: tmp/autorun/bios-integrity.json
assets:
schedule: "*-*-* 04:30"
network: true
steps:
- [python, scripts/sync_release_assets.py]