Commit Graph
2529 Commits
Author SHA1 Message Date
Abdessamad Derraz 8e3ba2cfd3 chore: merge scraper and tool fixes 2026-11-06 16:36:32 +01:00
Abdessamad Derraz 1bbbdcd4e4 fix: keep tool names out of the probe path check 2026-11-06 15:46:54 +01:00
Abdessamad Derraz 057fd2d0f6 fix: isolate every test module however it runs
tests/__init__.py keeps a run offline, its scratch under tmp/ and its
installer cache its own, but only discover -s tests -t . imported it.
discover tests ran with RETROBIOS_OFFLINE unset, scratch on the /tmp
tmpfs and the user's real installer cache, the setting in which e2e
fixtures had written into the shared .cache/large. Every test module now
imports the package first, and a test fails on a module that does not.
test_verify_listing.py took its line with the previous commit.
2026-11-06 15:14:05 +01:00
Abdessamad Derraz 7dcb427e27 fix: hint at --verbose when no bundled name shows
With no required firmware among the bundled entries the line read
'20 (20 more, --verbose lists them)'. It now reads '20 (--verbose lists
them)', and reading-reports.md quotes the run as it prints.
2026-11-06 15:13:55 +01:00
Abdessamad Derraz cca439a903 fix: run refused writers in a copy of the tree 2026-11-06 15:13:30 +01:00
Abdessamad Derraz 1b4b31fb32 fix: run the read-only tools from the repo root
Only verify.py and generate_pack.py moved to the repository root. From
any other folder slots, region_audit, provenance_report and bundled_only
said database.json was missing and to rebuild it, identity and
list_platforms named a missing folder, and game_rules ended on a
FileNotFoundError traceback. work_from_repo_root moves to common.py and
the eight tools call it: defaults are read from the root, a path the
user typed from where it was typed.
2026-11-06 15:09:35 +01:00
Abdessamad Derraz bf10ebaa82 docs: say verify --all covers every platform
The --all help still read 'Verify all active platforms' after the
archived status was dropped. --all covers every registered platform.
2026-11-06 15:06:42 +01:00
Abdessamad Derraz 1fe37d16ac fix: keep stderr after the report it closes
verify.py --emulator fbneo --strict > log 2>&1 put the ERROR verdict on
the first line of the log: redirected, stdout is block-buffered and
stderr is not, and run_cli flushed stdout only after main returned.
run_cli now sets stdout to line buffering, so every tool it runs keeps
the two streams in the order they were written.
2026-11-06 15:06:26 +01:00
Abdessamad Derraz f2132cc2c0 fix: spell out the checks under a missing file
Without --verbose each missing core file was followed by a bare '[size]'
line that told the reader nothing. It now reads 'validates size', the
verbose wording without the expected values.
2026-11-06 15:03:05 +01:00
Abdessamad Derraz 8bdf19c7dd feat: progress lines for online profile_sync 2026-11-06 15:02:53 +01:00
Abdessamad Derraz c42bd0f688 fix: count the bundled names verify leaves out
'Bundled with their emulator, not collected: 154' listed the 14 required
firmware names and nothing said the rest existed. The line now ends with
'139 more, --verbose lists them', as the Core files line does.
2026-11-06 15:02:47 +01:00
Abdessamad Derraz 1834d20dc9 fix: name the file a YAML syntax error is in
Every profile is loaded whatever the command, so one misindented key in
an unrelated profile stopped verify.py --emulator handy and generate_pack
on a 37-line yaml traceback. The sources common.py reads (profiles,
platform files, registry, shared groups, targets) now raise
SourceUnreadable, which run_cli prints on one line with the file, the
line and the validate_schemas command. It is not a FileNotFoundError:
ten callers skip a platform whose file is absent, and a damaged one must
not be skipped the same way.
2026-11-06 15:02:01 +01:00
Abdessamad Derraz 7080459011 fix: read No-Intro from a mirror that rebuilds 2026-11-06 14:59:40 +01:00
Abdessamad Derraz 8b55ab45a5 fix: suggest the database beside a mistyped --db
--db databse.json, with database.json in the same folder, was told to run
generate_db --force --output databse.json: a rescan of the whole
collection into a second file under the typo. load_database now names the
closest .json in that folder first, and gives the rebuild command only
when none is close.
2026-11-06 14:57:48 +01:00
Abdessamad Derraz 6a5b384e02 fix: say when a RetroPie help is updated in part 2026-11-06 14:56:11 +01:00
Abdessamad Derraz de8f20f409 fix: count listed systems in the --system hint
A mistyped --system said '--list-systems lists all 579' and the listing
prints 532: the hint counted every spelling across profiles, and
system_index merges the ones that differ in case or separator before
printing. Both verify.py and generate_pack.py now pass that count to
suggest_names, whose listed parameter exists for this case.
2026-11-06 14:55:40 +01:00
Abdessamad Derraz 144cd46df1 fix: count one export line per addition path 2026-11-06 14:55:23 +01:00
Abdessamad Derraz c9e5ea67e7 fix: count a missing core file once in the summary
The closing summary of verify --all grouped platform to names: sfiii2a.zip,
missing on 15 platforms, was printed 15 times on a 1 060-character line and
counted 24 files for 10 distinct ones. It now groups by file and names the
platforms by the shorter side (every platform but BizHawk, MinUI, MiSTer
FPGA).
2026-11-06 14:54:38 +01:00
Abdessamad Derraz 82bccd9dae fix: refuse Python 3.8 in every build script
Only generate_pack.py and pipeline.py checked the interpreter. verify.py,
slots.py, validate_pr.py, cross_reference.py and the probe tools started
under 3.8 and stopped halfway on a str.removeprefix AttributeError. The
check now lives in common.py, which every build script imports, and in
core_probe.py and region_check.py, which do not. generate_site.py imports
common before siterender, which raised a TypeError at load under 3.8.
2026-11-06 14:52:50 +01:00
Abdessamad Derraz 651c64c763 fix: drop cut scratch paths from probe records 2026-11-06 14:52:05 +01:00
Abdessamad Derraz 190c069767 chore: check every registered scraper's pin 2026-11-06 14:49:56 +01:00
Abdessamad Derraz 1f3fc52013 fix: read the EmuDeck Windows cores from Windows 2026-11-06 14:49:42 +01:00
Abdessamad Derraz b56ec21728 feat: check type hints in CI against a baseline
Every signature carries annotations and nothing read them. mypy 1.18.2
over install.py and scripts/ reports 993 errors today, among them the
six pack helpers typed zipfile.ZipFile that every caller also hands a
PackRecord. The errors sit in mypy-baseline.txt and the schema job fails
on any error outside it, or on an entry that no longer occurs. Third-party
packages are left out so the result does not depend on the machine: 6 s
locally, run twice and on 3.12 and 3.13 with the same errors.
2026-11-06 14:47:20 +01:00
Abdessamad Derraz ae0b23a801 fix: show the check's own error in refresh_stale 2026-11-06 14:46:17 +01:00
Abdessamad Derraz 395910ed2b fix: one-line errors for --record-coreinfo 2026-11-06 14:45:26 +01:00
Abdessamad Derraz 3b137cd266 fix: name a missing folder in the folder check 2026-11-06 14:44:31 +01:00
Abdessamad Derraz 2d1c889d41 refactor: name the database option --db everywhere 2026-11-06 14:42:05 +01:00
Abdessamad Derraz a99682beeb chore: merge origin/main 2026-11-06 13:46:16 +01:00
Abdessamad Derraz b8d9b2a90c fix: send searched words only as file names
A search that found nothing sent any short text, which a name could be.
Its words now go out only as a file name or a hash, every other search
is counted as "other" without them.
2026-11-06 13:11:15 +01:00
Abdessamad Derraz 8100d671f9 chore: merge origin/main 2026-11-06 13:02:20 +01:00
Abdessamad Derraz c1b0149a81 chore: merge site insights 2026-11-06 12:56:13 +01:00
Abdessamad Derraz b23840bc21 feat: add site insights report
Search Console queries by page, file names searched against the
collection, GoatCounter totals and GitHub traffic in one Markdown report.
2026-11-06 12:55:57 +01:00
Abdessamad Derraz ebe9745e7f feat: count visits without cookies
Pages, referring domains and a few actions (searches that found
nothing, downloads, missing addresses, "Did this page help?") are sent
to GoatCounter. Nothing is stored on the device, and nothing is sent
under Do Not Track or Global Privacy Control. wiki/privacy.md says what
is sent.
2026-11-06 12:55:57 +01:00
Abdessamad Derraz 44565a07bb feat: read the EmuELEC device in the installer
EmuELEC's scripts/image writes COREELEC_DEVICE to os-release (v4.8, line
189), as a GameForce image shows. The installer reads it the way it does
for AmberELEC and applies install/targets/emuelec.json. MiSTer FPGA takes
EmuELEC's place in the test of a platform without targets.
2026-11-06 12:09:09 +01:00
Abdessamad Derraz 5e90a455e4 feat: add EmuELEC hardware targets
The v4.8 release publishes four images (Amlogic-ng, GameForce,
OdroidGoAdvance, OdroidM1, read from its asset names). The metapackage
(packages/sx05re/emuelec/package.mk:8-77) builds the S922X cores, dolphinSA
and mame only on Amlogic-ng, Amlogic-no, RK356x and OdroidM1, and removes
duckstation on the handhelds with a sed that replaces the first match only:
pcsx_rearmed, written twice, keeps its 64-bit build. Each package is mapped
to its cores by the .so or program its install step copies, PKG_ARCH is
applied as scripts/build:28-30 does, and es_systems.cfg is edited per
device as emuelec-emulationstation/package.mk:114-143 does. Amlogic-ng and
OdroidM1 list 124 cores, the handhelds 117.
2026-11-06 12:09:05 +01:00
Abdessamad Derraz 14cf0a13f2 chore: keep the platform status from returning 2026-11-06 11:52:51 +01:00
Abdessamad Derraz 4d3f589ef4 refactor: drop the archived platform status
RetroPie was the only platform with status: archived, which kept it out
of verify --all, slots, the pipeline and list_platforms unless
--include-archived was typed. release_pack already decides which packs a
release publishes and the installer serves every platform, so the status
set nothing apart. The field, its schema entry and the flag are gone, and
RetroPie is covered by every full run like any other platform.
2026-11-06 11:52:34 +01:00
Abdessamad Derraz 2e98be9366 fix: accept the bundled line in the report excerpt
verify now prints a line for files installed with their emulator, and
reading-reports.md quotes it. The shapes the excerpt may hold did not
list it.
2026-11-06 11:10:33 +01:00
Abdessamad Derraz 25b11fa497 chore: merge site fixes, fourth pass 2026-11-06 10:33:56 +01:00
Abdessamad Derraz d0c65dca22 chore: merge pack and verify fixes 2026-11-06 10:28:27 +01:00
Abdessamad Derraz c0e4bd8379 chore: merge resolver fixes, fourth pass 2026-11-06 10:28:27 +01:00
Abdessamad Derraz a29b836730 chore: merge installer fixes, fourth pass 2026-11-06 10:28:14 +01:00
Abdessamad Derraz 1396350795 fix: count build file trees as pinned
check_source_paths read the trees profiles pin from source and upstream
only, so the two platform trees the record now holds for pin_build
files read as orphaned and the record as stale.
2026-11-06 10:14:02 +01:00
Abdessamad Derraz e491ff1bc6 chore: merge tooling fixes, fourth pass 2026-11-06 09:50:16 +01:00
Abdessamad Derraz 508dc4d039 docs: list the new test modules
test_twin_declaration_status and test_verify_data_dir_notice join the testing guide table, and test_large_file_cache calls unittest.main after its last class.
2026-11-06 09:44:28 +01:00
Abdessamad Derraz 21f4122c83 fix: collisions keep the platform's order
Walking the systems sorted reordered the declarations doubled.py
reports from, so the export report named the CoCo reader first. The
declarations keep the platform's order again, and the packed file is
picked from the sorted walk the builder makes.
2026-11-06 09:43:06 +01:00
Abdessamad Derraz 79a21ba1f1 feat: name the files in emulator search snippets
A search for gba_bios.bin or scph5500.bin landed on a page described as
"It reads 4 files, none of them required". The snippet now names the
console and the files, required first, in the longest form that fits.
2026-11-06 09:28:03 +01:00
Abdessamad Derraz 856cb5618f fix: link a build file only where it is recorded
Every source link on the site leads to a path the source record saw at
a commit. The Build file row linked pin_build files unchecked, eight of
them at commits the record did not hold. build_source_paths now records
the build files read at a commit, and the row links only those: a tag
can move and stays plain text. The record run also picked up two cited
paths it had not seen before (pom2 CMakeLists.txt, pcem configure.ac).
2026-11-06 09:14:42 +01:00
Abdessamad Derraz 462b61a4bc refactor: compute slot choices once per manifest
generate_manifest called slot_choices, and pack_overrides called it again with the same five arguments. pack_overrides takes the choices when the caller holds them. Measured on generate_pack.py --all --manifest --offline: 48 calls and 7.0 s in slot_choices before, 24 calls and 4.3 s after (wall 45.7 s and 42.4 s, one run each on a loaded host). The manifests written are identical.
2026-11-06 09:11:37 +01:00
Abdessamad Derraz 9633ce3dd4 refactor: normalize each system id once
SelectionIndex.build and system_group called _norm_system_id 1.7 million times per manifest run for about a thousand ids, each through the manufacturer prefix loop. The function reads only its argument and two constant tables, so its answer is cached. Measured on the call volume of one manifest run (630 ids, 1,686,618 calls): 1.53 s uncached, 0.07 s cached. The end-to-end manifest run moves inside its own noise on this host (45.9 to 50.6 s across four runs).
2026-11-06 09:08:09 +01:00