The scraper read emulatorLauncher master on every run, so an extractor
change was diffed together with upstream's commits since the pin. It now
rereads the commit platforms/retrobat.yml pins. --revision master moves
it, and check_freshness and refresh_stale pass that through the
registry's rescrape_args.
The extractors also read master (f681f435): Kronos and YabaSanshiro
fall back to sega_101.bin then mpr-17933.bin when the chosen BIOS is
absent, Ymir writes a forced BIOS only under its override flag, and a
folder created below bios (Ymir's cdb) is a passes_path record. At the
pin the 187 records are unchanged.
The amiberry and hatarib extractors give a search record and its copy
record one list object, and PyYAML wrote it as &id001 and *id001.
platforms/retrobat.yml, batocera.yml and knulli.yml carried such
anchors. The platform dumper now writes every list out, and the two
files the RetroBat scrape does not rewrite are dumped again through it:
their content is unchanged.
ROCKNIX listed retroarch, which stood for every libretro profile (351
resolved), while its devices build 124 libretro packages. The platform
list now names the cores those packages install on at least one device.
The targets scraper read a package's name as its core: vice-lr installs
vice_x128, vice_x64, vice_xpet, vice_xplus4 and vice_xvic, which reached
no device. It now reads each package's makeinstall_target, the file
copied to ${INSTALL}/usr/lib/libretro under its target name, through
TARGET_ARCH and ENABLE_32BIT (pcsx_rearmed-lr, desmume-lr). 155
profiles resolve, each built by a device, and no device loses one.
tests/__init__.py keeps a run offline, its scratch under tmp/ and its
installer cache its own, but only discover -s tests -t . imported it.
discover tests ran with RETROBIOS_OFFLINE unset, scratch on the /tmp
tmpfs and the user's real installer cache, the setting in which e2e
fixtures had written into the shared .cache/large. Every test module now
imports the package first, and a test fails on a module that does not.
test_verify_listing.py took its line with the previous commit.
The target scraper kept its own copy of the EmuScripts table and took
.ps1 files the platform scraper never read, so emudeck-we's template
skeleton was published as a core and removed afterwards by an override.
Both scrapers now read emulator_of_script, and the platform scraper
lists the installers at the pinned revision, as launchers: and the
SteamOS target already did. Targets rescraped, override dropped.
The platform scraper kept lr-dosbox-pure as dosbox-pure, which matched the standalone DOSBox Pure Unleashed profile, while the target scraper wrote dosbox_pure. Both now read core_name(), and platforms/retropie.yml is rescraped at its pinned revision 46128cd2: only the libretro core names change, and six Unleashed files leave the RetroPie manifest.
The games importers wrote through fixed .part names, the new options had
no help text, and romfolders stopped on the first setup the offline cache
lacked. RetroArch's rom folder map is regenerated from its source spec
(path null, as the schema says). The installer's Python 3.8 job imports
the games tests without pyyaml, so the region rules load lazily.
RetroDECK listed every component standalone only. Azahar, Cemu, Dolphin,
MAME, melonDS, PCSX2 and PPSSPP also run through RetroArch: RetroDECK/ES-DE
es_systems.xml at 15e1b236 loads <name>_libretro.so beside each standalone
command, and the cores ship whole from the buildbot nightly
RetroArch_cores.7z (retroarch/component_recipe.json:250-262). Their
libretro layouts, Azahar's save-dir files among them, now reach the pack.
Each OS publishes Batocera's systems dict (knulli-systems, and
batocera-systems at AmberELEC's newest prerelease commit and EmuELEC v4.8),
read with the shared parser. Cores come from each OS's own ES list.
AmberELEC and EmuELEC declare neogeo/ and ports/ beside bios/ from
/storage/roms, so their packs extract there and core_destination puts
core files under bios/.
download.py authenticated its release lookup and download.sh did not, so
past the anonymous quota the shell twin failed with 403. The header goes
to api.github.com only and is read from stdin, out of the process list.
scraper_cli wrote a platform YAML with no system when the parse came back
empty, and exited 0. It now exits 1 before writing, unless the scraper
declares lists_files = False (RetroPie, whose files are inherited); RetroPie
in turn raises when its archive declares no package.
The MAME and FBNeo hash scrapers cached an empty parse for 24 hours and
reported OK. Both now raise before writing, and an empty cache is never
served as fresh.
--emulator accepts a buildbot core name from a profile's cores:, and a
mistyped target gets the closest names instead of every target. diff_truth
shows destinations and reports unpaired entries as evidence, not removals.
The verify summary counts discrepancies, a shared destination names both
systems, a truth hash conflict names both cores, and pipeline steps carry
their place in the real run instead of N/8. Every GitHub API caller takes
the token from github_token(), which falls back to gh auth token.
RetroPie's YAML listed no standalone_cores, so every package was laid
out as a libretro core and the standalone files of amiberry, dolphin,
hatari, mame and ppsspp never reached the pack. Packages without the
lr- prefix (bar the retroarch frontend) are now standalone, and names
shipped both ways are dual-mode.
The scraper declared every standalone package launched both ways,
reasoning that RetroArch carries every libretro core. package.mk says
otherwise: it adds ARMSX2 as armsx2-sa only (add_emu_core ps2 armsx2
armsx2-sa true), so the libretro build's pcsx2/resources entries
reached ROCKNIX's list as armsx2-sa files the standalone installs
itself. An emulator is now dual-mode only when package.mk also adds it
as a retroarch core: ten of the 63 standalone names.
EmuHawk finds every firmware by SHA1 against FirmwareDatabase.FirmwareRecords, and its cores are its own builds. The registry and the scraper both listed libretro cores whose systems overlap BizHawk's, so the pack shipped Clock Signal, O2EM, FBNeo, VICE and Citra files into Firmware/ that EmuHawk never reads. Both lists are now empty, as MiSTer's are: the database alone decides.
Six platforms are read from a branch: System.dat, batocera-systems.json, checkBIOS.sh, the MiSTer BiosDB, the RetroPie scriptmodules and the RetroDECK component manifests. Their YAML named the branch, so the native export patched whatever the branch had become since the scrape. Each scraper now resolves the branch head once, reads every file at that commit and writes it into source. The export follows a recorded tarball and a recorded tree as it already followed a raw file URL.
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.
validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
Batocera, RetroBat, Recalbox and ROCKNIX offer some emulators both as a
libretro core and standalone. Filed as standalone only, the libretro
layout was dropped: libretro Dolphin had no dolphin-emu/Sys/GC files.
Platforms now name dual-mode emulators and each gets both layouts.
Thirty platform system ids, such as sufami, pokemini, vsmile, and
Recalbox's comma-joined msx1,msx2, matched no system any profile
names, so no core file was attributed to them and --system missed them.
The scraper maps now carry those names, and a test holds every platform
system to a profile system apart from two named folds.
RetroDECK filed coco as tandy-coco and Dragon as dragon-32-64, MiSTer
filed COCO3 as tandy-coco and Recalbox kept its raw ids, so none of
them met xroar's coco, coco3 and dragon64 and its ROMs reached no
system. The scraper maps now agree and a test holds every shared
native id to one system.
BizHawk embeds ares as Ares64, its N64 core, whose 64DD IPL its own
FirmwareDatabase declares. Mapped to the whole ares profile, it pulled
thirty systems into the BizHawk pack and made slots accuse
FirmwareDatabase.cs of a wrong MSX.rom it never declared for ares.
The registry list added clk and gsplus, which RetroDECK does not ship
as components, current or archived, and overrode the list the scraper
reads from the component manifests.
RetroBat's platform file carried no core list, and a registry list
copied from Batocera stood in with ports RetroBat never runs and none
of its standalone emulators marked as such. The scraper now reads
es_systems.cfg at the release tag: libretro cores, and every other
emulator as standalone.
es_bios.xml names only the cores that need a BIOS, so a hand-kept
registry list stood in, with mame and mamemess that Recalbox does not
ship and without vice, ymir, vircon32 and others it does. The scraper
now reads each system.ini of recalbox-romfs2 at the scraped tag, and
the registry list is gone.
verify.py and generate_pack.py must reach the same verdict on the same
file, and CLAUDE.md calls any divergence critical, but each spelled out
mode == 'existence' in its own words. Both now ask nativemode whether
the frontend reads the file's bytes, which is the one fact the rest
follows from, and a test holds the two answers together.
The BaseScraper contract the wiki asks contributors to implement had no
caller anywhere, so nothing proved compare_with_config, has_changes or
test_connection still worked.
Abdessamad Derraz