Commit Graph
86 Commits
Author SHA1 Message Date
Abdessamad Derraz 4a1d875f43 feat: name systems in the install manifests
Each manifest carries system_titles, the name the site prints for every system its files list (Sony PlayStation, not sony-playstation or psx), so the installer can show it. Older installers ignore the field.
2026-10-23 15:55:46 +02:00
Abdessamad Derraz 94d18bd7e5 feat: name the system in each lookup record 2026-10-23 10:54:34 +02:00
Abdessamad Derraz e3a54d2423 feat: serve each schema at its own $id 2026-10-23 10:46:05 +02:00
Abdessamad Derraz 0b4504992b feat: read batocera md5 written as a list
batocera.linux 6b08b873 lets a biosFiles item declare its md5 as a
list of strings, and checkBios accepts a file matching any of them.
The scrapers join the values with commas, as for Recalbox, and record
md5_list so the export writes the list back as a list, byte for byte.
2026-10-23 07:39:25 +02:00
Abdessamad Derraz 8b26a8fcac feat: cite sources and files on reference pages
Platform pages link each reader's source line at its pinned revision, name the collection file that serves each entry, show launcher records with their lines, name systems with the id beside them and the list they were read from in words. The largest system lists move to pages of their own so a platform page stays under 500 KB.

Gaps: plain numbers with a bar, a badge only for missing required firmware, Files to source as a heading. Cross-reference: counts per platform, one page per platform, one row per upstream project. Exports: gaps listed once, status, category and the serving file in the CSVs and SQLite, the commit in each API envelope, the provenance Wanted list as CSV. Catalog dates in words on the provenance page.
2026-10-23 02:58:51 +02:00
Abdessamad Derraz 2d26385efe feat: place save-dir files in each system folder
Batocera, KNULLI, Recalbox, RetroBat and RetroDECK hand a libretro core <saves>/<system> at each launch. Each copy lands below every system that runs the core. EmuDeck keeps one unsorted saves/retroarch/saves. The unsupported platforms record why.
2026-10-22 21:16:59 +02:00
Abdessamad Derraz 9a1bf4baa6 feat: record the systems each libretro core runs 2026-10-22 21:16:47 +02:00
Abdessamad Derraz 1cd6c9ee4d feat: place save-dir files in packs and lists
load_from: save_dir entries of a libretro build were left out of every
pack. They now go below the platform's save folder (registry
save_directory: RetroArch saves/, Lakka savefiles/, by core), the
manifest marks them root: save, and verify replays the same placement.
Platforms keeping saves per system or per game list them as omitted.
2026-10-22 19:58:26 +02:00
Abdessamad Derraz 33041b6bd1 feat: find a file by any hash, out of the search
Hashes, link lists, install commands, member tables and long reference
tables leave the site search: the index drops from 21.0 MB (4.88 MB
gzip) to 6.8 MB (1.33 MB gzip). Names and prose stay searchable.

files/?hash= takes a SHA1, MD5, CRC32 or SHA256 and reads one shard of
api/v1/lookup/ (about 22 KB gzip, was 675 KB) listing the collection
card, the platforms that declare the value and the profiles that check
it. files/?sha1= still opens the card, and a sha1- anchor missing from
a page is looked up there, so a card can move between pages.

Excluded blocks are section elements: the search plugin compares
elements by tag, so a div ended at the first nested div.
2026-10-22 12:46:40 +02:00
Abdessamad Derraz 4bc90cf06e chore: merge the elec core names 2026-10-22 06:29:22 +02:00
Abdessamad Derraz 9648b9fc3b feat: read elec ports and per-system cores
EmuELEC's ports system has one emulator, `ports`, the dispatch to a
script ports.yaml generates. The scraper now reads each port's engine
(the libretro core handed to emuelecRunEmu.sh with -C, else port_exe),
so devilutionx, sonic2013, soniccd, sonic-mania, reminiscence and
rigelengine are examined. Standalone emulators keep the spelling the
launch code dispatches on instead of invented lowercase forms, and
media players and runtimes go to the overrides' remove_cores.

Each system of the AmberELEC, EmuELEC and KNULLI lists now carries the
cores its ES list offers it, in menu order, under `cores:`.
2026-10-22 06:28:24 +02:00
Abdessamad Derraz 8b952968d6 feat: add a --no-game-data pack narrowing
The install manifest marks each game data file "category": "game_data":
a platform entry only game data profile entries read, or a core file its
profile declares game data. Firmware carries no category. Every manifest
says "categorized": true so the installer can refuse an older list.

generate_pack.py --no-game-data cuts those files from the platform list
and the core profiles before the build, names each one with the cores that
read it as game data, and tags the pack _NoGameData through _narrowings.
verify.py and the pack check take the same cut, so --verify-packs and the
report describe the pack the builder wrote. Modes without a platform
(--emulator, --system alone, --from-md5) and --all-variants refuse it.
2026-10-22 06:19:05 +02:00
Abdessamad Derraz 841017c887 chore: merge the minui family platforms 2026-10-22 04:09:17 +02:00
Abdessamad Derraz d0802c5360 feat: place core files per handed system dir
A platform can hand each libretro core a system directory of its own
(core_system_dirs). platform_layouts then gives one view per directory,
so the gap report, the builder, the manifest, slots and cross_reference
place a core's files below each folder it is handed, and a core handed
none gets no file. Resolution strips the folder again and searches the
collection by the core's own path.
2026-10-22 04:04:31 +02:00
Abdessamad Derraz 2bdac466a9 feat: add knulli, amberelec and emuelec platforms
Each OS publishes Batocera's systems dict (knulli-systems, and
batocera-systems at AmberELEC's newest prerelease commit and EmuELEC v4.8),
read with the shared parser. Cores come from each OS's own ES list.
AmberELEC and EmuELEC declare neogeo/ and ports/ beside bios/ from
/storage/roms, so their packs extract there and core_destination puts
core files under bios/.
2026-10-22 03:42:20 +02:00
Abdessamad Derraz d5a25190a3 fix: place standalone files in the linked folder
standalone_path is relative to the directories a standalone build
resolves it against, now declared per profile (standalone_directories).
A launcher record that links or passes a whole folder of the BIOS tree
under one of them gives that folder, and platform_layouts rebases the
standalone view once for the builder, the manifest, verify and slots
(launchers.standalone_root, common.standalone_destination).

Flycast on RetroPie and ROCKNIX reads bios/dc, on EmuDeck
bios/flycast/bios, on Batocera (FLYCAST_BIOS_PATH) and RetroBat
(Dreamcast.BiosPath) bios/dc. The ROCKNIX, Batocera and RetroBat
scrapers now record the patched data directory, the environment
variable and the setting that say so.
2026-10-21 12:39:03 +02:00
Abdessamad Derraz d6a0639330 chore: merge audit fixes (packrules)
# Conflicts:
#	scripts/packextras.py
2026-10-21 11:52:28 +02:00
Abdessamad Derraz cd589b2742 chore: merge audit fixes (mame) 2026-10-21 11:48:30 +02:00
Abdessamad Derraz 8643d4a4ea chore: merge audit fixes (site)
# Conflicts:
#	emulators/flycast.yml
#	emulators/primehack.yml
2026-10-21 11:48:30 +02:00
Abdessamad Derraz 56e8cbe046 feat: file mame device sets by their machines
The -listxml names, for every machine, its parent (romof), the devices of its default configuration (device_ref) and the cards its slots offer (slotoption). A machine takes the system of the first set on its romof chain a written entry lists, and passes it to each device it can instantiate. A device set reached from one system gets system:, from several the new per-entry systems:, from none neither, and keeps serving every system of the profile. The gap report, the core extras and slots read the entry's own systems, so generate_pack --system and the installer pick by them.
2026-10-21 11:33:39 +02:00
Abdessamad Derraz ef69687fe1 fix: replay held omissions in install regions
The builder keeps a held regional fallback when the requested file is
missing (since 2026-10-17), and the comment beside the manifest's
omissions still said the opposite. A file the builder holds but the
manifest cannot download sat in omitted_files counted unheld, so the
installer kept a fallback the pack drops. Such an omission now carries
held: true and narrow() counts it held, as the builder does.
2026-10-21 11:18:53 +02:00
Abdessamad Derraz c8c4f3f486 chore: merge round 2 fixes (installer) 2026-10-21 10:17:44 +02:00
Abdessamad Derraz 69965ebb25 feat: link each platform and system to its data
An emulator page linked its YAML and api/v1 JSON, platform and system
pages nothing. Each platform page now links its YAML, its own
api/v1/platforms/<id>.json and its install manifest, and each system
page its api/v1/systems/<slug>.json, both validated by the envelope
schema.
2026-10-21 10:15:11 +02:00
Abdessamad Derraz 5e4d9199c2 feat: accept platform system names in --system
The manifest now carries system_names, each system's native_system names from the platform's own file, and --system, its suggestions and --list-systems read them: --platform batocera --system psx installs sony-playstation.
2026-10-21 09:59:25 +02:00
Abdessamad Derraz 43337c6719 fix: match release packs by exact name
release_record rebuilt nothing from the name builder: a substring match
took RetroArch_v1.22.2_AndroidArm64V8A_BIOS_Pack.zip for the RetroArch
pack. The record now refuses any pack that is not a registered platform's
full pack and stores the platforms each pack serves, which the readers
use instead of the pack name.
2026-10-21 09:51:55 +02:00
Abdessamad Derraz 5fc2423ca5 fix: apply --required-only to data directories
The Required packs and manifests carried every data tree, the FreeIntv
overlays included. A data_directories entry now accepts required: false,
and the platform, emulator and manifest walks leave such a tree out under
--required-only. FreeIntv reads its overlays only with the multi-screen
option and draws a placeholder keypad without one (src/libretro.c:294-296,
350-369).
2026-10-21 05:05:49 +02:00
Abdessamad Derraz 15ffeddfe3 feat: read RetroPie launchers from scriptmodules
The scraper walks each package's functions the way RetroPie runs them in one shell (assignments, arrays, for loops, brace expansion, $md_id, calls into other packages) and records every link, copy, move and setting that names $biosdir, the addEmulator flags (sdltrs -romfile3), the BIOS read outside $biosdir (basilisk --rom, gngeopi -B) and the launch scripts that exit without their file (fs-uae kick13.rom, reicast dc_boot.bin).

Each package also gets a package record with its rp_module_repo and the systems it registers, which maps a package to the build it compiles. The schema gains package, repo, systems and the moves and package kinds. platforms/retropie.yml is regenerated at 46128cd2: only the launchers section is new.
2026-10-20 21:13:09 +02:00
Abdessamad Derraz 57af8e4921 feat: read ROCKNIX launchers from its packages
The scraper reads the start scripts and the configs each package installs at the tag the BIOS list is pinned to: configs that name a BIOS path, the yabasanshiro search, directories handed to DuckStation and the PCSX2 builds, and the ln -s commands that put bios/ where an emulator reads. A package that ships one config per device gets a device field only where the values differ, as melonDS does on AMD64.
2026-10-20 20:45:19 +02:00
Abdessamad Derraz 2556a03ed3 feat: describe platform launchers in the schema
A launcher record says what a platform's own launch code looks for and hands to an emulator, read from that code at the pinned revision.
2026-10-20 20:14:30 +02:00
Abdessamad Derraz f111e1c505 docs: built_by names who ships a build
The resolver reads built_by as the platforms on which a core name means this build. Its definition said platforms that compile a libretro build, which excludes a standalone build a platform installs from the project's release. The field now says what it decides: platforms that ship this build under its core name, compiled or installed from the release.
2026-10-20 16:57:24 +02:00
Abdessamad Derraz e5174a07bf feat: mark files a core finds by content
DuckStation identifies its BIOS by MD5 in a folder it scans
(bios.cpp:208), so its entries' names are labels. The export wrote them
into Batocera's and ROCKNIX's lists as files to hold: a user with only
scph5501.bin read 104 missing BIOS where DuckStation boots. Labels also
lent their md5 to a name another core opens, and paired by name they
replaced System.dat's scph5000.bin hashes.

found_by: name | content, on a profile or a file and keyed by build like
load_from. The truth records which cores open each name (_named_by). A
label pairs with a platform entry by hash only, is never added, merges
with an opened name only on a shared hash, and is counted in the export
report rather than listed as missing.
2026-10-20 16:32:22 +02:00
Abdessamad Derraz 40bc07aa3a feat: record files a launcher never opens
unread_on names the platforms whose launcher points the emulator at other files. RetroBat writes absolute BIOS paths into standalone BlastEm's config, so the default names were reported as required and missing there. Such an entry leaves that platform's gap report and pack, and its emulator page says where it is never opened.
2026-10-20 10:27:15 +02:00
Abdessamad Derraz 1ff2bfc3c0 fix: the API schema holds each endpoint's fields
The data page promises that v1 fields are only ever added, and the envelope schema left items unconstrained, so a renamed field passed validation. Each kind now lists the fields its items carry and their types.
2026-10-20 06:18:25 +02:00
Abdessamad Derraz f7d57dd91d feat: a platform package can differ from upstream
bundled describes the emulator's own release, and a platform package can differ both ways: Batocera's hcl package installs the binary alone and sends users to fetch the data, while its blueMSX core package copies the machines and databases beside the core. not_bundled_on and installed_on name those platforms, and the builder reads them through one rule.
2026-10-20 06:11:10 +02:00
Abdessamad Derraz ddc7432825 feat: packs leave out what the emulator installs
A file or tree installed with the emulator is already on the user's machine, so packs, install manifests and verify's pack count leave it out. bundled is scoped by mode like any entry: Dolphin's Sys and PPSSPP's assets ship with the standalone releases, not with the libretro cores, so they now follow the libretro build only and still reach RetroArch packs. PrimeHack, standalone only, ships its own Sys.
2026-10-20 06:04:10 +02:00
Abdessamad Derraz 77389236e6 feat: record a release with no genuine copy
Paid Android emulators announce versions that circulate only rebuilt or re-signed. The profile stays on the newest unmodified build, and version_source.no_genuine_copy names the release searched for, so check_freshness stops flagging it until the page announces a later one.
2026-10-20 04:49:24 +02:00
Abdessamad Derraz 282f8d5d0a feat: watch the release page of off-forge profiles
About forty profiles describe projects published outside any forge (Altirra, ePSXe, DraStic, mednafen, xroar). profile_sync skipped them as an unsupported host and check_freshness reported the same UNKNOWN line every run, so a new release adding a firmware went unseen. A profile can now name the page announcing its releases and a pattern for the version: --check-version reads it, and check_freshness --profiles calls the profile stale when the page announces another version than core_version.
2026-10-20 04:28:27 +02:00
Abdessamad Derraz 7d1e9f8816 fix: keep RomM files where its player reads them
RomM lists a platform's firmware folder without descending (base_handler.py:1002, recursive=False) and EmulatorJS writes the chosen file, or each zip member, under its bare name. 543 files of the RomM pack sat below a folder where no core could read them. The registry now marks RomM's folder flat, and such files are omitted with the reason below_platform_folder in the manifest and the installer.
2026-10-19 06:57:36 +02:00
Abdessamad Derraz 4231902a6c docs: write prose without dashes or semicolons
Profiles, the wiki, comments, docstrings, command output and schema descriptions carried em-dashes, clause-joining semicolons, unicode arrows, box-drawing banners and check-mark glyphs. Each sentence was rewritten rather than having its character swapped. Hashes, paths, source_ref values and every cited file:line are unchanged, which a comparison of each parsed profile against the previous revision confirms.

validate_pr comments now say PASS and FAIL in words, and source_ref values in a profile's structured blocks render as code. test_prose_style fails if any of these comes back, in tracked files or in the generated site.
2026-10-19 06:47:31 +02:00
Abdessamad Derraz 0d586ef6bf feat: give every file an address by content 2026-10-19 04:40:21 +02:00
Abdessamad Derraz 10cfdb4de0 feat: install the data trees the pack carries
Manifests list each data directory's source archive and the hash of
every file; the installer rebuilds the tree from the archive and keeps
only the files whose hash matches.
2026-10-18 16:18:39 +02:00
Abdessamad Derraz 9ea1630caa feat: install case twins where case is kept
The RetroArch list is folded for Windows and macOS. Its manifest
now carries the names a case-keeping directory also needs, and the
installer adds them after probing the BIOS directory.
2026-10-18 13:52:37 +02:00
Abdessamad Derraz 3c856bbc10 fix: lay out both builds of a dual-mode emulator
Batocera, RetroBat, Recalbox and ROCKNIX offer some emulators both as a
libretro core and standalone. Filed as standalone only, the libretro
layout was dropped: libretro Dolphin had no dolphin-emu/Sys/GC files.
Platforms now name dual-mode emulators and each gets both layouts.
2026-10-18 13:19:53 +02:00
Abdessamad Derraz ae3dc63bc3 fix: accept RomM files on size and any hash
RomM's verify_file_hashes compares the size, then md5, sha1 or crc. We
compared md5 alone, so a fixture entry with one mistyped hash was left out
of the pack RomM would accept it from. The rule is declared in romm.yml
and read by verify, the builder and the manifest.
2026-10-18 07:24:33 +02:00
Abdessamad Derraz 8a9d6919d9 feat: record the pin a prose reread confirmed
A prose citation whose text lived under several pins stayed in
read again until the text changed, so fixnes, zesarux and emucorex,
exact at their pins, would have refused every later bump.
prose_written_at names the pin a reader confirmed; it settles the
readings and never overrides a history with one candidate.
2026-10-18 02:49:24 +02:00
Abdessamad Derraz bd6617cc6a fix: describe absent claims in the manifest schema 2026-10-17 23:09:46 +02:00
Abdessamad Derraz 5a3d9bdc9d fix: resolve the DSP ROMs as a pair the code takes
Dolphin and its forks hash dsp_rom.bin and dsp_coef.bin together and
warn on any combination outside known_roms. The profiles now carry the
accepted pairs, and the resolver keeps both files inside the first pair
the collection holds whole: MMJR and Ishiiruka were served the free ROM
beside the official coefficients.
2026-10-17 18:20:00 +02:00
Abdessamad Derraz e09a37d21e feat: keep platform-built cores out of all_libretro
Recalbox compiles libretro/mame at 0.258 and 0.278 itself; typed
libretro, both profiles were resolved for every all_libretro platform
although the buildbot RetroArch installs from has neither. A profile
now names the platforms that build it (built_by), and is reached only
when a platform lists it by name.
2026-10-17 04:44:25 +02:00
Abdessamad Derraz e659228777 fix: let target installs match their target pack 2026-10-17 00:39:57 +02:00
Abdessamad Derraz b2a13b1dda fix: require integer sizes in profile lists 2026-10-16 15:58:43 +02:00