Commit Graph
1616 Commits
Author SHA1 Message Date
Abdessamad Derraz 468b5e1d3e feat: keep a replaced BIOS file beside its folder
A file in place whose contents differ (another revision, a personal
dump) was overwritten and lost. It is now moved to
retrobios-backup/<date-time>/ beside the folder written to, under the
same path, before the verified copy replaces it. Beside, not inside:
DuckStation, SwanStation and Rustation read every file of the BIOS
folder and BizHawk walks its firmware folder recursively.

Identical files are never kept. --dry-run counts what would be kept,
the summary and the summary event name the folder, files and bytes,
and --restore FOLDER puts the files back. Data trees and standalone
copies keep their replaced files the same way.
2026-10-22 14:00:00 +02:00
Abdessamad Derraz 274fc37673 chore: merge the site weight and speed work 2026-10-22 13:46:50 +02:00
Abdessamad Derraz be5be996db feat: hold the site to page and index budgets
validate_site.py fails a page over 1 MB of HTML, a search index over
8.5 MB (1.7 MB gzip), a skipped heading level, a table header without
text, a script from another host and a diagram drawn at runtime. The
limits sit above what the site now measures (0.88 MB, 7.2 MB, 1.43 MB),
each with its reason, and each has a test.
2026-10-22 13:13:55 +02:00
Abdessamad Derraz 36ac7b0cd6 feat: split the heavy site pages
A manufacturer page over 500 KB of Markdown becomes a summary: its
largest consoles move to pages of their own, cut into parts when one
console is still too large, and every console keeps its heading there.
MAME's 1,564 cards move to four pages, its page keeping a row per entry
with the card's old anchor. RetroDECK's arcade list gets its own page.
site.js follows a moved card by content or by its row.

The largest page drops from 10.5 MB to 0.88 MB. Emulator groups of the
navigation open by initial (65 KB to 41 KB before the content). File
cards are level 3 under a level 2, and the hook keeps the table of
contents of card pages to level 2: platform pages no longer list every
system id. Key/value cards name their columns. The sweep runs before
the decoration, which kept every page no generator wrote any more.
2026-10-22 13:13:52 +02:00
Abdessamad Derraz 2d2addd6d2 fix: align the suite with the site layout
The redesign moved the technical questions to technical-faq.md, made the Get your files rows generated from each platform key and reduced getting-started.md to links. Tests now read those places: the pack-row test reads the rendered page. The new site scripts go through run_cli and refuse abbreviated options. The three maintenance image tools are exempt by name from the build dependency rule: their output is committed and the build only reads it. siteguides.py now triggers the site deploy, and the recipe counts match the store.
2026-10-22 13:11:07 +02:00
Abdessamad Derraz 33041b6bd1 feat: find a file by any hash, out of the search
Hashes, link lists, install commands, member tables and long reference
tables leave the site search: the index drops from 21.0 MB (4.88 MB
gzip) to 6.8 MB (1.33 MB gzip). Names and prose stay searchable.

files/?hash= takes a SHA1, MD5, CRC32 or SHA256 and reads one shard of
api/v1/lookup/ (about 22 KB gzip, was 675 KB) listing the collection
card, the platforms that declare the value and the profiles that check
it. files/?sha1= still opens the card, and a sha1- anchor missing from
a page is looked up there, so a card can move between pages.

Excluded blocks are section elements: the search plugin compares
elements by tag, so a div ended at the first nested div.
2026-10-22 12:46:40 +02:00
Abdessamad Derraz 42a0498d20 chore: run site tools through run_cli
The image and diagram scripts accepted abbreviated options and bypassed
run_cli. The FAQ checks read faq.md only, while the questions on hashes
and checks moved to technical-faq.md.
2026-10-22 12:46:34 +02:00
Abdessamad Derraz 427a8183e3 feat: draw the site mark and favicons as vectors
The header logo was a 256 px compressed raster whose navy chip vanished on the indigo bar. The chip is now an SVG, white with the glyph cut out, and the favicon set (SVG, 32 px and an opaque 180 px touch icon) comes from scripts/make_site_icons.py.
2026-10-22 12:34:44 +02:00
Abdessamad Derraz baf38ffbee refactor: compute coverage once, index proofs
EntryCategories asked every core in turn: 3.8 million proofs for 7,430
entries. ProofIndex looks each key up once, and verify --all --json is
byte-identical (24 s to 17 s). generate_site and generate_readme now share
one coverage, cached in tmp/coverage-cache keyed by its inputs, so the
README step drops from 25 s to 3 s.
2026-10-22 12:23:26 +02:00
Abdessamad Derraz f67f4ead1b chore: align the merged site with its checks
The two redesign branches and the archive repair met in the testing guide and the prose rules: list test_archive_recipes, drop three clause semicolons, write the forbidden glyphs as escapes in the image test. README and mkdocs.yml regenerated from the merged generator.
2026-10-22 12:04:42 +02:00
Abdessamad Derraz f39bcebb22 chore: merge the site visual system 2026-10-22 12:00:19 +02:00
Abdessamad Derraz db1d679e4e chore: merge the site funnel 2026-10-22 11:59:37 +02:00
Abdessamad Derraz cea5a3d0b2 fix: count the Marathon data trees as game data
bios/Bungie holds the Marathon scenarios Aleph One reads, entries its profile files as category game_data. Listed with the other publishers' trees, they leave the system files count and sit under Game engines and PC games on the site.
2026-10-22 11:54:03 +02:00
Abdessamad Derraz 6353ee53df feat: lay the site out as a reading funnel
Five tabs from Start here to Contribute, a level on every page, a device picker on Get your files that lists every registered platform, and platform pages that open on where the files go, the command, the pack and how to check. FAQ and troubleshooting keep the questions asked before installing, the technical ones move to their own page, and getting started points to the pages that took its content.
2026-10-22 11:50:53 +02:00
Abdessamad Derraz 533b85586f feat: group platforms by the device they run on
device_group in the registry places each platform on the device picker, in the navigation and on the home page. scripts/siteguides.py holds the guide text per platform: what it is, its BIOS folder, where the install command runs.
2026-10-22 11:50:48 +02:00
Abdessamad Derraz 1b9ea86e1b feat: draw site images from data, host them
scripts/sitediagrams.py writes the explainer, install flow, verification modes, MinUI card, pipeline and data layer diagrams as inline SVG from the registry, the platform files, the install manifests and the pipeline step list. A page asks for one with an image comment, which the decoration pass resolves. The two mermaid diagrams of the architecture page are replaced, so no page loads mermaid from unpkg.

Platform logos and system icons are copied once by fetch_site_images.py with their source and owner recorded, and the build rewrites the remote links to the copies. validate_site refuses an image from another host. Three registry logos named a personal avatar instead of the project's (RomM, ROCKNIX, MiSTer FPGA) and MyMinUI's was a GitHub identicon, now dropped.

One 1200x630 sharing card per section, a 404 page with search and the two entry pages, and a list of screenshots that need a real device.
2026-10-22 11:45:47 +02:00
Abdessamad Derraz 57cbff781a chore: merge the remaining platform core profiles 2026-10-22 11:15:32 +02:00
Abdessamad Derraz 87a4e8e1b6 fix: send unreleased platforms to the installer
A release recorded without a platform's pack linked its button to that release, which serves no pack for AmberELEC, EmuELEC, KNULLI, MinUI, NextUI or MyMinUI. The cell now names the release and points to the installer, which reads the file lists on main. Core folder links resolve each core name to the profile page that describes it (mednafen_vb is beetle_vb).
2026-10-22 11:04:25 +02:00
Abdessamad Derraz 4501601769 fix: show where each platform's pack goes
Platform pages labelled base_destination, the root the platform's own list writes from, as the BIOS path, so EmuDeck read bios/ where the pack goes into ~/Emulation/bios/. The row now gives the registry's extract_to, the folder the README and the pack guide name. A platform that names no file (MinUI and its forks) showed 0/0 coverage and an empty systems table; its page now lists each core folder with the files the pack puts in it.
2026-10-22 10:56:35 +02:00
Abdessamad Derraz 3276754102 fix: end profile history at a deleted namesake 2026-10-22 08:15:15 +02:00
Abdessamad Derraz e1aac191e7 fix: skip platforms with no native list
export_native counted MinUI, NextUI and MyMinUI as failures although the registry marks them native_list: none: their code names no BIOS file, so there is no list of theirs to correct.
2026-10-22 07:52:08 +02:00
Abdessamad Derraz cd7d993cac feat: settle entries by their system's cores
A system that lists the cores it runs is settled by those cores alone, so an unprofiled core of another system no longer leaves its entries open. Readers whose core runs the entry's system are named first, and cores that only share the file name are listed as such.
2026-10-22 06:30:56 +02:00
Abdessamad Derraz 4bc90cf06e chore: merge the elec core names 2026-10-22 06:29:22 +02:00
Abdessamad Derraz 9648b9fc3b feat: read elec ports and per-system cores
EmuELEC's ports system has one emulator, `ports`, the dispatch to a
script ports.yaml generates. The scraper now reads each port's engine
(the libretro core handed to emuelecRunEmu.sh with -C, else port_exe),
so devilutionx, sonic2013, soniccd, sonic-mania, reminiscence and
rigelengine are examined. Standalone emulators keep the spelling the
launch code dispatches on instead of invented lowercase forms, and
media players and runtimes go to the overrides' remove_cores.

Each system of the AmberELEC, EmuELEC and KNULLI lists now carries the
cores its ES list offers it, in menu order, under `cores:`.
2026-10-22 06:28:24 +02:00
Abdessamad Derraz a0b27d612d fix: align tests with the ten-value pack result
Two test modules still unpacked nine values from verify_pack_against_platform. The category evidence joined readers with semicolons, which the prose rule refuses on generated pages, and three comments used clause semicolons.
2026-10-22 06:23:15 +02:00
Abdessamad Derraz 8ba65de1ca chore: merge the game data filter 2026-10-22 06:19:55 +02:00
Abdessamad Derraz aed958ccd1 docs: document the game data filter 2026-10-22 06:19:13 +02:00
Abdessamad Derraz 8b952968d6 feat: add a --no-game-data pack narrowing
The install manifest marks each game data file "category": "game_data":
a platform entry only game data profile entries read, or a core file its
profile declares game data. Firmware carries no category. Every manifest
says "categorized": true so the installer can refuse an older list.

generate_pack.py --no-game-data cuts those files from the platform list
and the core profiles before the build, names each one with the cores that
read it as game data, and tags the pack _NoGameData through _narrowings.
verify.py and the pack check take the same cut, so --verify-packs and the
report describe the pack the builder wrote. Modes without a platform
(--emulator, --system alone, --from-md5) and --all-variants refuse it.
2026-10-22 06:19:05 +02:00
Abdessamad Derraz be9036f2b2 feat: document the category of every entry
Held entries carry their category and readers too, not only missing ones, so each platform page says which cores read every file it lists and where the code does it. Severity still applies to firmware gaps only.
2026-10-22 05:46:09 +02:00
Abdessamad Derraz a75ea05b86 feat: document what settles each category
Each missing platform entry now carries its evidence: the profile entries that read it with their source reference, the cores of its system examined without finding a reader, or why the question stays open. verify --verbose prints it, --json carries it and the platform pages show it.
2026-10-22 05:43:06 +02:00
Abdessamad Derraz f66adfdcbc feat: categorize missing platform entries
A platform list can name a game's data files or a file no core it runs opens. Verify gives each missing entry a category read from the profiles of the platform's cores: firmware, game data or unread. Only firmware gaps carry a severity and fail --strict; the others are counted on their own line, and the pack check no longer counts them as pack errors. Missing firmware stays a conformance error.
2026-10-22 05:38:57 +02:00
Abdessamad Derraz b7b407698f fix: count files the collection lacks apart
The pack check failed a pack for a declared file nothing in the collection resolves to, though the builder cannot ship it and verify already reports it missing. Such destinations are now counted as not in the collection; a held file absent from the pack stays an error.
2026-10-22 04:28:43 +02:00
Abdessamad Derraz e05844e611 chore: merge the handheld port 2026-10-22 04:12:09 +02:00
Abdessamad Derraz 841017c887 chore: merge the minui family platforms 2026-10-22 04:09:17 +02:00
Abdessamad Derraz 43bdf83973 feat: add minui, nextui and myminui platforms
minarch hands every core SDCARD_PATH/Bios/<TAG> as its system directory
and opens no file in it. The scrapers read each pak's launch.sh (core,
tag, devices), each device's core makefile (the project a library is
built from: MyMinUI's dosbox_libretro.so is dosbox-pure) and the minarch
statement, at a pinned commit. The installer recognises the card from
.system/version.txt and refuses a folder inside it.
2026-10-22 04:04:36 +02:00
Abdessamad Derraz d0802c5360 feat: place core files per handed system dir
A platform can hand each libretro core a system directory of its own
(core_system_dirs). platform_layouts then gives one view per directory,
so the gap report, the builder, the manifest, slots and cross_reference
place a core's files below each folder it is handed, and a core handed
none gets no file. Resolution strips the folder again and searches the
collection by the core's own path.
2026-10-22 04:04:31 +02:00
Abdessamad Derraz 7d87828b98 docs: explain the handheld port
The installer page says how to get the port (PortMaster's catalog once
accepted, or RetroBIOS.zip in the autoinstall folder), what it shows and
needs, and why it carries install.py. The download page links to it
from the handheld table.
2026-10-22 03:59:12 +02:00
Abdessamad Derraz 87c81e9694 feat: ship the handheld port with each release
build_port.py runs after split_pack.py, so RetroBIOS.zip is in the
signed checksum list and uploaded with the packs. check_release.py takes
the port from that list: it is no pack, so release.json does not record
it.
2026-10-22 03:58:45 +02:00
Abdessamad Derraz ecadb3cb32 feat: add the RetroBIOS PortMaster port
ports/RetroBIOS runs install.py on a handheld with PortMaster's own
screens (pugwash fifo_control through PortMasterDialog.txt), using only
what PortMaster 2024.02.26 and the 8.5.23 copy JELOS ships. It reads the
installer through --events. scripts/build_port.py writes RetroBIOS.zip
in the layout PortMaster's release build gives a port, with the
install.py the bootstraps pin, reproducibly. tests/test_port drives the
screens with stubbed dialog functions and a fake installer.
2026-10-22 03:58:42 +02:00
Abdessamad Derraz 44a9c8d9bb docs: list knulli, amberelec and emuelec
The download page drops a row whose platform has no pack in the
recorded release instead of linking a release that lacks it.
2026-10-22 03:42:25 +02:00
Abdessamad Derraz 2bdac466a9 feat: add knulli, amberelec and emuelec platforms
Each OS publishes Batocera's systems dict (knulli-systems, and
batocera-systems at AmberELEC's newest prerelease commit and EmuELEC v4.8),
read with the shared parser. Cores come from each OS's own ES list.
AmberELEC and EmuELEC declare neogeo/ and ports/ beside bios/ from
/storage/roms, so their packs extract there and core_destination puts
core files under bios/.
2026-10-22 03:42:20 +02:00
Abdessamad Derraz 2fc62bd249 refactor: share the batocera-systems parser
extract_systems_dict, requirements_from_systems, platform_systems and
configgen_cores move to module level so the scrapers of lists with the
same shape read them with the same code.
2026-10-22 03:42:11 +02:00
Abdessamad Derraz f2c0d13ab2 chore: merge round 4 fixes (links)
Both branches extracted RetroBat's Flycast BiosPath: value_folders
already produces the record, so the dedicated extractor is dropped and
its test reads the generic one.
2026-10-21 13:19:44 +02:00
Abdessamad Derraz 43fe5a7b15 chore: merge round 4 fixes (profiles) 2026-10-21 13:17:10 +02:00
Abdessamad Derraz 3e5deed8fd chore: merge round 4 fixes (retrobat) 2026-10-21 13:17:10 +02:00
Abdessamad Derraz d9809efa36 fix: pack emudeck libretro builds as dual mode
RetroArch_installCores (emuDeckRetroArch.sh) downloads flycast, mame,
melonds, mgba, mupen64plus_next, ppsspp and scummvm as libretro cores
next to their standalone apps. Listed as standalone only, they got one
layout: once Flycast's standalone files moved to flycast/bios/, the
libretro core lost dc/airlbios.zip, dc/naomi2.zip and dc/segasp.zip.
The scraper now writes them as dual_mode_cores.
2026-10-21 12:45:36 +02:00
Abdessamad Derraz 8036d36fe8 fix: drop names DraStic never opens
DraStic opens nds_bios_arm9.bin, nds_bios_arm7.bin and nds_firmware.bin and checks their size. Its nine aliases are in none of its binaries, and neither platform launcher passes them: ROCKNIX links the two BIOS names from its bios folder, and RetroPie runs DraStic from $md_conf_root/nds/drastic (drastic.sh:48), now a passes_path record. The collection answers DraStic's names with its DS dumps in generate_db, where repository names belong.
2026-10-21 12:44:33 +02:00
Abdessamad Derraz 20829d09ce fix: place launcher links and platform BIOS paths
A links record naming a file is where the emulator reads it: ROCKNIX links /storage/roms/bios/nds_bios_arm9.bin into DraStic's system folder (start_drastic.sh:37-41). $biosdir, ${biosPath} and $bios_path are the BIOS directory as RetroPie, EmuDeck and RetroDECK write it, and every place of those platforms was dropped as an unknown folder. A core name the launcher replaces is no longer packed when held: EmuDeck passes Complex_4627v1.03.bin to xemu (emuDeckXemu.sh:60-61), which never opens Complex_4627.bin.
2026-10-21 12:44:33 +02:00
Abdessamad Derraz bd6ab38fe1 feat: extract the remaining retrobat generators
The launcher layer now reads the Saturn emulators whose option replaces
the default path, DeSmuME's pair and firmware, the melonDS DSi files and
NAND region, jzIntv arguments, BlastEm's SetBiosPath, Dolphin, Raine,
GSplus, the DOSBox and capsimg copies, Tsugaru's folder chain, RPCS3,
EKA2L1, BizHawk's GB switches and the folders that MAME, FBNeo, Snes9x,
Flycast, ScummVM, XRoar, XM6 Pro, ArcadeDuck, Demul and B-Syndrome get.
Every line of these generators that names the bios folder must be read
by a record or a stated exception, or the scrape stops.
2026-10-21 12:39:15 +02:00
Abdessamad Derraz d5a25190a3 fix: place standalone files in the linked folder
standalone_path is relative to the directories a standalone build
resolves it against, now declared per profile (standalone_directories).
A launcher record that links or passes a whole folder of the BIOS tree
under one of them gives that folder, and platform_layouts rebases the
standalone view once for the builder, the manifest, verify and slots
(launchers.standalone_root, common.standalone_destination).

Flycast on RetroPie and ROCKNIX reads bios/dc, on EmuDeck
bios/flycast/bios, on Batocera (FLYCAST_BIOS_PATH) and RetroBat
(Dreamcast.BiosPath) bios/dc. The ROCKNIX, Batocera and RetroBat
scrapers now record the patched data directory, the environment
variable and the setting that say so.
2026-10-21 12:39:03 +02:00