Commit Graph

239 Commits

Author SHA1 Message Date
Roël Couwenberg
32b391ef69 feat: Addition of a RomM package (#37)
* feat: Initial addition of the RomM scraper and package

* docs: Added RomM to the verification modes table

* chore: Clean up scraper by removing workaround code from main() and an unused system translation map
2026-03-25 19:24:59 +01:00
Abdessamad Derraz
1ad10eddb7 feat: include platform version in pack filenames 2026-03-25 18:55:35 +01:00
Abdessamad Derraz
69131f4ad1 fix: emulator validation is informational, not a platform failure 2026-03-25 17:34:56 +01:00
Abdessamad Derraz
c61d40d8ef docs: rewrite readme for clarity and discoverability 2026-03-25 17:18:43 +01:00
Abdessamad Derraz
82ed2c5e62 fix: deploy-pages v5 does not exist, revert to v4 2026-03-25 16:57:01 +01:00
Abdessamad Derraz
38059e0fdf chore: regenerate database, readme, mkdocs nav 2026-03-25 16:49:49 +01:00
Abdessamad Derraz
dfd8f8657c feat: re-profile 4 emulators, add ymir 2026-03-25 16:49:49 +01:00
Abdessamad Derraz
4b09205bc9 fix: zero warnings on mkdocs build, update actions to v4/v5 2026-03-25 16:30:08 +01:00
Abdessamad Derraz
c5b267a6fb fix: anchor mismatches in platform and emulator index pages 2026-03-25 16:23:25 +01:00
Abdessamad Derraz
851f53ba7f refactor: extract wiki to source files, use deploy-pages action 2026-03-25 16:02:11 +01:00
Abdessamad Derraz
6e237e3b6d chore: regenerate database, readme, mkdocs nav 2026-03-25 15:54:17 +01:00
Abdessamad Derraz
9765bbc3da feat: add site deploy workflow, triggers only on content changes 2026-03-25 15:53:20 +01:00
Abdessamad Derraz
5529f8474a chore: regenerate database and readme 2026-03-25 15:41:46 +01:00
Abdessamad Derraz
8a8765c0ee feat: re-profile 9 emulators, add 9 vice machine profiles 2026-03-25 15:41:36 +01:00
Abdessamad Derraz
a6150a43bd feat: group emulators by classification, slim mkdocs nav, add pymdownx extensions 2026-03-25 15:29:58 +01:00
Abdessamad Derraz
0196fff8c7 feat: improve site UX (quick start, system summary, collapsible sections, wiki index, actionable gaps) 2026-03-25 15:24:38 +01:00
Abdessamad Derraz
904edd65e4 docs: document CI workflows, inheritance, MAME clones, tests, scrapers 2026-03-25 15:14:29 +01:00
Abdessamad Derraz
02a7c58fca docs: complete wiki coverage, document all scripts and edge cases 2026-03-25 15:02:23 +01:00
Abdessamad Derraz
f8a325260f feat: add wiki pages (architecture, tools, profiling, data model) 2026-03-25 14:56:37 +01:00
Abdessamad Derraz
313637663a docs: soften tone, explain methodology without dismissing other sources 2026-03-25 14:51:57 +01:00
Abdessamad Derraz
593466b655 feat: add methodology and ground truth narrative to readme and site 2026-03-25 14:50:09 +01:00
Abdessamad Derraz
23d76d54fd fix: correct rendering of complex YAML fields in site pages 2026-03-25 14:38:40 +01:00
Abdessamad Derraz
1cd43c3224 feat: exploit 100% of emulator YAML fields in site generation 2026-03-25 14:32:48 +01:00
Abdessamad Derraz
b3c1462a5e feat: exploit all emulator YAML fields in site generation 2026-03-25 14:28:02 +01:00
Abdessamad Derraz
75bfd04687 feat: full cross-linking web between all site pages 2026-03-25 14:17:10 +01:00
Abdessamad Derraz
3d2762bbc3 feat: cross-reference platform -> core -> systems -> upstream 2026-03-25 14:03:36 +01:00
Abdessamad Derraz
0f4fed2f47 feat: enrich site with full YAML data, cross-references, classification stats 2026-03-25 13:56:28 +01:00
Abdessamad Derraz
dbc26b11c1 refactor: move fetch_large_file to common, auto-download on db rebuild 2026-03-25 13:19:12 +01:00
Abdessamad Derraz
910428c6f1 fix: resolve large files from cache in database paths 2026-03-25 12:52:20 +01:00
Abdessamad Derraz
21465effff feat: add readme and site generation to pipeline 2026-03-25 12:34:03 +01:00
Abdessamad Derraz
3cba6b5c01 chore: disable build workflow until validated 2026-03-25 12:27:29 +01:00
Abdessamad Derraz
cfec0c614f Revert "chore: disable auto-trigger on push, manual release only"
This reverts commit d4e5af2c74.
2026-03-25 12:26:39 +01:00
Abdessamad Derraz
d4e5af2c74 chore: disable auto-trigger on push, manual release only 2026-03-25 12:25:44 +01:00
Abdessamad Derraz
f27b3b6caf refactor: simplify build workflow, remove auto-commit 2026-03-25 12:25:13 +01:00
Abdessamad Derraz
47e6174ed4 fix: pack naming, large file preservation, discrepancy reporting 2026-03-25 12:23:40 +01:00
Abdessamad Derraz
a7bcd9b252 feat: re-profile 20 emulators, add trident and vice_x128 2026-03-25 12:23:32 +01:00
Abdessamad Derraz
93b071730a fix: fallback to system tempdir for zip normalization 2026-03-25 08:38:06 +01:00
Abdessamad Derraz
ebb55a445b feat: re-profile 40 emulators, harden CI workflows
profile emulators pd777 through tic80, add frozen snapshots
(puae2021, snes9x2002/2005/2010, stella2014/2023).

CI: replace github-script with gh CLI, add test execution,
job-level permissions, propagate changed output, pin jsonschema.
2026-03-25 07:00:17 +01:00
Abdessamad Derraz
0543165ed2 feat: re-profile 22 emulators, refactor validation to common.py
batch re-profiled nekop2 through pokemini. mupen64plus renamed to
mupen64plus_next. new profiles: nes, mupen64plus_next.
validation functions (_build_validation_index, check_file_validation)
consolidated in common.py — single source of truth for verify.py
and generate_pack.py. pipeline 100% consistent on all 6 platforms.
2026-03-24 22:31:22 +01:00
Abdessamad Derraz
94000bdaef fix: align verify and pack validation, pipeline 100% consistent
generate_pack.py now applies emulator-level validation (crc32, sha1,
adler32) matching verify.py behavior. existence mode: validation is
informational (file present = OK). md5 mode: validation downgrades
to UNTESTED. clone resolution moved to common.py resolve_local_file.
all 6 platforms pass consistency check.
2026-03-24 22:21:47 +01:00
Abdessamad Derraz
ae4846550f fix: clone resolution in common.py, move clone map to root
moved _mame_clones.json out of bios/ (was indexed by generate_db.py
as BIOS file). clone resolution now in common.py resolve_local_file
so all tools (verify, pack, cross_reference) resolve clones
transparently. removed duplicate clone code from generate_pack.py.
added error handling on os.remove in dedup.py. consistency check
now passes for Batocera/EmuDeck/Lakka/RetroArch (4/6 platforms).
2026-03-24 21:57:49 +01:00
Abdessamad Derraz
85308edd73 fix: dedup edge cases — preserve non-ZIP different-name files
non-ZIP files with different names but same content (64DD_IPL_US.n64
vs IPL_USA.n64) are now preserved — each name may be needed by a
different emulator. only same-name duplicates and MAME ZIP clones
are removed. added empty directory cleanup post-dedup.
2026-03-24 21:39:25 +01:00
Abdessamad Derraz
fb1007496d chore: deduplicate bios/ — remove 427 files, save 227 MB
true duplicates (same file in multiple dirs): removed copies, kept
canonical. MAME device clones (different names, identical content):
removed copies, created _mame_clones.json mapping for pack-time
assembly via deterministic ZIP rebuild. generate_pack.py resolves
clones transparently. 95 canonical ZIPs serve 392 clone names.
2026-03-24 21:35:50 +01:00
Abdessamad Derraz
8fcb86ba35 feat: deterministic MAME ZIP assembly in packs
all ZIP files (neogeo.zip, pgm.zip, etc.) are rebuilt with fixed
metadata before packing: sorted filenames, epoch timestamps, fixed
permissions, deflate level 9. same ROM atoms = same ZIP hash, always.
115 internal ZIPs verified identical across two independent builds.
enables version-agnostic ZIP assembly from ROM atoms indexed by CRC32.
2026-03-24 15:17:12 +01:00
Abdessamad Derraz
34e4c36f1c feat: pack integrity verification, manifests, SHA256SUMS
post-generation verification: reopen each ZIP, hash every file,
check against database.json. inject manifest.json inside each pack
(self-documenting: path, sha1, md5, size, status per file).
generate SHA256SUMS.txt alongside packs for download verification.

validation index now uses sets for hashes and sizes to support
multiple valid ROM versions (MT-32 v1.04-v2.07, CM-32L variants).
69 tests pass, pipeline complete.
2026-03-24 14:56:02 +01:00
Abdessamad Derraz
69987cc0bb feat: re-profile 30 emulators, beetle to parallel_n64
source-verified: mesen FdsBios.bin→alias, MesenDB.txt phantom (embedded).
mupen64plus 64DD_IPL.bin→IPL.n64 (code path fix). neocd 000-lo.lo/ng-lo.rom
phantoms (algorithmically generated). new profiles: mesen-s, mupen64plus
next develop/gles2/gles3. regenerate database.
2026-03-24 14:06:20 +01:00
Abdessamad Derraz
cda012b10a fix: source-verified corrections from review
dosbox/dosbox_pure: added validation: [sha1, size] to MT-32 ROM
entries — MUNT rejects unrecognized SHA1 (ROMInfo.cpp:171-180).
duckstation: fixed BIOS count to 105 (24 PS1 + 80 PS2 + 1 OpenBIOS).
added exclusion_note to 6 profiles with files: [] (dosbox_svn,
dosbox_svn_ce, doukutsu_rs, emux_chip8, emux_nes, fake08).
2026-03-24 12:21:32 +01:00
Abdessamad Derraz
491ea2f9ba fix: source-verified corrections on 7 profiles
beetle_pce: +core_classification, +upstream, cores=[mednafen_pce],
alt_names→aliases. 6 files confirmed loaded by code.
mame: removed 19 phantom XaviX entries (MACHINE_IS_BIOS_ROOT commented
out in source). 80 real BIOS sets verified.
mame2010: fixed galgbios.zip source_ref, documented cheat.zip as
non-functional in libretro port (cheatpath never initialized).
ishiiruka: trimmed verbose advisory notes to factual descriptions.
mamearcade: added system: field to 29 bios_zip entries, 24 new systems.
mednafen_ngp: removed redundant cores: on alias, added note:.
mednafen_gba: normalized validation format to flow sequence.
2026-03-24 12:17:45 +01:00
Abdessamad Derraz
11db9892bf feat: add sha256 validation support to verify.py 2026-03-24 11:49:58 +01:00
Abdessamad Derraz
685713a7e6 feat: add sect233r1 ECDSA verification for 3DS OTP cert
pure python GF(2^233) field arithmetic, binary curve point operations,
and ECDSA-SHA256 on sect233r1. verifies OTP CTCert against nintendo
root CA public key. zero dependencies. sign+verify round-trip tested,
n*G=O verified, wrong key/message rejection confirmed.
2026-03-24 11:45:16 +01:00