Commit Graph

45 Commits

Author SHA1 Message Date
Abdessamad Derraz
9e184f76fc fix: swanstation ps1_rom.bin accepts multiple sizes
SwanStation accepts PS1 (512KB), PS2 (4MB), and PS3 (0x3E66F0)
BIOS sizes but only uses the first 512KB. MD5 validates the
extracted content, not the full file. List all accepted sizes
to eliminate the false size mismatch discrepancy.

validation.py: support size as list in emulator profiles.
generate_site.py: handle list sizes in emulator page display.

All 18 original hash mismatches are now resolved: 0 discrepancies.
2026-04-02 07:39:30 +02:00
Abdessamad Derraz
95b7a9813c feat: add custom site stylesheet with theme and components 2026-04-02 00:48:02 +02:00
Abdessamad Derraz
22829cfab9 feat: improve site ux with cards, badges and hash truncation 2026-04-02 00:46:21 +02:00
Abdessamad Derraz
91925120c9 feat: unify gap analysis with verify results and source provenance
Single source of truth for gap page: verification status from
verify.py (verified/untested/missing/mismatch), file provenance
from cross_reference (bios/data/large_file/missing).

cross_reference.py: _find_in_repo -> _resolve_source returning
source category, stop skipping storage: release/large_file,
add by_path_suffix lookup, all_declared param for global check.

generate_site.py: gap page now shows verification by platform,
18 hash mismatches, and core complement with provenance breakdown.
2026-04-01 22:33:37 +02:00
Abdessamad Derraz
0a272dc4e9 chore: lint and format entire codebase
Run ruff check --fix: remove unused imports (F401), fix f-strings
without placeholders (F541), remove unused variables (F841), fix
duplicate dict key (F601).

Run isort --profile black: normalize import ordering across all files.

Run ruff format: apply consistent formatting (black-compatible) to
all 58 Python files.

3 intentional E402 remain (imports after require_yaml() must execute
after yaml is available).
2026-04-01 13:17:55 +02:00
Abdessamad Derraz
a2d30557e4 chore: remove unused imports from generate_site.py 2026-04-01 13:13:06 +02:00
Abdessamad Derraz
6eca4c416a chore: remove dead wiki generator functions, update docs
Remove 4 unused functions from generate_site.py (generate_wiki_index,
generate_wiki_architecture, generate_wiki_tools, generate_wiki_profiling)
that contained stale data. Wiki pages are sourced from wiki/ directory.

Update generate_site.py contributing section with correct test counts
(249 total, 186 E2E, 8 pack integrity) and pack integrity documentation.
2026-04-01 13:05:34 +02:00
Abdessamad Derraz
e5859eb761 refactor: dry pack integrity into cli and update docs
Move verification logic to generate_pack.py --verify-packs (single
source of truth). test_pack_integrity.py is now a thin wrapper that
calls the CLI. Pipeline step 6/8 uses the same CLI entry point.

Renumber all pipeline steps 1-8 (was skipping from 5 to 8/9).

Update generate_site.py with pack integrity test documentation.
2026-04-01 12:31:10 +02:00
Abdessamad Derraz
03002515fe fix: readme and site coverage use data_dir and supplemental indexes 2026-03-31 13:58:52 +02:00
Abdessamad Derraz
d0dd05ddf6 docs: add wiki pages for all audiences, fix .old.yml leak
9 new wiki pages: getting-started, faq, troubleshooting,
advanced-usage, verification-modes, adding-a-platform,
adding-a-scraper, testing-guide, release-process.

Updated architecture.md with mermaid diagrams, tools.md with
full pipeline and target/exporter sections, profiling.md with
missing fields, index.md with glossary and nav links.

Expanded CONTRIBUTING.md from stub to full contributor guide.

Filter .old.yml from load_emulator_profiles, generate_db alias
collection, and generate_readme counts. Fix BizHawk sha1 mode
in tools.md, fix RetroPie path, fix export_truth.py typos.
2026-03-30 23:58:12 +02:00
Abdessamad Derraz
eafabd20f3 refactor: skip writing generated files when content unchanged
write_if_changed in common.py compares content after stripping
timestamps (generated_at, Auto-generated on, Generated on).
applied to generate_db, generate_readme, generate_site.
eliminates timestamp-only diffs in database.json, README.md,
mkdocs.yml, and 423 docs pages.
2026-03-30 14:33:44 +02:00
Abdessamad Derraz
2e21d64a08 refactor: harden codebase and remove unicode artifacts
- fix urllib.parse.quote import (was urllib.request.quote)
- add operator precedence parens in generate_pack dedup check
- narrow bare except to specific types in batocera target scraper
- cache load_platform_config and build_zip_contents_index results
- add selective algorithm support to compute_hashes
- atomic write for fetch_large_file (tmp + rename)
- add response size limit to base scraper fetch
- extract build_target_cores_cache to common.py (dedup verify/pack)
- hoist _build_supplemental_index out of per-platform loop
- migrate function-attribute caches to module-level dicts
- add @abstractmethod to BaseTargetScraper.fetch_targets
- remove backward-compat re-exports from common.py
- replace em-dashes and unicode arrows with ASCII equivalents
- remove decorative section dividers and obvious comments
2026-03-29 23:15:20 +02:00
Abdessamad Derraz
b4c5d77e4b refactor: deduplicate yaml import pattern via require_yaml() 2026-03-29 17:07:27 +02:00
Abdessamad Derraz
f1855641c5 docs: add granular pack options to tools reference 2026-03-28 09:29:31 +01:00
Abdessamad Derraz
d2cc9b8f29 feat: add doom engine wad files, emulatorjs base config 2026-03-25 23:12:53 +01:00
Abdessamad Derraz
4b09205bc9 fix: zero warnings on mkdocs build, update actions to v4/v5 2026-03-25 16:30:08 +01:00
Abdessamad Derraz
c5b267a6fb fix: anchor mismatches in platform and emulator index pages 2026-03-25 16:23:25 +01:00
Abdessamad Derraz
851f53ba7f refactor: extract wiki to source files, use deploy-pages action 2026-03-25 16:02:11 +01:00
Abdessamad Derraz
a6150a43bd feat: group emulators by classification, slim mkdocs nav, add pymdownx extensions 2026-03-25 15:29:58 +01:00
Abdessamad Derraz
0196fff8c7 feat: improve site UX (quick start, system summary, collapsible sections, wiki index, actionable gaps) 2026-03-25 15:24:38 +01:00
Abdessamad Derraz
904edd65e4 docs: document CI workflows, inheritance, MAME clones, tests, scrapers 2026-03-25 15:14:29 +01:00
Abdessamad Derraz
02a7c58fca docs: complete wiki coverage, document all scripts and edge cases 2026-03-25 15:02:23 +01:00
Abdessamad Derraz
f8a325260f feat: add wiki pages (architecture, tools, profiling, data model) 2026-03-25 14:56:37 +01:00
Abdessamad Derraz
313637663a docs: soften tone, explain methodology without dismissing other sources 2026-03-25 14:51:57 +01:00
Abdessamad Derraz
593466b655 feat: add methodology and ground truth narrative to readme and site 2026-03-25 14:50:09 +01:00
Abdessamad Derraz
23d76d54fd fix: correct rendering of complex YAML fields in site pages 2026-03-25 14:38:40 +01:00
Abdessamad Derraz
1cd43c3224 feat: exploit 100% of emulator YAML fields in site generation 2026-03-25 14:32:48 +01:00
Abdessamad Derraz
b3c1462a5e feat: exploit all emulator YAML fields in site generation 2026-03-25 14:28:02 +01:00
Abdessamad Derraz
75bfd04687 feat: full cross-linking web between all site pages 2026-03-25 14:17:10 +01:00
Abdessamad Derraz
3d2762bbc3 feat: cross-reference platform -> core -> systems -> upstream 2026-03-25 14:03:36 +01:00
Abdessamad Derraz
0f4fed2f47 feat: enrich site with full YAML data, cross-references, classification stats 2026-03-25 13:56:28 +01:00
Abdessamad Derraz
6ee162f8fb chore: add MAME and RetroDECK ROM sets 2026-03-19 23:26:49 +01:00
Abdessamad Derraz
257ec1a527 fix: round 2 audit fixes, updated emulator profiles
Scripts:
- fix generate_site nav regex destroying mkdocs.yml content
- fix auto_fetch comma-separated MD5 in find_missing
- fix verify print_platform_result conflating untested/missing
- fix validate_pr path traversal and symlink check
- fix batocera_scraper brace counting and escaped quotes in strings
- fix emudeck_scraper hash search crossing function boundaries
- fix pipeline.py cwd to repo root via Path(__file__)
- normalize SHA1 comparison to lowercase in generate_pack

Emulator profiles:
- emux_gb/nes/sms: reclassify from alias to standalone profiles
- ep128emu: remove .info-only files not referenced in source
- fbalpha2012 variants: full source-verified profiles
- fbneo_cps12: add new profile
2026-03-19 15:00:18 +01:00
Abdessamad Derraz
38d605c7d5 fix: audit fixes across verify, pack, security, and performance
- fix KeyError in compute_coverage (generate_readme, generate_site)
- fix comma-separated MD5 handling in generate_pack check_inside_zip
- fix _verify_file_hash to handle multi-MD5 for large files
- fix external downloads not tracked in seen_destinations/file_status
- fix tar path traversal in _is_safe_tar_member (refresh_data_dirs)
- fix predictable tmp path in download.py
- fix _sanitize_path to filter "." components
- remove blanket data_dir suppression in find_undeclared_files
- remove blanket data_dir suppression in cross_reference
- add status_counts to verify_platform return value
- add md5_composite cache for repeated ZIP hashing
2026-03-19 14:04:34 +01:00
Abdessamad Derraz
08f68e792d refactor: centralize hash logic, fix circular imports and perf bottlenecks 2026-03-18 11:51:12 +01:00
Abdessamad Derraz
81278bd2e4 fix: system icons (systematic theme), retropie logo 2026-03-18 11:25:14 +01:00
Abdessamad Derraz
a52ab19cf8 fix: full hashes, list format for system files 2026-03-18 11:15:11 +01:00
Abdessamad Derraz
300e5d7439 fix: redesign home page UX, fix broken retropie logo 2026-03-18 11:09:36 +01:00
Abdessamad Derraz
54c0f1d27e refactor: review fixes, DRY coverage, filter test nav
- Extract compute_coverage to common.py (was duplicated)
- Filter test cores from nav and emulator index
- Use absolute URL for README download links
- Consistent page titles with site name suffix
- Safer mkdocs.yml nav rewrite with regex
- Build all_platform_names once in gap analysis
2026-03-18 11:05:13 +01:00
Abdessamad Derraz
e218763500 feat: add emulator logos to profiles and site 2026-03-18 10:57:00 +01:00
Abdessamad Derraz
6885681c65 feat: add platform logos to registry and site 2026-03-18 10:55:47 +01:00
Abdessamad Derraz
32e4f6e580 fix: review fixes for generate_site.py 2026-03-18 10:39:23 +01:00
Abdessamad Derraz
0b1ed3cb1a feat: add gap analysis page + platform tracking 2026-03-18 10:31:02 +01:00
Abdessamad Derraz
883e153a62 fix: clean platform/emulator page layout 2026-03-18 10:27:08 +01:00
Abdessamad Derraz
b15b062782 feat: add mkdocs site generator, 332 pages
generate_site.py reads database.json + platforms/*.yml + emulators/*.yml
and produces a complete MkDocs Material documentation site:
- Home: stats, downloads, coverage dashboard
- 7 platform pages with per-file verification status
- 60 system pages grouped by manufacturer with cross-references
- 260 emulator pages with source code analysis
- Contributing guide

mkdocs.yml with Material theme, system fonts, auto dark mode.
Generated docs/ in .gitignore (built in CI only).
2026-03-18 10:22:00 +01:00