From 348c56e94c8ede9d8589b4175e7e15edad3ed391 Mon Sep 17 00:00:00 2001 From: Abdessamad Derraz <3028866+Abdess@users.noreply.github.com> Date: Wed, 21 Oct 2026 04:03:31 +0200 Subject: [PATCH] docs: put the project's working rules in the wiki Principles, collection rules and the inverted model in architecture.md, research and citation rules in profiling.md, platform and scraper rules in their pages, tool conventions in tools.md, test and commit expectations in testing-guide.md, installer constraints in installer.md. Each rule sits where a reader looks for it, with its reason. Also fixes the stale release-job line, the suite command without -t ., and the unsourceable definition. --- wiki/adding-a-platform.md | 11 +++ wiki/adding-a-scraper.md | 10 +++ wiki/advanced-usage.md | 8 +- wiki/architecture.md | 131 ++++++++++++++++++++++++++++++++- wiki/installer.md | 16 ++++ wiki/profiling.md | 146 ++++++++++++++++++++++++++++++++++++- wiki/testing-guide.md | 57 ++++++++++++++- wiki/tools.md | 26 ++++++- wiki/verification-modes.md | 12 +++ 9 files changed, 407 insertions(+), 10 deletions(-) diff --git a/wiki/adding-a-platform.md b/wiki/adding-a-platform.md index 80da491d..b74b5379 100644 --- a/wiki/adding-a-platform.md +++ b/wiki/adding-a-platform.md @@ -328,6 +328,13 @@ If a file already exists at the output path, the CLI preserves fields that the scraper does not generate (e.g. `data_directories`, manually added metadata). Only the `systems` section is replaced. +The YAML is the scraper's output and is never edited by hand: the next scrape +would undo the edit, and the file would stop describing the platform. Anything +an export has to write back (the platform's own system id, path, name) is +recorded by the scraper and never reconstructed later, because several of the +platform's systems can fold into one of ours. A rescrape can add systems, so +read `verify.py --platform` before committing the regenerated YAML. + Verify the result: ```bash @@ -451,6 +458,10 @@ For platforms that only run on one target (e.g. RetroBat on Windows, RomM in the browser), create a static YAML file in `platforms/targets/` instead of a scraper. Set `target_scraper: null` in the registry. +Its target writes `cores: null`, which means every core the platform declares. +A core list written by hand there falls behind the scraped platform and drops +cores the pack ships. + ## Step 7: Add install detection (optional) The `install` section in `_registry.yml` tells `install.py` how to detect diff --git a/wiki/adding-a-scraper.md b/wiki/adding-a-scraper.md index e2628a2c..5f898f45 100644 --- a/wiki/adding-a-scraper.md +++ b/wiki/adding-a-scraper.md @@ -211,6 +211,16 @@ System ID consistency matters for cross-platform operations. The same console mu Scraped data reflects what the upstream declares, which may not match reality. The scraper's job is faithful transcription of upstream data, not correction. Corrections happen in the emulator profiles (source-verified) and in `_shared.yml` (curated). If a scraper meets a hash or filename that differs from the emulator source, log a warning and still include the upstream value. The divergence will surface during truth diffing. +A few rules keep that transcription honest: + +- An entry the upstream file does not carry is marked: `shared_group`, + `mirror_of` or `curated`. Unmarked, it would be counted and exported as + something the platform declares. +- A scraper that reads nothing raises. An empty input (a refused API call, a + moved file) must not rewrite a platform or target file as empty. +- Names that YAML would turn into numbers or dates are written quoted, so the + next scrape produces the same file. + ## Creating a target scraper Target scrapers determine which emulator cores are available on each hardware diff --git a/wiki/advanced-usage.md b/wiki/advanced-usage.md index 27b467ee..667c53de 100644 --- a/wiki/advanced-usage.md +++ b/wiki/advanced-usage.md @@ -331,7 +331,7 @@ The diff reports: ### Export to native formats -Rewrite a platform's own BIOS file, corrected. The output is the file its +Write a platform's own BIOS file, updated from the emulators' code. The output is the file its maintainers keep, not our data in their syntax: hashes the truth can prove are applied, entries the truth says nothing about are left alone, and files the truth knows and the platform lacks are added. @@ -342,11 +342,11 @@ python scripts/export_native.py --platform recalbox --fetch # es_bios.xml python scripts/export_native.py --all --fetch # every platform, into exports/ ``` -The corrected files are committed under `exports//`, and +The updated files are committed under `exports//`, and `exports/.json` names every change the run made. The site's [updated platform files](../exports.md) page is built from those reports: -one row per platform, a download link, and each corrected hash, filled hash -and withheld correction listed in full. +one row per platform, a download link, and each updated hash, added hash +and change not applied listed in full. Seven of the formats carry code as well as data: Batocera's and ROCKNIX's scripts, EmuDeck's shell library, BizHawk's C# database, RetroDECK's diff --git a/wiki/architecture.md b/wiki/architecture.md index c9efa1d4..14289f14 100644 --- a/wiki/architecture.md +++ b/wiki/architecture.md @@ -1,5 +1,37 @@ # Architecture +## Principles + +Every part of the toolchain follows from a few rules. + +- **The emulator's source code is the authority.** A profile records what the + code opens, under which names, what it checks and whether it runs without + the file. Documentation, `.info` files and platform lists are starting + points to verify. When they disagree with the code, the code wins. +- **Every statement rests on a fact anyone can check.** A field, a deletion or + a wording names what settles it: a `file:line` at a pinned revision, a hash, + a command and its output. Where no fact settles a question, the answer is + "not verified" until one is found. A sentence on this site once said + Batocera matches md5 prefixes. Its script compares the full string + (`batocera-systems:1088`). +- **The collection serves the user.** A pack carries what the user has to + obtain to run their emulators. When something forces the user or a + contributor to work around the toolchain, the toolchain is changed rather + than the workaround documented. +- **Profiles document the emulator, not this repository.** A profile can be + taken into another project as is. No field refers to this repository's + folders, file names or database, nor to one platform's conventions. The + toolchain resolves a profile against the collection. +- **Other projects are peers.** A finding about another project says what + changes, never who erred, and comes with its evidence. Our data can be + wrong too, and an issue that shows it is as welcome as a finding sent + upstream. +- **An inconsistency found is fixed now.** A fix comes with a test that fails + without it, so the same mistake cannot return unnoticed. +- **Collect everything a working setup needs.** BIOS, firmware, keys, fonts, + boot and DSP ROMs, databases, data files. Age and licence are not criteria. + A file the collection lacks is an acquisition target, not an end state. + ## Directory structure ``` @@ -109,6 +141,23 @@ even if the platform does not declare them. Files the platform declares but no profile references are kept as well (flagged during cross-reference), since the upstream may cover cases not yet profiled. +A platform YAML is regenerated by its scraper and never edited by hand: an +edit would be lost at the next scrape, and the file would stop describing the +platform. What the platform gets wrong is answered in the other two layers. +Another name for the same file is an `aliases:` entry in the profile, never a +second entry in a platform YAML or in `_shared.yml`. A name a platform's own +launcher reads, such as the 22 names Batocera's DuckStation generator tries +per region, describes the platform and lives in its `launchers:` records (see +[launchers](adding-a-platform.md#launchers)). + +Some entries in a platform YAML are not the platform's own: files a +`_shared.yml` group injects (`shared_group`), copies the RomM scraper mirrors +to a twin system (`mirror_of`) and additions the libretro scraper makes +(`curated`). `native_layer()` removes them wherever the question is what the +platform itself declares: the export, the truth diff and the site's +comparison tables. Counted as the platform's, they were written into +System.dat as if libretro had declared them. + ```mermaid graph TD PY[Platform YAML
scraped from upstream] --> PG[Pack generation] @@ -124,6 +173,25 @@ graph TD style ZIP fill:#2d333b,stroke:#adbac7,color:#adbac7 ``` +## The inverted model + +Packs are built today from what each platform declares, completed by what the +cores need. The direction the project works toward is the inverse: generate +each platform's file from what the emulators' code needs, file by file, and +offer it to that platform's maintainers as a source they can check and adopt. +The [updated platform files](../exports.md) page publishes these files, every +change traceable to code, presented as suggestions. Keeping a BIOS list in +step with every emulator release is a large amount of work, and the files are +written to make it lighter. + +An entry a platform declares is never removed from its file on the strength of +a missing match. When this was measured, an entry the generated data did not +pair was still read in 1189 of 2776 cases, by an alias, a hash, a directory +entry or a core outside the resolved set. A removal needs positive proof: +every core of the system profiled, each with a reference showing the closed +set of names it opens. Until then the entry is listed as a removal candidate +with the cores examined (see [export_native.py](tools.md#export_nativepy)). + ## Dump-catalog provenance Snapshots of the dump-preservation catalogs (Redump, No-Intro, TOSEC) live in @@ -182,6 +250,64 @@ the bytes, for both the platform and the emulator pack paths. Its fixture holds a romset, because without one the comparison never reaches the rebuild path and passes while the real packs still move. +## Collection rules + +How files enter `bios/`, and why. + +- **Variants keep every layer served.** A different dump under an existing + name goes to `.variants/`. The primary carries the name the emulator's + source uses, and a dump a platform pins under an older name stays a variant, + so each layer receives the bytes it checks. A destination that already + exists is something a platform depends on: a `segaboot.gcm` that Batocera + requires was once overwritten by the dump RetroDECK requires, and is now + kept beside it. +- **One content, one file.** A byte-identical copy under another name is + removed, and the name it carried becomes a profile alias. Trees addressed by + path keep their equal files, since equality is part of their layout: those + folders are `NODEDUP_DIRS` in `scripts/dedup.py`. +- **Archives the project builds are TorrentZip.** `build_torrentzip()` makes + an archive's bytes depend on its contents alone, so anyone can rebuild it. + An upstream romset whose MD5 a platform pins stays verbatim in `.variants/`. + A member renamed by hand in an existing archive gives bytes that match no + distributed romset and that nobody can regenerate. +- **A pack carries what the user must obtain.** A file the emulator installs + itself (`bundled: true`) is not kept once nothing else needs it, and step 4d + of the pipeline fails on such a copy. Its sha1 and size are written into the + profile first, so the entry keeps the file's identity. +- **A platform's older revision needs the core's own claim first.** When a + platform pins an older revision of a data file, collecting it makes the + platform's hash the only content proof at that destination, and a core + profile without a hash then receives the old file. The profile first + declares the hash of the file at its own `source_commit` + (`find_pinned_revision.py` finds the revision behind a pinned md5). +- **A transformation of a held dump is collected by reproducing it.** A byte + swap, a permutation, a concatenation or a published patch is replayed from + its source and checked against the expected hash, and the profile states + what the file derives from. +- **A dump says what it is.** The version string inside a ROM names its model, + whatever folder it arrived in. Dumps once filed for one handheld announced + another model in their own strings. +- **Game data lives under the tree its engine reads.** Engines open generic + names in their own folders: `pak0.pak` is Quake II's under `baseq2/` and + Half-Life's under `valve/`. Each game's files are stored under that tail so + they resolve by path, and `tests/test_game_data_homonyms.py` fails when one + game's entry reaches another game's file by name. +- **A file a platform pins by hash is collected into `bios/`.** A data + directory cache that happens to hold it is not enough, because the installer + cannot fetch from the cache. +- **Collected means committed.** A file indexed by the database but never + committed gives the installer a 404, so + `git ls-files --others --exclude-standard bios` is empty before manifests + are regenerated. +- **MAME and FBNeo archives are named the way the emulator opens them.** A + MAME device's archive takes the short name of its `DEFINE_DEVICE_TYPE`, not + the `ROM_START` label. An FBNeo archive is named after its `RomDesc`, once + the driver's `GetZipName` function strips the system prefix (`msx_msx` + opens `msx.zip`). BIOS root sets change between MAME versions and are listed + from each version's source. A distributed non-merged archive also holds the + ROMs of every device the machine pulls in, which is what a `-listxml` DAT + describes and why [romset recipes](tools.md#romset-recipes) come from DATs. + ## Storage tiers | Tier | Meaning | @@ -223,9 +349,8 @@ platform's own verification mode. files, encryption, excessive expansion and suspicious compression ratios. - standalone-emulator copies require explicit `--standalone-copies` consent. - CI writes only what its job needs: `validate.yml` holds `pull-requests: write` - for the validation comment and labels, `deploy-site.yml` holds the Pages - deploy identity, and the release job holds `contents: write` but runs only - when someone dispatches it. + for the validation comment and labels, and `deploy-site.yml` holds the Pages + deploy identity. No workflow builds or publishes a release. - `safe_extract_zip()` prevents zip-slip path traversal attacks - `deterministic_zip` rebuilds MAME ZIPs so same ROMs always produce the same hash diff --git a/wiki/installer.md b/wiki/installer.md index d3263ce6..d646ce5c 100644 --- a/wiki/installer.md +++ b/wiki/installer.md @@ -465,6 +465,22 @@ Standalone copies are stricter, since they write outside the tree the user selected: a symlink already sitting at the destination is skipped rather than followed. +## Changing the installer + +`install.py` runs on the user's machine, often an older handheld or a stock +RetroPie image, so it stays on Python 3.8 and the standard library alone. +Tests refuse any 3.9 syntax or API, and one runs the installer end to end under +a real 3.8 interpreter, which CI installs beside 3.12 so the check cannot skip. +The build scripts need 3.12 and are not bound by this. + +Every change to `install.py` recomputes the SHA-256 pinned in `install.sh` and +`install.ps1` in the same commit. The tests compare the pins with the file. + +Narrowing by system, core, region or target replays the pack builder's rules +from fields the manifest carries, and a parity test compares an installer +selection with the pack built the same way. Given a manifest without those +fields, the installer refuses to narrow rather than install everything. + ## Exit status | Code | Meaning | diff --git a/wiki/profiling.md b/wiki/profiling.md index ef7320a7..d0d17163 100644 --- a/wiki/profiling.md +++ b/wiki/profiling.md @@ -67,6 +67,24 @@ fresh pass for each new profile. When fbneo_neogeo was profiled, the NeoGeo subset referenced BIOS files that the main arcade analysis hadn't encountered. A fresh look avoids carrying over blind spots. +A declared upstream that answers 404 is not proof the project is gone. The +account can differ from the Android package id (`aenu.aps3e` lives at +`aenu1/aps3e`), a repository can be renamed, and a live repository can belong +to the parent project rather than the one that publishes the package. A code +search on the package id usually names the real repository. + +A closed-source emulator is profiled from what it ships: decompiled IL for a +.NET or Unity build, the string table and import directory of a native binary, +the debug information a library was not stripped of. A source drop published +after the fact can be older than every shipped build, so its option and file +name strings are compared with a genuine binary before it is trusted. + +An Android package is fingerprinted before a byte of it is read. Mirrors carry +rebuilt and re-signed copies of paid emulators, and a profile read from one +describes someone else's modification. Read the signing certificate, compare +it with a build known to be genuine, and prefer an older genuine build to a +newer repack. The profile names the rejected copies. + ### 2. Trace file loading Read the code flow, tracing from the entry point. @@ -93,6 +111,23 @@ object. BlastEm reads a list of ROM paths from a configuration structure. same_cdi opens CD-i BIOS files through a machine initialization routine. The loading flow varies widely between emulators. +Check which build reaches the code. When one tree builds a desktop and an +Android app, the Android project can remove sources from its compile and wire +other entry points, so a file the shared code opens may never be read on +Android. A platform header can also leave a symbol undefined whatever the +comment above it says. Read the project file and the macro block before +calling a load path live, and scope the file in its `note:` when only one +build reads it. + +Everything the code opens from outside goes in the profile, not only BIOS +images: fonts, keyboard maps, NV memory, samples, DSP firmware, game databases, +data directories, native libraries loaded by name. For a library shipped +inside a package, the test is whether the emulator's own code names it. +ArcadeFlashWeb ships a whole Qt deployment, and its profile lists the few files +its source names, such as the Flash runtime passed on the command line. +Operating system runtimes (`VCRUNTIME140.dll`) are prerequisites written in +`notes:`, never files. + ### 3. Determine required vs optional This is decided by code behavior, not by judgment: @@ -113,6 +148,11 @@ are marked with `hle_fallback: true` and `required: false`. The file still ships in packs (better experience for the user), but its absence does not raise alarms during verification. +MAME is read the same way. A missing ROM is fatal only when it is neither +`ROM_OPTIONAL` nor `NO_DUMP` (`src/emu/romload.cpp`), so a set made only of +`NO_DUMP` entries is not required, and MAME hashes the members of an archive, +never the archive: those hashes go in `contents:`. + ### 3b. Decide whether the file carries a region Set `region:` only when the **code branches on a territory to select the @@ -169,6 +209,18 @@ the regions No-Intro gives the same dump. Each disagreement is a line to read in the source, not a correction to apply: the field records the code's selection, and a Hong Kong PS2 is an NTSC-J console. +### 3c. Record the search order + +When several files can fill one slot, `priority:` records the order the code +looks for them, lowest first, read from the source alone. Two cores can rank +the same file differently: pcsx1 walks `scph1001`, `scph5501`, `scph7001` in +that order while DuckStation prefers `scph5501` (5) to `scph1001` (10). The +two uses of the field then answer differently. Choosing one file per slot +(`--one-per-slot`) keeps every candidate, since dropping one would drop +another core's first choice. Ordering a list keeps the best rank any core +gives and notes the disagreement beside it, because a list drops nothing and +the file the emulator prefers is the one a reader copies first. + ### 4. Document divergences When the libretro port differs from the upstream: @@ -225,6 +277,41 @@ read at. Profiles written before this field existed carry `profiled_date` and `core_version` instead. They gain `source_commit` as they are re-verified. +### Citations and pins + +Every choice in a profile carries a `source_ref` to the original emulator's +code, and to the wrapper's too when the wrapper changes the path. Line numbers +only hold at one revision, so a few rules keep them checkable. + +- **The pin and the references move together.** A profile rewritten from a + reading of the current head records that head's sha in `source_commit` in + the same edit. References updated without the pin describe one revision + while the pin names another, and the next comparison reads the old code at + line numbers that only make sense in the new. +- **Paths start at the repository root.** `src/xenia/ui/imgui_drawer.cc:39` + anchors, `ui/imgui_drawer.cc:39` reports GONE although the file exists, and + the failure looks like deleted code. A bare file name anchors only while one + file in the tree carries it. +- **Citations in prose count.** Every `file:line` in notes and free text is + checked like a `source_ref`, and a pin does not advance while one of them + still describes the old revision. +- **Line numbers are read, not copied.** Search tools disagree on whether lines + count from 0 or 1. Read the cited span at the pinned revision before writing + it. +- **A frozen line is pinned to its tag.** A profile documenting a release that + lives as a tag in a still active repository (`pcsx2-legacy` at PCSX2 v1.6.0) + is checked against its own revision and never rebased onto the head. +- **A fork's base comes from the forge.** `upstream_commit` of a fork is the + merge base the compare API returns. A vendored `version.h` can name the + project the fork once came from. When a package ships prebuilt binaries + built from different commits of its own repository, the pin is the revision + whose sources describe the shipped bytes, found by hashing each binary + against the history. +- **Upstream moved, the profile is read again.** When the code a profile cites + is rewritten or moved, the profile is re-profiled at the new revision. A + profile held on a revision nobody ships documents a program that no longer + exists. + ### Writing style Notes in a profile describe what the core does, kept focused on: @@ -264,6 +351,50 @@ A few field conventions that protect the toolchain: the bytes can be extracted and hashed. If the source performs no hash check at all, the hash is omitted from the profile. The .info or docs may list an MD5, but source confirmation makes it more reliable. + A data file the build generates is rebuilt from the tree and compared + byte for byte with the official release before its hash is recorded + (PrBoom+ builds `prboom-plus.wad` with its own tool). A firmware URL that + embeds an MD5, as Sony's update URLs do, names the revision the code + expects, and records `md5:` without `validation: [md5]`. A hash that comes + from neither the code nor the shipped file is not written, even labelled + informative. + +- A name a platform's launcher reads (a Batocera generator, a RetroBat + launcher) is not an alias: it describes the platform, and the platform's + scraper records it under `launchers:`. `aliases:` holds the names the + emulator itself opens. + +- For a file the code finds by content (`found_by: content`), `name:` is a + label. It reuses the name other profiles give that exact dump, so it never + collides with another dump's established name: ares labels borrowed from + other cores once put its revision on a path where RetroBat pins a different + dump. + +- `bundled: true` keeps a file out of packs because the user already has it. + `not_bundled_on`, `installed_on` and `unread_on` refine that per platform + from that platform's own package or launcher, each with the reference that + shows it. + +- `unsourceable:` stops a different file with the same name from being served + in the entry's place. It never means the file cannot be found, and the entry + stays an acquisition target. XM7's kanji ROMs carry it because the + collection's `KANJI.ROM` files are other machines' fonts. + +- `path:` is the full destination, file name included. A directory-only value + collapses several entries into one count. On a profile that is + `type: standalone`, `mode: standalone` sits at the top only: repeated on each + file it hides them from the cross-reference. + +- An entry of a multi-system profile carries `system:`, read from the code. + The toolchain never guesses it, and an entry without one is counted as + unattributed. + +- Every name in `cores:`, and every numeric file name, is a quoted string: + `name: 01` reads back as `1`. `cores:` lists every name the buildbot and the + platforms use for the core, since a name no profile claims is a core that + target filtering skips. + +- Metadata goes in fields, never in parentheses inside a string. ### 6. Validate @@ -287,6 +418,19 @@ value to compare (`validation: [crc32]` and no `crc32`), a `data_directories` ref the registry does not name, a `path:` written in this repository's layout instead of the emulator's, a citation no declared revision can reach. +Three more checks catch what the schema cannot see: + +- `python scripts/profile_sync.py --emulator NAME` reports 0 to re-read. A + batch of GONE references on a new profile means the path prefix, not + upstream drift. +- In `python scripts/verify.py --emulator NAME --json`, the status counts add + up to the number of entries. An entry the collection was never meant to + hold that reports OK is a homonym being served, and needs `unsourceable:`. +- A profile that becomes `standalone + libretro` joins every RetroArch pack. + Generate the RetroArch manifest with and without it into a scratch + `--output-dir` and compare the destinations: a standalone layout written + into `path:` shows up as files at destinations no core reads. + ### Lessons learned These are patterns that have come up while building profiles. Sharing @@ -417,6 +561,6 @@ partitions), and `game_count`. | `contents` | structure of files inside a BIOS ZIP (`name`, `description`, `size`, `crc32`) | | `storage` | `embedded` (default), `external`, `user_provided`, or `large_file`/`release` for files stored as release assets (listed in `.gitignore`) | | `agnostic` | true if any file under the system path within size constraints satisfies the requirement | -| `unsourceable` | reason why the file cannot be sourced (acknowledged gap) | +| `unsourceable` | why the collection cannot hold these exact bytes. It stops a different file with the same name from being served in the entry's place, while every hash proof still runs. The entry stays an acquisition target | | `destination` | target path within the BIOS directory | diff --git a/wiki/testing-guide.md b/wiki/testing-guide.md index 270190f0..bf4c3c1d 100644 --- a/wiki/testing-guide.md +++ b/wiki/testing-guide.md @@ -182,7 +182,13 @@ disambiguated by a hash suffix on purpose. output or formatting. 5. **Run the full suite.** After adding your test, run `python -m unittest - discover tests -v` to verify nothing else broke. + discover -s tests -t . -v` to verify nothing else broke. `-t .` imports + `tests/__init__.py`, which keeps the suite offline. + +A fix comes with a test that fails without it. Write the test first, watch it +fail on the current code, then fix: a test written after the fix may never +have exercised the case it names. When a reading of the code finds a fact a +script could have established, the script gains that check in the same change. Example skeleton: @@ -240,6 +246,55 @@ If a change passes tests but breaks the pipeline, it's worth investigating befor Ideally, tests, code, and documentation ship together. When profiles and platform configs are involved, updating them in the same change helps keep everything in sync. +A run that dies with a segmentation fault or an exception the code cannot +raise is run again once, alone, before anything is investigated. Only a +failure that reproduces is a defect in the code. + +## Prose and tone + +`tests/test_prose_style.py` holds every tracked file, and the generated site, +to the way the project writes: + +- No em-dash or en-dash, no semicolon joining two clauses, no unicode arrows, + no ellipsis character, no box-drawing banners, no emoji or check marks. + Rewrite the sentence rather than swapping the character. +- None of the marketing vocabulary the test lists, and no superlatives. A + number with its criterion says more. +- Headings in sentence case, never repeating the site name. + +Formatting stays: bold labels, lists, tables and admonitions make a page +easier to use. When markup shows raw on the site, the rendering is fixed and +the formatting kept. + +What the test cannot read is tone. Text that names another project says what +changes, not who erred: "hash updated to the emulator source", "files added", +"required flags aligned with the code". No "wrong", "bogus" or "phantom" about +someone else's work. Notes in a profile say what the core does, nothing about +what it is not. A reply on an issue or a pull request thanks once, gives the +facts, describes the script or the file rather than the person, and leaves +the door open. + +## Landing a change + +A commit subject is English, 50 characters at most, `type: description` with a +lowercase word after the colon. The types are `feat`, `refactor`, `chore`, +`docs` and `fix`. The message does not use bug, regression, revert, dead code, +broken, wrong, hack, workaround or implement (derived forms included), nor the +marketing vocabulary above, and carries no Co-Authored-By trailer. A body +explains the why when the subject cannot: the scenario that failed, the +upstream `file:line` that decides it, what users will see. + +``` +feat: add retrobat platform support +refactor: extract shared utils to common.py +``` + +Files are staged by name, never with `git add -A` or `git add .`, which sweep in +deletions and work that is not yours. Generated artifacts (`database.json`, +`install/*.json`, `README.md`, `CONTRIBUTING.md`, `exports/`) are committed +with the data they derive from, never alone: a manifest committed before the +files it points at gives the installer a 404. + ## CI integration The `validate.yml` workflow runs `python -m unittest discover -s tests -t . -v` on both diff --git a/wiki/tools.md b/wiki/tools.md index d5cc06b8..8164c5d4 100644 --- a/wiki/tools.md +++ b/wiki/tools.md @@ -67,6 +67,30 @@ Pipeline flags: `--check-buildbot` runs the whole pipeline online. For the freshness check alone, run `python scripts/check_buildbot_system.py`. +## Conventions every tool keeps + +- **A flag is applied or refused.** A mode that accepts a flag it does not + apply leaves the user believing they filtered. Four such cases were found + (`verify --emulator --region` among them), and a test now runs every mode + against every flag. +- **Every dimension that narrows an artifact appears in its name.** Region, + target, required-only, standalone and source variant are part of the pack + and manifest names. `--manifest --target switch` once overwrote the + manifest every installer downloads with a narrowed one. +- **A step that did not run says SKIPPED**, never OK. +- **verify.py and generate_pack.py read the same functions.** A count that + differs between the report and the pack is a defect, which is what step 5 + checks. +- **Generators write through `write_if_changed()`.** A new timestamp alone + rewrites nothing, so a diff on a generated file always means content. +- **One run per output directory.** Pack generation takes a lock on + `/.lock`, and a second run on the same folder stops at once + rather than read a half-written ZIP. Experiments use a scratch + `--output-dir` under `tmp/`. +- **Read the exit status before the output.** A generator that crashed leaves + the previous artifact in place, which reads exactly like a run that changed + nothing. + ## Individual tools ### generate_db.py @@ -238,7 +262,7 @@ python scripts/diff_truth.py --all # diff all platforms ### export_native.py -Rewrite each platform's own BIOS file, corrected: System.dat, es_bios.xml, +Write each platform's own BIOS file, updated from the emulators' code: System.dat, es_bios.xml, batocera-systems, checkBIOS.sh, FirmwareDatabase.cs, the RetroDECK manifests, MiSTer's BiosDB, RetroPie's scriptmodules. Every platform in the registry, archived included. diff --git a/wiki/verification-modes.md b/wiki/verification-modes.md index 5437f031..0900d44a 100644 --- a/wiki/verification-modes.md +++ b/wiki/verification-modes.md @@ -104,6 +104,10 @@ wrong hash beside a right one does not withhold the file. `verify.py`, the pack builder and the install manifest apply that rule. ZIP files are not opened, only the container is checked. +`verification_mode` answers whether the frontend reads the bytes, and +`hash_rule` answers which comparison it makes. A frontend whose acceptance is +not a single digest gets a new `hash_rule` value, never a new mode. + ### ROCKNIX verification **Source**: `rocknix-systems`, function `checkBios` @@ -266,6 +270,14 @@ When platform verification passes but emulator validation fails, the file has a This happens most often in existence-mode platforms where any file with the right name is accepted. +A local file that contradicts a declared hash is handled by the platform's +own mode, never by one global rule. An existence frontend never reads the +bytes, so the file is packed and the disagreement reported: a stale hash in +System.dat must not remove a file RetroArch would have loaded. An md5 or sha1 +frontend would reject it, so it is left out of the pack and counted as +`unsafe excluded`, which is distinct from `missing`. The pack builder and the +pack verification apply the same rule. + ### Variant search `validated_choice()` (`scripts/validation.py`) decides the file a platform