chore: merge android retroarch folder per build

# Conflicts:
#	install.ps1
#	install.sh

install.ps1

@@ -11,7 +11,7 @@
 
     $ErrorActionPreference = "Stop"
     $defaultInstallUrl = "https://raw.githubusercontent.com/Abdess/retrobios/main/install.py"
-    $defaultInstallSha256 = "7273e615d62c11711c3aa2841d3aeeef3bdd0ad14304c03edcfb7973e2ad2426"
+    $defaultInstallSha256 = "03b97db812dcdc161aecedd55f0c8996fababa27f47e3c0c90d52a4f0365565b"
     $maximumInstallerBytes = 2MB
     $installerExit = 0
     $installer = if ($PSScriptRoot) { Join-Path $PSScriptRoot "install.py" } else { $null }

install.py

@@ -440,6 +440,8 @@ ANDROID_RETROARCH_PACKAGES = (
     "com.retroarch",
     "com.retroarch.ra32",
 )
+# Where Android keeps each app's private data.
+ANDROID_APP_DATA = Path("/data/data")
 
 
 def _is_android() -> bool:
@@ -1171,8 +1173,78 @@ def _android_storage_root() -> Path:
     return Path(value) if value else Path("/storage/emulated/0")
 
 
-def _detect_android() -> list[tuple[str, Path]]:
-    """Find the RetroArch system directory on Android.
+# Where each RetroArch package keeps its folders, RetroArch v1.22.2
+# (pkg/android/phoenix/build.gradle:46-95). com.retroarch and
+# com.retroarch.aarch64 are each published twice under one id: by
+# retroarch.com (normal, aarch64) and on Google Play (playStoreNormal,
+# playStorePlus). com.retroarch.ra32 has no Google Play flavor. The Google
+# Play build hands RetroArch Android/data/<package>/files as the storage its
+# defaults are built from, the other builds the shared storage root
+# (MainMenuActivity.java:182-183).
+_ANDROID_RETROARCH_PLAY_IDS = ("com.retroarch", "com.retroarch.aarch64")
+# A Google Play versionCode is the manifest's 1597175267 (AndroidManifest.xml:5)
+# plus one per half week since 1609632000 (build.gradle:76, 87, 168-175), so
+# it stays below that start time. The other flavors use the build time in
+# seconds (build.gradle:42), above it for every build since 2021. The launcher
+# writes the code to retroarch.cfg (UserPreferences.java:150), which
+# RetroArch keeps (configuration.c:2623).
+_ANDROID_PLAY_VERSION_CEILING = 1609632000
+
+
+def _android_cfg_value(cfg: Path, key: str) -> str | None:
+    """A value of an Android retroarch.cfg, None when unreadable or unset."""
+    try:
+        lines = cfg.read_text(encoding="utf-8", errors="replace").splitlines()
+    except OSError:
+        return None
+    for line in lines:
+        name, sep, value = line.partition("=")
+        if sep and name.strip() == key:
+            return value.strip().strip('"').strip("'")
+    return None
+
+
+def _android_play_build(package: str, storage: Path, cfgs: tuple[Path, ...]) -> bool | None:
+    """True for the Google Play build of this package, False for the
+    retroarch.com one, None when nothing it wrote tells them apart.
+
+    The version code in its config decides. Without a readable config, the
+    folder each build writes does: the Google Play build never asks for
+    shared storage (MainMenuActivity.java:199-202), and only it builds its
+    defaults under Android/data/<package>/files/RetroArch.
+    """
+    if package not in _ANDROID_RETROARCH_PLAY_IDS:
+        return False
+    for cfg in cfgs:
+        code = _android_cfg_value(cfg, "bundle_assets_extract_version_current")
+        if code and code.isdigit() and int(code) > 0:
+            return int(code) < _ANDROID_PLAY_VERSION_CEILING
+    if (storage / "Android" / "data" / package / "files" / "RetroArch").is_dir():
+        return True
+    if (storage / "RetroArch").is_dir():
+        return False
+    return None
+
+
+def _android_writable(folder: Path) -> bool:
+    """Whether this shell can create folder: its nearest existing parent is
+    writable. Android 11 and later keep Android/data/<package> of other apps
+    from Termux."""
+    for parent in (folder, *folder.parents):
+        if parent.exists():
+            return parent.is_dir() and os.access(parent, os.W_OK)
+    return False
+
+
+def _android_out_of_reach(name: str, folder: Path, what: str) -> str:
+    """The line said for an emulator whose BIOS folder this shell cannot write."""
+    return (f"{name} is installed, but this shell cannot write its BIOS folder "
+            f"{folder} without root: {what} are not copied there.")
+
+
+def _android_retroarch_builds() -> list[tuple[Path | None, str]]:
+    """Each installed RetroArch build: its system folder when this shell can
+    write it, else None and the line that says why.
 
     RetroArch v1.22.2: the launcher picks retroarch.cfg in the app external
     files directory, else in the internal one
@@ -1180,24 +1252,58 @@ def _detect_android() -> list[tuple[str, Path]]:
     UserPreferences.java:36-89), and passes it as CONFIGFILE
     (MainMenuActivity.java:178, read at frontend/drivers/platform_unix.c:
     1428-1444). A system_directory left at default resolves against
-    <shared storage>/RetroArch/system (platform_unix.c:1670-1697), which is
-    also where the directory sits when no config has been written yet.
+    <storage>/RetroArch/system (platform_unix.c:1670-1697), storage being
+    what the launcher passes as SDCARD: Android/data/<package>/files for the
+    Google Play build, the shared storage root for the others.
+    """
+    storage = _android_storage_root()
+    builds: list[tuple[Path | None, str]] = []
+    for package in ANDROID_RETROARCH_PACKAGES:
+        external = storage / "Android" / "data" / package / "files"
+        if not (ANDROID_APP_DATA / package).exists() and not external.exists():
+            continue
+        cfgs = (external / "retroarch.cfg",
+                ANDROID_APP_DATA / package / "files" / "retroarch.cfg")
+        play = _android_play_build(package, storage, cfgs)
+        play_root, shared_root = external / "RetroArch", storage / "RetroArch"
+        named = next(filter(None, (_android_cfg_value(cfg, "system_directory")
+                                   for cfg in cfgs)), "default")
+        if play is None and named == "default":
+            builds.append((None,
+                f"RetroArch ({package}) is installed, but nothing it has written tells "
+                f"which build it is: the Google Play build reads {play_root / 'system'}, "
+                f"the retroarch.com build reads {shared_root / 'system'}. "
+                "Name the folder with --dest."))
+            continue
+        root = play_root if play else shared_root
+        folder = next(filter(None, (_parse_retroarch_system_dir(cfg, root) for cfg in cfgs)),
+                      root / "system")
+        private = (storage / "Android" / "data", ANDROID_APP_DATA)
+        if any(folder == base or base in folder.parents for base in private) \
+                and not _android_writable(folder):
+            build = "Google Play build" if play else "retroarch.com build"
+            builds.append((None, _android_out_of_reach(
+                f"RetroArch ({build}, {package})", folder, "BIOS files")))
+            continue
+        builds.append((folder, ""))
+    return builds
+
+
+def _detect_android() -> list[tuple[str, Path]]:
+    """Find the RetroArch system directory of each installed build on
+    Android, as _android_retroarch_builds() resolves it (RetroArch v1.22.2,
+    platform_unix.c:1670-1697).
+
+    With no package seen, <shared storage>/RetroArch is the folder of a
+    retroarch.com build whose private data this shell cannot see.
     """
     found: list[tuple[str, Path]] = []
-    storage = _android_storage_root()
-    default_root = storage / "RetroArch"
-
-    for package in ANDROID_RETROARCH_PACKAGES:
-        for cfg in (
-            storage / "Android" / "data" / package / "files" / "retroarch.cfg",
-            Path("/data/data") / package / "files" / "retroarch.cfg",
-        ):
-            system_dir = _parse_retroarch_system_dir(cfg, default_root)
-            if system_dir:
-                found.append(("retroarch", system_dir))
-                return found
-
-    if default_root.exists():
+    builds = _android_retroarch_builds()
+    for folder, _line in builds:
+        if folder is not None and ("retroarch", folder) not in found:
+            found.append(("retroarch", folder))
+    default_root = _android_storage_root() / "RetroArch"
+    if not builds and default_root.exists():
         found.append(("retroarch", default_root / "system"))
     return found
 
@@ -1210,7 +1316,6 @@ def _detect_android() -> list[tuple[str, Path]]:
 # shell/android-studio/flycast/src/main/jni/src/Android.cpp:143-159), its
 # home being Android/data/com.flycast.emulator/files, which Android 11 and
 # later keep from other apps.
-ANDROID_APP_DATA = Path("/data/data")
 _ANDROID_APP_FOLDERS = (
     ("DraStic", "com.dsemu.drastic", "/data/data/com.dsemu.drastic/files/DraStic/system",
      ("nds_bios_arm7.bin", "nds_bios_arm9.bin", "nds_firmware.bin")),
@@ -1222,12 +1327,14 @@ _ANDROID_APP_FOLDERS = (
 
 def android_unreachable_folders() -> list[str]:
     """A line for each Android emulator installed here whose BIOS folder
-    this shell cannot write, said instead of skipping it in silence.
+    this shell cannot write, said instead of skipping it in silence. A
+    RetroArch build whose folder is out of reach, or whose build cannot be
+    told, is among them.
 
     An app's private folder can be tested for existence by any app, so the
     emulator is known to be installed even where its folder is out of reach.
     """
-    lines = []
+    lines = [line for folder, line in _android_retroarch_builds() if folder is None]
     storage = str(_android_storage_root())
     for name, package, folder, files in _ANDROID_APP_FOLDERS:
         target = Path(folder.replace("$STORAGE", storage, 1))
@@ -1237,8 +1344,7 @@ def android_unreachable_folders() -> list[str]:
             continue
         if target.is_dir() and os.access(target, os.W_OK):
             continue
-        lines.append(f"{name} is installed, but this shell cannot write its BIOS folder "
-                     f"{target} without root: {', '.join(files)} are not copied there.")
+        lines.append(_android_out_of_reach(name, target, ", ".join(files)))
     return lines
 
 

install.sh

@@ -18,7 +18,7 @@ esac
 TEMP_INSTALLER=""
 TEMP_DIRECTORY=""
 DEFAULT_INSTALL_URL="https://raw.githubusercontent.com/Abdess/retrobios/main/install.py"
-DEFAULT_INSTALL_SHA256="7273e615d62c11711c3aa2841d3aeeef3bdd0ad14304c03edcfb7973e2ad2426"
+DEFAULT_INSTALL_SHA256="03b97db812dcdc161aecedd55f0c8996fababa27f47e3c0c90d52a4f0365565b"
 MAX_INSTALLER_BYTES=2097152
 
 cleanup() {

tests/test_install.py

@@ -541,7 +541,9 @@ class TestAndroidDetection(unittest.TestCase):
         storage = Path(_tempdir(self))
         files = storage / "Android" / "data" / "com.retroarch.aarch64" / "files"
         files.mkdir(parents=True)
-        (files / "retroarch.cfg").write_text('system_directory = "default"\n')
+        (files / "retroarch.cfg").write_text(
+            'system_directory = "default"\n'
+            'bundle_assets_extract_version_current = "1745000000"\n')
         os.environ["EXTERNAL_STORAGE"] = str(storage)
         self.assertEqual(
             install._detect_android(),
@@ -2724,6 +2726,130 @@ class AndroidCitationsNameTheRelease(unittest.TestCase):
             self.assertNotIn("2836-2851", text)
 
 
+class AndroidRetroArchFolderFollowsItsBuild(unittest.TestCase):
+    """The Google Play build of RetroArch builds its defaults under
+    Android/data/<package>/files/RetroArch, the retroarch.com build under
+    the shared storage root (RetroArch v1.22.2, MainMenuActivity.java:
+    182-183), and both publish com.retroarch and com.retroarch.aarch64.
+    Detection sent every build to <shared storage>/RetroArch/system."""
+
+    PLAY_CODE = 1597175870
+    SITE_CODE = 1745000000
+
+    def _run(self, make):
+        root = Path(_tempdir(self))
+        app_data, storage = root / "data", root / "storage"
+        app_data.mkdir()
+        storage.mkdir()
+        make(app_data, storage)
+        try:
+            with unittest.mock.patch.object(install, "ANDROID_APP_DATA", app_data), \
+                    unittest.mock.patch.dict(os.environ, {"EXTERNAL_STORAGE": str(storage)}):
+                return storage, install._detect_android(), install.android_unreachable_folders()
+        finally:
+            for folder in root.rglob("*"):
+                if folder.is_dir():
+                    folder.chmod(0o755)
+
+    @staticmethod
+    def _install(app_data, storage, package, cfg=None, visible=True):
+        (app_data / package).mkdir()
+        if visible:
+            files = storage / "Android" / "data" / package / "files"
+            files.mkdir(parents=True)
+            if cfg is not None:
+                (files / "retroarch.cfg").write_text(cfg)
+
+    @staticmethod
+    def _cfg(code, system="default"):
+        return (f'system_directory = "{system}"\n'
+                f'bundle_assets_extract_version_current = "{code}"\n')
+
+    def test_google_play_build_reads_its_own_files(self):
+        storage, found, lines = self._run(lambda data, storage: self._install(
+            data, storage, "com.retroarch", self._cfg(self.PLAY_CODE)))
+        folder = storage / "Android/data/com.retroarch/files/RetroArch/system"
+        self.assertEqual(found, [("retroarch", folder)])
+        self.assertEqual(lines, [])
+
+    def test_retroarch_com_build_reads_shared_storage(self):
+        storage, found, lines = self._run(lambda data, storage: self._install(
+            data, storage, "com.retroarch.aarch64", self._cfg(self.SITE_CODE)))
+        self.assertEqual(found, [("retroarch", storage / "RetroArch/system")])
+        self.assertEqual(lines, [])
+
+    def test_ra32_has_no_google_play_build(self):
+        storage, found, _lines = self._run(lambda data, storage: self._install(
+            data, storage, "com.retroarch.ra32", 'system_directory = "default"\n'))
+        self.assertEqual(found, [("retroarch", storage / "RetroArch/system")])
+
+    def test_play_plus_folder_seen_without_a_config(self):
+        def make(data, storage):
+            self._install(data, storage, "com.retroarch.aarch64")
+            (storage / "Android/data/com.retroarch.aarch64/files/RetroArch").mkdir()
+        storage, found, _lines = self._run(make)
+        self.assertEqual(found, [(
+            "retroarch",
+            storage / "Android/data/com.retroarch.aarch64/files/RetroArch/system")])
+
+    def test_play_folder_out_of_reach_is_named(self):
+        if os.geteuid() == 0:
+            self.skipTest("root writes any folder")
+
+        def make(data, storage):
+            self._install(data, storage, "com.retroarch", self._cfg(self.PLAY_CODE))
+            (storage / "Android/data/com.retroarch/files").chmod(0o555)
+        storage, found, lines = self._run(make)
+        self.assertEqual(found, [])
+        self.assertEqual(len(lines), 1)
+        self.assertIn("RetroArch (Google Play build, com.retroarch) is installed", lines[0])
+        self.assertIn(str(storage / "Android/data/com.retroarch/files/RetroArch/system"),
+                      lines[0])
+
+    def test_hidden_private_data_and_shared_folder_mean_retroarch_com(self):
+        def make(data, storage):
+            self._install(data, storage, "com.retroarch", visible=False)
+            (storage / "RetroArch").mkdir()
+        storage, found, lines = self._run(make)
+        self.assertEqual(found, [("retroarch", storage / "RetroArch/system")])
+        self.assertEqual(lines, [])
+
+    def test_undecided_build_names_both_folders(self):
+        storage, found, lines = self._run(lambda data, storage: self._install(
+            data, storage, "com.retroarch.aarch64", visible=False))
+        self.assertEqual(found, [])
+        self.assertEqual(len(lines), 1)
+        self.assertIn(
+            str(storage / "Android/data/com.retroarch.aarch64/files/RetroArch/system"),
+            lines[0])
+        self.assertIn(str(storage / "RetroArch/system"), lines[0])
+
+    def test_named_folder_needs_no_build(self):
+        storage, found, _lines = self._run(lambda data, storage: self._install(
+            data, storage, "com.retroarch", f'system_directory = "{storage}/bios"\n'))
+        self.assertEqual(found, [("retroarch", storage / "bios")])
+
+    def test_each_installed_build_is_found(self):
+        def make(data, storage):
+            self._install(data, storage, "com.retroarch", self._cfg(self.PLAY_CODE))
+            self._install(data, storage, "com.retroarch.ra32", self._cfg(self.SITE_CODE))
+        storage, found, _lines = self._run(make)
+        self.assertEqual(found, [
+            ("retroarch", storage / "Android/data/com.retroarch/files/RetroArch/system"),
+            ("retroarch", storage / "RetroArch/system"),
+        ])
+
+    def test_detection_prints_the_retroarch_line(self):
+        err = io.StringIO()
+        root = Path(_tempdir(self))
+        (root / "data" / "com.retroarch").mkdir(parents=True)
+        with unittest.mock.patch.object(install, "ANDROID_APP_DATA", root / "data"), \
+                unittest.mock.patch.dict(os.environ, {"EXTERNAL_STORAGE": str(root / "s")}), \
+                contextlib.redirect_stderr(err):
+            install.detect_platforms("android")
+        self.assertIn("RetroArch (com.retroarch) is installed", err.getvalue())
+
+
 class AndroidEmulatorsOutOfReachAreSaid(unittest.TestCase):
     """DraStic keeps its BIOS in its private data and Flycast in
     Android/data, which Termux cannot write on Android 11 and later: both

wiki/installer.md

@@ -397,12 +397,15 @@ under scoped storage.
 |----------|----------|----------------|
 | `<shared storage>/Android/data/<package>/files/retroarch.cfg` | RetroArch (Android) | `system_directory` |
 | `/data/data/<package>/files/retroarch.cfg` | RetroArch (Android) | `system_directory` |
-| `<shared storage>/RetroArch/` | RetroArch (Android) | `<shared storage>/RetroArch/system` |
+| `<shared storage>/Android/data/<package>/files/RetroArch/` | RetroArch (Android, Google Play build) | `<shared storage>/Android/data/<package>/files/RetroArch/system` |
+| `<shared storage>/RetroArch/` | RetroArch (Android, retroarch.com build) | `<shared storage>/RetroArch/system` |
 
 The packages are `com.retroarch.aarch64`, `com.retroarch` and
-`com.retroarch.ra32`, the flavours the frontend publishes. Both config paths
-are unreadable from Termux on Android 11 and later, so the directory itself is
-what answers on current devices.
+`com.retroarch.ra32`, the flavours the frontend publishes. The first two are
+published both on Google Play and on retroarch.com, and the two builds keep
+their folders in different places (see [Android](#android)). Both config
+paths are unreadable from Termux on Android 11 and later, so the directories
+themselves are what answers on current devices.
 
 ES-DE and LaunchBox are reported when present and nothing is written to them:
 they reference emulators rather than owning a BIOS directory. ES-DE is looked
@@ -496,13 +499,31 @@ Android reports itself as Linux, so the runtime is named from its own
 variables instead: `ANDROID_ROOT` and `ANDROID_DATA`, which Termux inherits,
 with `/system/build.prop` covering a shell started without them.
 
-Detection then looks for RetroArch, the only Android frontend keeping a shared
-BIOS directory. `retroarch.cfg` is read from the app external files directory
-and then from the internal one, the order the frontend itself follows
-(RetroArch v1.22.2, `UserPreferences.java:36-89`), and a `system_directory`
-left at `default` resolves against `<shared storage>/RetroArch`, where the
-directory also sits when no config has been written yet
-(`platform_unix.c:1670-1697`).
+Detection then looks for each installed RetroArch package. Its
+`retroarch.cfg` is read from the app external files directory and then from
+the internal one, the order the frontend itself follows (RetroArch v1.22.2,
+`UserPreferences.java:36-89`), and a `system_directory` it names is used as
+written. Left at `default`, it resolves against `<storage>/RetroArch`
+(`platform_unix.c:1670-1697`), where `<storage>` depends on the build
+(`MainMenuActivity.java:182-183`):
+
+| Build | Packages | `<storage>` |
+|-------|----------|-------------|
+| Google Play | `com.retroarch`, `com.retroarch.aarch64` | `<shared storage>/Android/data/<package>/files` |
+| retroarch.com | `com.retroarch`, `com.retroarch.aarch64`, `com.retroarch.ra32` | `<shared storage>` |
+
+The two builds share their package ids (`pkg/android/phoenix/build.gradle:46-95`),
+so the build is told by what it wrote. The version code the launcher records
+in `retroarch.cfg` (`bundle_assets_extract_version_current`) is below
+1609632000 for a Google Play build and the build time in seconds for the
+others (`build.gradle:42, 168-175`). Without a readable config, a
+`RetroArch` folder under `Android/data/<package>/files` is the Google Play
+build, and `<shared storage>/RetroArch` is the retroarch.com build, the
+Google Play build never asking for shared storage
+(`MainMenuActivity.java:199-202`). When nothing tells them apart, the run
+names both folders and asks for `--dest`. The Google Play folder sits in
+`Android/data`, which Termux cannot write on Android 11 and later: the run
+then names it, as it does for Flycast below.
 
 Two Android emulators keep their BIOS elsewhere. DraStic reads
 `/data/data/com.dsemu.drastic/files/DraStic/system`, private to the app, and
@@ -514,8 +535,8 @@ run names it and the files that are not copied. Where the Flycast folder is
 writable (Android 10 and earlier), it is a standalone copy target.
 
 Under scoped storage the two RetroArch config paths are unreadable from Termux on
-Android 11 and later, so on current devices the shared directory is what
-answers. Running `termux-setup-storage` once is what makes it reachable:
+Android 11 and later, so on current devices the folders are what answers.
+Running `termux-setup-storage` once is what makes the shared one reachable:
 
 ```bash
 pkg install python