feat: keep a profile checkable when its forge goes

Four Switch profiles were unverifiable and said so on stderr every pass.
yuzu and suyu answer 451, citron's host stopped resolving, and
git.eden-emu.dev returns 403 to anything that is not a browser. Each
aborted its own report, so nothing could be said about any of them, and
the noise repeated on every run over the whole corpus.

A withdrawn forge is now a fact rather than a failure. GoneError covers
451, 410 and a host that does not resolve; none is retried, since three
attempts with backoff end in the same place. Those profiles land in
their own summary bucket, out of the review backlog where nobody could
act on them anyway. A 403 stays what it was, a refusal, because small
Forgejo instances behind anti-bot filters issue it routinely.

A profile can now name a source_mirror, consulted after source and
upstream so a live primary always decides attribution. Reaching it took
two more changes: a repository that refuses is muted for the rest of the
pass instead of ending it, keyed by host as well as slug because a
mirror carries the same slug on another forge; and a refused miss is not
cached, or the mute would answer for the mirror that was about to be
asked.

eden now reads from its Codeberg copy, which holds the same head and the
pinned commit: 5 refs, all anchored, where the profile could not be
checked at all. yuzu, suyu and citron have no mirror that serves
content, and now say so once instead of failing loudly.
This commit is contained in:
Abdessamad Derraz committed 2026-09-04 15:35:00 +02:00
1 parent e6d325ed1b
commit ca307a4ef1
6 files changed
+230 -14

No files matched your search

+34
View File
@@ -11,6 +11,7 @@ import hashlib
import http.client
import json
import os
import socket
import tempfile
import time
import urllib.error
@@ -69,6 +70,17 @@ class RateLimitError(UpstreamError):
"""The forge refused the request for quota reasons."""
class GoneError(UpstreamError):
"""The upstream is not coming back.
A legal takedown, a resource the forge reports as gone, or a host that
no longer resolves. Retrying costs time and ends in the same place, and
a caller sweeping every profile wants this told apart from a forge
having a bad minute: one is a fact about the project, the other is
weather.
"""
@dataclass(frozen=True)
class Repo:
host: str
@@ -161,6 +173,9 @@ def _http_failure(url: str, exc: urllib.error.HTTPError) -> UpstreamError:
"""
if exc.code == 429:
return RateLimitError(f"{url}: HTTP 429")
if exc.code in (410, 451):
reason = "withdrawn for legal reasons" if exc.code == 451 else "gone"
return GoneError(f"{url}: HTTP {exc.code}, {reason}")
if exc.code == 403 and exc.headers is not None:
remaining = exc.headers.get("X-RateLimit-Remaining")
if remaining is not None and remaining.strip() == "0":
@@ -168,6 +183,23 @@ def _http_failure(url: str, exc: urllib.error.HTTPError) -> UpstreamError:
return UpstreamError(f"{url}: HTTP {exc.code}")
def _host_is_unresolvable(exc: BaseException) -> bool:
"""Whether a connection failure is the name itself, not the network.
URLError carries the cause in `reason`, and a wrapped one carries it in
`__cause__`; the walk is bounded because either chain can be cyclic.
"""
seen: BaseException | None = exc
for _ in range(8):
if seen is None:
break
if isinstance(seen, socket.gaierror):
return True
nested = getattr(seen, "reason", None)
seen = nested if isinstance(nested, BaseException) else seen.__cause__
return False
def _fetch(url: str, accept_json: bool = False) -> bytes | None:
"""Body of a GET, or None on 404.
@@ -188,6 +220,8 @@ def _fetch(url: str, accept_json: bool = False) -> bytes | None:
if isinstance(failure, RateLimitError) or exc.code < 500:
raise failure from exc
except (urllib.error.URLError, http.client.HTTPException, OSError) as exc:
if _host_is_unresolvable(exc):
raise GoneError(f"{url}: host does not resolve") from exc
failure = UpstreamError(f"{url}: {exc}")
if attempt + 1 < RETRIES:
_sleep(RETRY_BACKOFF[attempt])