Skip to content

ci(conan): fetch sources through ConanCenter's backup, not the upstream host - #801

Merged
andiwand merged 1 commit into
mainfrom
ci/conan-source-backup
Aug 31, 2026
Merged

ci(conan): fetch sources through ConanCenter's backup, not the upstream host#801
andiwand merged 1 commit into
mainfrom
ci/conan-source-backup

Conversation

@andiwand

@andiwand andiwand commented Aug 31, 2026

Copy link
Copy Markdown
Member

🤖 Generated with Claude Code

CI has been failing intermittently on conan install:

uchardet/0.0.8: WARN: network: Error 418 downloading file
  https://www.freedesktop.org/software/uchardet/releases/uchardet-0.0.8.tar.xz
uchardet/0.0.8: Waiting 5 seconds to retry...
ERROR: uchardet/0.0.8: Error in source() method, line 56

418 is freedesktop's bot filter, not an outage — the same URL serves 200 from a
desk right now. It rejects data-centre IPs, so conan's retry loop can never win.

Why the conan cache doesn't already prevent this

Three things stack up:

  1. uchardet has no prebuilt binary for the wasm profile (emcc/3.1.73,
    cppstd=20). The log shows all 13 compatible configurations checked and none
    found, so it is always --build missingsource() → freedesktop.
  2. A first run on a new branch has no cache to restore. GitHub cache scoping
    means a branch sees only its own entries plus refs/heads/main's — never
    another branch's — so a fresh branch is always cold for wasm.
  3. And main has no conan-wasm-* entry at all. gh cache list shows it on
    three feature branches only. actions/cache does not save when the job
    fails, and main's wasm job fails on exactly this. The failure prevents the
    cache that would prevent the failure.

The fix

Conan 2's source-backup mechanism. ConanCenter mirrors every recipe's sources,
addressed by the sha256 the recipe already declares, so
core.sources:download_urls can put the mirror ahead of the recipe's own url —
for every dependency, not just this one. origin stays behind it as the
fallback for sources the backup does not carry.

Every workflow already runs conan config install .github/config/conan before
resolving (all 8 call sites: wasm, android, apple, python, tidy, conan and both
build_test jobs), so no workflow file changes.

The downloaded tarball also lands in $CONAN_HOME/sources, inside the path the
conan cache already covers, so warm runs never hit the mirror either.

Verification

Built uchardet/0.0.8 from source in a throwaway CONAN_HOME with conan
2.26.1 and this config installed. freedesktop was never contacted:

uchardet/0.0.8: Sources for https://www.freedesktop.org/software/uchardet/releases/uchardet-0.0.8.tar.xz
  found in remote backup https://c3i.jfrog.io/artifactory/conan-center-backup-sources/

Not in this PR

The repo sits at 9.97 GB of the 10 GB Actions cache quota (41 entries), so
LRU eviction churns constantly — three branches each holding a 1058 MB wasm
conan cache is 3.1 GB of duplicated content on its own. The clean fix is to stop
feature branches from writing conan caches: swap actions/cache for
cache/restore + cache/save guarded by if: github.ref == 'refs/heads/main',
the pattern ccache already uses in these same files and the one the cache-key
comment at the top of build_test.yml describes. Seven files, worth doing
separately.

…am host

`uchardet` has no prebuilt binary for the wasm profile, so every wasm run
builds it from source, and its tarball lives on freedesktop.org, which
answers a GitHub runner with 418 — a bot filter on data-centre IPs, not an
outage, so conan's retry loop can never win.

The conan cache cannot cover this. A branch sees only its own cache entries
and `main`'s, so a first run on a new branch is always cold; and `main` has
no wasm entry to fall back on, because `actions/cache` does not save when
the job fails and main's wasm job fails on exactly this. The failure
prevents the cache that would prevent the failure.

`core.sources:download_urls` puts ConanCenter's source backup — the same
tarballs, addressed by the sha256 the recipe already declares — ahead of
the recipe's own url, for every dependency rather than this one. `origin`
stays behind it for sources the backup does not carry. Every workflow
already runs `conan config install .github/config/conan` before resolving,
so no workflow file changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XABfEapaADjFQCt1vjmDF
@andiwand
andiwand force-pushed the ci/conan-source-backup branch from 660ad39 to fabb734 Compare August 31, 2026 06:43
@andiwand
andiwand merged commit 6b18f91 into main Aug 31, 2026
25 checks passed
@andiwand
andiwand deleted the ci/conan-source-backup branch August 31, 2026 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant