Skip to content

pccs: report a missing rustls crypto provider instead of panicking - #80

Merged
ameba23 merged 1 commit into
flashbots:mainfrom
SeismicSystems:pccs-install-crypto-provider
Aug 13, 2026
Merged

pccs: report a missing rustls crypto provider instead of panicking#80
ameba23 merged 1 commit into
flashbots:mainfrom
SeismicSystems:pccs-install-crypto-provider

Conversation

@samlaf

@samlaf samlaf commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Human Summary

Forgot to setup a rustls provider in our code, and hence the collateral prefetcher was just panicing in the background and cluttering our logs with huge stack traces (see this for example):

thread 'integration::test_two_node_root_key_bootstrap' (398771) panicked at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-0.13.4/src/async_impl/client.rs:2484:5:
No rustls crypto provider is configured. When using the `rustls-no-provider` feature you must install a crypto provider before building a Client. For example: `rustls::crypto::aws_lc_rs::default_provider().install_default().unwrap();` See https://docs.rs/rustls/latest/rustls/#cryptography-providers for details.
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/panicking.rs:689:5
   1: core::panicking::panic_fmt
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/panicking.rs:80:14
   2: reqwest::async_impl::client::default_rustls_crypto_provider
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-0.13.4/src/async_impl/client.rs:2484:5
   3: <reqwest::async_impl::client::default_rustls_crypto_provider as core::ops::function::FnOnce<()>>::call_once
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/ops/function.rs:250:5
   4: <core::option::Option<alloc::sync::Arc<rustls::crypto::CryptoProvider>>>::unwrap_or_else::<reqwest::async_impl::client::default_rustls_crypto_provider>
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/option.rs:1064:21
   5: <reqwest::async_impl::client::ClientBuilder>::build
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-0.13.4/src/async_impl/client.rs:721:26
   6: <pccs::Pccs>::fetch_fmspcs::{closure#0}
             at /home/azureuser/.cargo/git/checkouts/attested-tls-58059c9da45390cb/4fb3f4e/crates/pccs/src/lib.rs:396:82
   7: <pccs::Pccs>::startup_prewarm_all_tdx::{closure#0}
             at /home/azureuser/.cargo/git/checkouts/attested-tls-58059c9da45390cb/4fb3f4e/crates/pccs/src/lib.rs:300:48
   8: <pccs::Pccs>::new::{closure#0}
             at /home/azureuser/.cargo/git/checkouts/attested-tls-58059c9da45390cb/4fb3f4e/crates/pccs/src/lib.rs:79:70
   9: <tokio::runtime::task::core::Core<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll::{closure#0}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/core.rs:375:24
  10: <tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<<pccs::Pccs>::new::{closure#0}>>>::with_mut::<core::task::poll::Poll<()>, <tokio::runtime::task::core::Core<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll::{closure#0}>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/loom/std/unsafe_cell.rs:16:9
  11: <tokio::runtime::task::core::Core<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/core.rs:364:30
  12: tokio::runtime::task::harness::poll_future::<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>::{closure#0}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:535:30
  13: <core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>::{closure#0}> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/panic/unwind_safe.rs:275:9
  14: std::panicking::catch_unwind::do_call::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>::{closure#0}>, core::task::poll::Poll<()>>
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/panicking.rs:581:40
  15: __rust_try
  16: std::panicking::catch_unwind::<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>::{closure#0}>>
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/panicking.rs:544:19
  17: std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>::{closure#0}>, core::task::poll::Poll<()>>
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/panic.rs:359:14
  18: tokio::runtime::task::harness::poll_future::<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:523:18
  19: <tokio::runtime::task::harness::Harness<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll_inner
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:210:27
  20: <tokio::runtime::task::harness::Harness<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::poll
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:155:20
  21: tokio::runtime::task::raw::poll::<<pccs::Pccs>::new::{closure#0}, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/raw.rs:337:13
  22: <tokio::runtime::task::raw::RawTask>::poll
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/raw.rs:267:18
  23: <tokio::runtime::task::LocalNotified<alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>::run
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/mod.rs:510:13
  24: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:816:30
  25: tokio::task::coop::with_budget::<(), <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/task/coop/mod.rs:167:5
  26: tokio::task::coop::budget::<(), <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/task/coop/mod.rs:133:5
  27: <tokio::runtime::scheduler::current_thread::Context>::run_task::<(), <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}>::{closure#0}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:368:43
  28: <tokio::runtime::scheduler::current_thread::Context>::enter::<(), <tokio::runtime::scheduler::current_thread::Context>::run_task<(), <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}>::{closure#0}>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:446:19
  29: <tokio::runtime::scheduler::current_thread::Context>::run_task::<(), <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#1}>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:368:28
  30: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:812:43
  31: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:856:68
  32: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context/scoped.rs:40:9
  33: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context.rs:176:38
  34: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::try_with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/thread/local.rs:462:12
  35: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/thread/local.rs:426:20
  36: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context.rs:176:17
  37: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:856:27
  38: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:756:24
  39: <tokio::runtime::scheduler::current_thread::CurrentThread>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:200:33
  40: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context/runtime.rs:65:16
  41: <tokio::runtime::scheduler::current_thread::CurrentThread>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/current_thread/mod.rs:188:9
  42: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/runtime.rs:371:52
  43: <tokio::runtime::runtime::Runtime>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at /home/azureuser/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/runtime.rs:345:18
  44: integration::integration::test_two_node_root_key_bootstrap
             at ./tests/integration/integration.rs:55:1
  45: integration::integration::test_two_node_root_key_bootstrap::{closure#0}
             at ./tests/integration/integration.rs:55:45
  46: <integration::integration::test_two_node_root_key_bootstrap::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/ops/function.rs:250:5
  47: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

LLM Summary

The workspace pins reqwest with rustls-no-provider, so building a
reqwest::Client requires the application to have installed a
process-level rustls crypto provider. That requirement was documented
nowhere and enforced only by the #[cfg(test)] install helpers, and the
one client pccs builds itself lives in fetch_fmspcs — called from the
pre-warm task Pccs::new spawns in its constructor. An application that
never installed a provider therefore got a panic inside a detached
task it cannot catch, dumping a full backtrace on every Pccs
construction.

That panic was worse than log noise:

  • Pccs::ready() deadlocked: the task died before finish_prewarm, so
    the outcome channel never resolved — and it cannot even close,
    because the Pccs instance being awaited holds the sender alive. An
    application doing the responsible thing and waiting for the cache
    before serving hung forever, with no error and no timeout.
  • Under panic = "abort" the same code killed the whole process, so
    the library's failure mode ranged from invisible to fatal depending
    on the consumer's build profile.

The blast radius was otherwise confined to the pre-warm: on-demand
collateral fetches and background refreshes go through dcap-qvl, whose
client bundles its own TLS provider, so verification kept working
while the FMSPC-discovery step died silently.

Check for the provider before building the client and return a new
PccsError::MissingCryptoProvider naming the fix. The pre-warm then
degrades through its existing failed-fetch path — one warning line
stating the consequence (no warm cache, collateral fetched on demand)
and the remedy — and ready() surfaces the same error immediately, so
applications that consider a warm cache mandatory get fail-fast
semantics. The precondition is now documented on Pccs.

Deliberately not chosen: installing aws-lc-rs as a silent fallback
when no provider is set. That would force the aws-lc-rs build
dependency on every consumer — the choice rustls-no-provider exists to
leave with the application — and an application installing its own
provider late with install_default().unwrap() could lose the race
against the pre-warm task. The rustls dependency added here carries no
provider feature: only the process-default lookup is used.

The workspace pins reqwest with rustls-no-provider, so building a
reqwest::Client requires the application to have installed a
process-level rustls crypto provider. That requirement was documented
nowhere and enforced only by the #[cfg(test)] install helpers, and the
one client pccs builds itself lives in fetch_fmspcs — called from the
pre-warm task Pccs::new spawns in its constructor. An application that
never installed a provider therefore got a panic inside a detached
task it cannot catch, dumping a full backtrace on every Pccs
construction.

That panic was worse than log noise:

- Pccs::ready() deadlocked: the task died before finish_prewarm, so
  the outcome channel never resolved — and it cannot even close,
  because the Pccs instance being awaited holds the sender alive. An
  application doing the responsible thing and waiting for the cache
  before serving hung forever, with no error and no timeout.
- Under panic = "abort" the same code killed the whole process, so
  the library's failure mode ranged from invisible to fatal depending
  on the consumer's build profile.

The blast radius was otherwise confined to the pre-warm: on-demand
collateral fetches and background refreshes go through dcap-qvl, whose
client bundles its own TLS provider, so verification kept working
while the FMSPC-discovery step died silently.

Check for the provider before building the client and return a new
PccsError::MissingCryptoProvider naming the fix. The pre-warm then
degrades through its existing failed-fetch path — one warning line
stating the consequence (no warm cache, collateral fetched on demand)
and the remedy — and ready() surfaces the same error immediately, so
applications that consider a warm cache mandatory get fail-fast
semantics. The precondition is now documented on Pccs.

Deliberately not chosen: installing aws-lc-rs as a silent fallback
when no provider is set. That would force the aws-lc-rs build
dependency on every consumer — the choice rustls-no-provider exists to
leave with the application — and an application installing its own
provider late with install_default().unwrap() could lose the race
against the pre-warm task. The rustls dependency added here carries no
provider feature: only the process-default lookup is used.
samlaf added a commit to SeismicSystems/enclave that referenced this pull request Aug 12, 2026
…#246)

Fixes SEI-201

Quote verification's collateral fetching (attested-tls) builds
rustls-backed HTTP clients. rustls 0.23 requires the application to
install a process-level default crypto provider; without one, the first
verification panics with "no process-level CryptoProvider available".

Install the aws-lc-rs provider at service start, before anything
verifies evidence. The install is best-effort: a provider installed
earlier by an embedding process wins, and any provider serves.

Related to, but orthogonal to, our upstream PR
flashbots/attested-tls#80.

@ameba23 ameba23 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch and sorry for not documenting. Ideally we want to not lock in to either aws-lc or ring but let users of the library choose. But it does cause issues like this.

@ameba23
ameba23 merged commit 41c474a into flashbots:main Aug 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants