[fix][admin] PIP-478: off-load the admin's v4 credential and lend it a bounded auth executor - #26327
Open
lhotari wants to merge 1 commit into
Open
[fix][admin] PIP-478: off-load the admin's v4 credential and lend it a bounded auth executor#26327lhotari wants to merge 1 commit into
lhotari wants to merge 1 commit into
Conversation
…a bounded auth executor PIP-478 states that every synchronous v4 plugin call is off-loaded, and after the core migration that was true everywhere except one place: BaseResource resolved the credential for each admin request on whatever thread issued the call. HttpClient.computeAuthHeaders off-loads the identical composition for the lookup path and names the self-deadlock it avoids — an OAuth2 or Athenz shim's getAuthData() refreshes its credential over synchronous HTTP, and for a broker calling its own admin client the caller is a request-handling thread. The composition now runs on a blocking executor, and produces the plugin's headers verbatim. The v4 composition is off-loaded to the framework's shared pool rather than the admin's own, because BaseResource is constructed by 30 subclasses that would each have to thread an executor through; the shared pool is exactly what PIP-478 provides for a component with none to lend. What a services-aware plugin gets is different, and is the second half of this change. PulsarAdminImpl bound its framework services with a null blocking executor. That stopped being a live defect when the init context grew a shared-pool fallback, but it left the SASL-over-HTTP challenge rounds putting their GSSAPI work on one process-wide pool shared with every other client in the JVM, so a stalled KDC reached by one admin throttled authentication for all of them. The admin now lends a small bounded pool of its own, created lazily and shut down with the admin, which keeps that blast radius inside the admin that owns the plugin. Its request threads are still never lent out — that was the hazard the original null was avoiding. The scheduler stays unbound: nothing on the admin path schedules periodic authentication work. Instead the bound init context now falls back to the shared scheduler and blocking pool when the bound services leave one null, so the SPI's "never null" contract holds on every path — previously, binding partial services was worse for a third-party plugin than binding none at all, since the no-services context has always guaranteed non-null.
Contributor
There was a problem hiding this comment.
Pull request overview
Off-loads synchronous admin authentication work and improves executor isolation and fallback behavior.
Changes:
- Off-loads legacy v4 admin credential resolution.
- Adds an admin-owned authentication executor.
- Adds non-null executor fallbacks and regression tests.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
BoundInitContextFallbackTest.java |
Tests bound-context executor fallbacks. |
V5AuthContexts.java |
Adds shared scheduler and executor fallbacks. |
AdminAuthOffloadTest.java |
Tests admin credential off-loading. |
PulsarAdminImpl.java |
Adds the admin authentication executor lifecycle. |
BaseResource.java |
Off-loads legacy authentication composition. |
Suppressed comments (1)
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/BaseResource.java:167
- This helper does not keep the whole composition on a blocking executor when
authenticationStagecompletes asynchronously: thethenApplycontinuation below invokes synchronousnewRequestHeaderon the thread that completesstage, potentially a Jersey/Netty I/O thread. Offload that continuation too so every synchronous v4 hook is covered.
private CompletableFuture<Map<String, String>> v4AuthHeaders(URI uri) {
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // with none to lend. The admin's own bounded pool is still what a services-aware plugin gets, so | ||
| // the work that can actually stall — a KDC or IdP round trip inside the plugin — stays isolated | ||
| // per admin. | ||
| return V5AuthContexts.supplyBlocking(null, () -> v4AuthHeaders(uri)).thenCompose(headers -> headers); |
Comment on lines
+624
to
+625
| ThreadPoolExecutor executor = new ThreadPoolExecutor(AUTH_BLOCKING_MAX_THREADS, | ||
| AUTH_BLOCKING_MAX_THREADS, 60L, TimeUnit.SECONDS, new LinkedBlockingQueue<>(), |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main Issue: #25890
PIP: #25890
Motivation
PIP-478 states that every synchronous v4 authentication plugin call is off-loaded, and after the core
migration (#26282) that was true everywhere except one place:
BaseResourceresolved the credentialfor each admin request on whatever thread issued the call.
That is the same hazard the lookup path already fixed.
HttpClient.computeAuthHeadersoff-loads theidentical composition and names the self-deadlock it avoids — a v4 OAuth2 or Athenz shim's
getAuthData()refreshes its credential over a synchronous HTTP exchange, and for a broker callingits own admin client the caller is a request-handling thread.
BaseResourcewas the lastcaller-thread credential resolution left in the codebase, and the one place PIP-478's claim was still
an overstatement.
Separately,
PulsarAdminImplbound its framework authentication services with a null blockingexecutor. That stopped being a live defect once the init context grew a shared-pool fallback, but it
left the SASL-over-HTTP challenge rounds putting their GSSAPI work on one process-wide pool shared
with every other client in the JVM — so a stalled KDC reached by one admin throttled authentication
for all of them.
Finally,
AuthenticationInitContext.scheduler()andblockingExecutor()are documented as nevernull, and the no-services context honours that. The bound context did not: it returned whateverthe component supplied, so binding partial services was worse for a third-party plugin than binding
none at all.
Modifications
BaseResourceoff-loads the v4 composition. The deprecatedauthenticationStage(...)/newRequestHeader(...)hooks now run on a blocking executor viaV5AuthContexts.supplyBlocking, andproduce the plugin's headers verbatim — off-loading changes when the work runs, not what it returns.
The v4 composition goes to the framework's shared pool rather than the admin's own:
BaseResourceis constructed by 30 subclasses that would each have to thread an executor through, and the shared pool
is exactly what PIP-478 provides for a component with none to lend. The trade-off is stated at the call
site rather than left silent.
The admin lends its own bounded pool to services-aware plugins. A small
ThreadPoolExecutor(ceiling 8 — an admin issues REST calls, not a fan-out of connection attempts), created lazily, core
threads timing out, daemon threads,
shutdown()afterauth.close()so a plugin shutting down over itstill has it. This keeps a stalled identity provider's blast radius inside the admin that owns the
plugin. The admin's request threads are still never lent out — that was the hazard the original
nullwas avoiding, and it remains avoided.The scheduler stays unbound: nothing on the admin path schedules periodic authentication work, and
binding a scheduled pool per admin to sit idle would cost more than it buys.
The bound init context falls back.
BoundInitContext.scheduler()/blockingExecutor()nowsubstitute the shared instances when the bound services leave one
null, so the SPI's "never null"contract holds on every path.
Verifying this change
This change added tests and can be verified as follows:
AdminAuthOffloadTest— pins that the v4 credential is resolved off the thread that issued theadmin call, and that the composed headers are unchanged. Mutation-verified: reverting the
off-load to a direct call fails exactly this assertion and nothing else.
BoundInitContextFallbackTest— pins that a context bound with anullscheduler still suppliesone, and that the fallback does not shadow executors a component did bind.
./gradlew sanityCheckandquickCheckpass.Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Threading model: admin requests no longer resolve the v4 credential on the calling thread — the
composition moves to a blocking executor, and the admin gains a small bounded pool of its own for
services-aware plugins. No behaviour change in what is sent; only where it is computed.