[improve][misc] PIP-478: remove the superseded PIP-337 TLS stack - #26322
[improve][misc] PIP-478: remove the superseded PIP-337 TLS stack#26322lhotari wants to merge 4 commits into
Conversation
fa40bd0 to
4afd91a
Compare
4afd91a to
cd252c9
Compare
…ry stack The PIP-337 TLS infrastructure has no callers left. The core migration replaced it with the PulsarTlsFactory SPI and the common.util.tls helpers, and the configuration keys that once named DefaultPulsarSslFactory by class name (sslFactoryPlugin, brokerClientSslFactoryPlugin) are gone — they now sit on the removed-key tolerance lists, which PulsarConfigurationLoaderTest and ConfigurationDataUtilsTest already pin. Deleted, each verified unreachable from any production path: PulsarSslFactory, DefaultPulsarSslFactory, PulsarSslConfiguration, KeyManagerProxy, TrustManagerProxy, KeyStoreSSLContext, SSLContextValidatorEngine, and the four tests that exist only to exercise them. SSLContextValidatorEngine had no references at all, not even from its sibling. SecurityUtility.createAutoRefreshSslContextForClient goes with them: it was the sole production referent of KeyManagerProxy and TrustManagerProxy and had no callers of its own. The rest of SecurityUtility is removed in the next commit, once its last live consumer is migrated. Despite its name and package, SslContextTest is not a general Netty test — every case constructs a DefaultPulsarSslFactory with a PulsarSslConfiguration. Assisted-by: Claude Code (Opus 5)
…move SecurityUtility SecurityUtility had exactly one live consumer left after the PIP-337 stack went: message crypto's BouncyCastle provider lookup. Point it at the decomposed equivalent and delete the monolith. MessageCryptoBc now resolves the provider through JcaProviders.requireBouncyCastleProvider(), which performs the same FIPS-agnostic lookup — non-FIPS "BC" or FIPS "BCFIPS", whichever is on the classpath. The lookup stays inside the initialization-on-demand holder, so a resolution failure still surfaces at first asymmetric-crypto use rather than at class loading. One deliberate difference: an absent BouncyCastle now fails with an IllegalStateException naming the caller instead of a RuntimeException wrapping the classpath-load failure. Same failure point, better message. Conscrypt registration is unaffected. Both classes register it eagerly in their static initializers, so it still lands in the JCA provider list; what SecurityUtility additionally installed — a process-global CN-tolerant hostname verifier — is dealt with in the next commit, where the classes it depended on are removed. Assisted-by: Claude Code (Opus 5)
…hing Pulsar carried its own hostname verifier — TlsHostnameVerifier plus SubjectName, PublicSuffixMatcher, PublicSuffixList and DomainType — whose only referent was SecurityUtility, now gone. Delete them; verification is whatever the configured provider implements under the standard endpoint-identification algorithm "HTTPS". Be precise about the compatibility story, because the obvious statement of it is wrong. The standard algorithm is NOT SAN-only. Per RFC 6125 the JDK and OpenSSL/BoringSSL engines consult a certificate's CN when it carries no dNSName SAN, and ignore the CN once any is present — verified here with real JDK handshakes against this repo's own fixtures: the CN-only hn-verification/broker-cert.pem is accepted for its CN host and rejected for another, and the SAN-bearing broker.cert.pem is rejected for its CN host. So on the default engines a CN-only certificate is accepted exactly as before. What does change is Conscrypt-pinned clients. SecurityUtility's static initializer installed the CN-tolerant verifier as Conscrypt's process-global default; without it, Conscrypt applies its own RFC 2818 verification and rejects a server certificate with no subjectAltName. That reaches a deployment only if it names Conscrypt as its client-side JSSE provider and still uses CN-only server certificates. Conscrypt is never the client-side default — it is defaulted only for TlsPurpose.WEB, i.e. server listeners, which do not verify hostnames. Four documents asserted the SAN-only guarantee and are corrected rather than kept: pip-478.md's summary line, its hostname-matching bullet, its breaking-change note, and the common.tls package javadoc. Deliberately no new "CN-only must be rejected" test: it would fail today on the default engines, which is the point. Both in-repo SAN-less server certificates are negative fixtures on the default engine and keep passing for the same reason as before — AuthenticationTlsHostnameVerificationTest fails on CN mismatch, not on SAN absence. Assisted-by: Claude Code (Opus 5)
…record Review found that three of the four CN/SAN passages were corrected and the fourth was not — and the one missed is the breaking-change bullet under Backward & Forward Compatibility, which is exactly where the other two send compatibility readers. It still announced "SAN required" and "a CN-only certificate no longer suffices", contradicting the Security Considerations text two sections above. An operator following the document's own routing would have reissued fleet certificates for a break that does not exist on the default engines. The bullet now states what actually changes: hostname verification is on by default, a matching SAN always suffices, the CN is still consulted on the default engines for a certificate carrying no dNSName SAN, and Conscrypt-pinned clients are what tighten. The mechanism is stated precisely too. Netty's native OpenSSL/BoringSSL engine does not verify the hostname itself — it delegates to the Java trust manager — so the JDK's HostnameChecker and its CN fallback govern both default engines, not BoringSSL's own matching. Also swept the framing out of the code: the broker TLS support javadoc claimed PIP-478 turns on "SAN-only verification", and eleven proxy tests plus two certificate-generation scripts carried the same phrase, so the corrected record would have been contradicted by the next file a reader opened. Two mislabels found in the same review, pre-existing but adjacent: the trust-manager fallback is documented as being for "a provider that offers none (e.g. Conscrypt)", but real Conscrypt registers both PKIX factories — the fallback is for a third-party provider, and the test stub named "Conscrypt-shaped" models a provider that does not exist. Renamed and re-described. The same javadoc still advertised a Conscrypt hostname-verifier propagation workaround whose body was removed when Conscrypt 2.6.1 landed. Assisted-by: Claude Code (Opus 5)
cd252c9 to
f97d49d
Compare
david-streamlio
left a comment
There was a problem hiding this comment.
A −3265 removal PR lives or dies on two questions: is the removed code actually dead, and is the compatibility story true. I checked both independently rather than reading them off the description, and the substance holds up well. The retraction in the compatibility section is the right call and clearly took work to get right.
Deadness — verified. No dangling import of any of the thirteen removed types anywhere in the tree, and no fully-qualified usage except the two deliberate string literals (PulsarConfigurationLoader:68, ConfigurationDataUtils:50) that name the old default FQCN for the tolerance check — which is exactly right, since those must survive the class they name. Every remaining textual hit is a false positive: Athenz's keyRefresher.getKeyManagerProxy() is an Athenz-library type, SchemaRegistryServiceImpl's convertFromDomainType is schema code, the rest are comments and PIP prose.
The BouncyCastle move — semantics preserved. Same resolution order (registered BC → BCFIPS → classpath), Security.addProvider side effect retained on the classpath path, and MessageCryptoBc's own BcProviderHolder is what preserves first-use timing, so the claim survives the move to a lazy holder. I also checked the failure mode a removal like this invites — nothing in the tree, production or test, resolves BouncyCastle by name (Security.getProvider("BC") / getInstance(…, "BC")), so losing SecurityUtility's eager registration strands nothing. Conscrypt genuinely is an eager static final field in both classes, so that part of the claim is exact.
Deleted tests — coverage has successors. SslContextTest is PIP-337-only despite the generic name, and each of its four methods maps onto a live one: serverPemContextAcrossEngineAndCiphers / clientPemContextAcrossEngineAndCiphers for the SslProvider×cipher matrix, serverKeystoreContextBuildsWithCiphers / clientKeystoreContextBuildsWithCiphers for the keystore paths, plus defaultProtocolsAppliedWhenPolicyLeavesThemUnset for the TLSv1.3/1.2 floor.
The compatibility claim itself — verified at the source. SecurityUtility.loadConscryptProvider() really did call Conscrypt.setDefaultHostnameVerifier(wrap(new TlsHostnameVerifier())) process-globally from a static field initializer, so scoping the break to Conscrypt-pinned clients is correct. The base branch's own JcaProviders CAVEAT predicted this PR almost verbatim. ClusterData retention is backed by real code (BrokerService.warnOnStalePip337ClusterFactory, deduped per cluster), the config tolerance does the right thing (default FQCN treated as unset, custom value rejected with a migration message), and the JdkSslContexts javadoc line about the Conscrypt propagation workaround was genuinely stale — no such code remains in the method.
Three things:
1. The sweep is incomplete, and it misses in the two suites the PR cites as its evidence.
The stated goal is that "the corrected record is not contradicted by the next file a reader opens." Three files still contradict it:
AuthenticationTlsHostnameVerificationTest.java:143— "CN-based matching is no longer supported (PIP-478)." That is verbatim the retracted claim. Also:128and:182, "SAN-based (RFC 2818) host-name verification". This file is not in the diff at all, and it is the first suite the PR names under Verifying this change.ProxyWithAuthorizationTest.java:77-79— "SAN-based (RFC 2818) hostname verification … CN-based matching is no longer supported (PIP-478, Pulsar 5.0), so this cert always fails hostname verification." This file is in the diff — the sweep corrected lines 186 and 245 and stopped short of this one.JcaProviders.java:298— "standard RFC 2818 (SAN-based) verification, matching PIP-478's removal of the deprecated CN-based matching" — three lines above the corrected text this PR adds at :300.
The ProxyWithAuthorizationTest one is worth more than staleness: it states the wrong reason the test passes. The PR's own analysis is that these suites "fail on CN mismatch, not on SAN absence" — the comment tells the next reader the opposite, next to the fixture that would mislead them.
I checked whether these are simply deferred to a later PR in the stack: they are not. All three survive unchanged at lh-pip-478-proxy-async-auth (#26328), the top of the stack, which contains this branch plus #26326 and #26327.
2. "RFC 2818" is the wrong citation, and the PR body repeats it.
RFC 2818 §3.1 mandates the CN fallback: "If a subjectAltName extension of type dNSName is present, that MUST be used as the identity. Otherwise, the (most specific) Common Name field … MUST be used." So Conscrypt refusing a SAN-less certificate is stricter than RFC 2818, not "its own RFC 2818 verification" (PR body, and pip-478.md:1247). The phrase is inherited from the SecurityUtility comment being deleted, which had it wrong too.
package-info.java gets this exactly right — "per RFC 6125 the JDK and OpenSSL engines consult the CN … Conscrypt does not fall back to the CN" — so the code doc and the PIP now characterize the same behaviour with different standards. Since this passage is the corrected record, I'd align the PIP and body on the package-info's framing: RFC 6125 for the fallback, and "does not implement the CN fallback RFC 2818 allows" for Conscrypt.
3. Nit: two build-file comments now misdescribe the timing.
pulsar-common/build.gradle.kts and pulsar-broker-common/build.gradle.kts read "tests that exercise JcaProviders (which loads org.bouncycastle.jce.provider.BouncyCastleProvider in a static initializer)". True of SecurityUtility (BC_PROVIDER was an eager static final), not of JcaProviders, which resolves BC through BouncyCastleHolder on first use — the very property the PR highlights as preserved. Mechanical rename carrying over a clause that no longer describes the class.
None of this touches the removal itself, which I think is sound. 1 and 2 are about the corrected record being self-consistent — and given that this PR exists partly to retract a false compatibility claim, a reader landing on AuthenticationTlsHostnameVerificationTest:143 would come away with the retracted version.
For disclosure: this is static analysis over the stacked branch. I did not run sanityCheck or the cited suites locally.
There was a problem hiding this comment.
Pull request overview
Removes the superseded PIP-337 TLS stack and documents provider-specific hostname verification behavior.
Changes:
- Deletes obsolete TLS factories, utilities, hostname verifier classes, and tests.
- Moves message-crypto provider lookup to
JcaProviders. - Corrects SAN-only compatibility claims.
Reviewed changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/certificate-authority/generate_keystore.sh |
Revises certificate guidance. |
tests/certificate-authority/ec/jks/key_store_generation.txt |
Revises SAN guidance. |
pulsar-proxy/.../SslContextFallbackSynthesisTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyWithoutServiceDiscoveryTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyWithAuthorizationTest.java |
Revises hostname-verification comments. |
pulsar-proxy/.../ProxyTlsTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyTlsFactoryTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyServiceTlsStarterTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyMutualTlsTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyKeyStoreTlsWithoutAuthTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyKeyStoreTlsWithAuthTest.java |
Corrects TLS comment. |
pulsar-proxy/.../ProxyKeyStoreTlsTransportTest.java |
Corrects TLS comment. |
pulsar-common/.../TrustManagerProxyTest.java |
Removes obsolete proxy test. |
pulsar-common/.../tls/JdkSslContextsTest.java |
Corrects provider test terminology. |
pulsar-common/.../netty/SslContextTest.java |
Removes legacy factory tests. |
pulsar-common/.../KeyManagerProxyTest.java |
Removes obsolete proxy test. |
pulsar-common/.../DefaultPulsarSslFactoryTest.java |
Removes legacy factory tests. |
pulsar-common/.../TrustManagerProxy.java |
Removes obsolete trust-manager proxy. |
pulsar-common/.../tls/JdkSslContexts.java |
Updates provider documentation. |
pulsar-common/.../tls/JcaProviders.java |
Documents native Conscrypt behavior. |
pulsar-common/.../SecurityUtility.java |
Removes superseded TLS utility. |
pulsar-common/.../PulsarSslFactory.java |
Removes PIP-337 SPI. |
pulsar-common/.../PulsarSslConfiguration.java |
Removes legacy configuration model. |
pulsar-common/.../keystoretls/SSLContextValidatorEngine.java |
Removes unused validator. |
pulsar-common/.../keystoretls/package-info.java |
Removes obsolete package metadata. |
pulsar-common/.../keystoretls/KeyStoreSSLContext.java |
Removes legacy keystore context. |
pulsar-common/.../KeyManagerProxy.java |
Removes obsolete key-manager proxy. |
pulsar-common/.../DefaultPulsarSslFactory.java |
Removes legacy factory implementation. |
pulsar-common/.../tls/TlsHostnameVerifier.java |
Removes custom hostname verifier. |
pulsar-common/.../tls/SubjectName.java |
Removes verifier helper. |
pulsar-common/.../tls/PublicSuffixMatcher.java |
Removes verifier helper. |
pulsar-common/.../tls/PublicSuffixList.java |
Removes verifier helper. |
pulsar-common/.../tls/package-info.java |
Documents provider-specific CN fallback. |
pulsar-common/.../tls/impl/TlsContexts.java |
Updates legacy references. |
pulsar-common/.../tls/DomainType.java |
Removes verifier helper. |
pulsar-common/build.gradle.kts |
Updates dependency comment. |
pulsar-client-messagecrypto-bc/.../MessageCryptoBc.java |
Uses JcaProviders for Bouncy Castle. |
pulsar-client-messagecrypto-bc/build.gradle.kts |
Updates provider-resolution comment. |
pulsar-broker-common/.../TlsFactorySupport.java |
Corrects hostname-verification documentation. |
pulsar-broker-common/build.gradle.kts |
Updates dependency comment. |
pip/pip-478.md |
Corrects compatibility and upgrade guidance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # enabled by default in 5.0 (PIP-478), the server cert must | ||
| # carry a SubjectAltName covering the connected host, otherwise every peer that now verifies hostnames | ||
| # rejects it with "No subject alternative names present". Tests reach the broker/proxy over loopback |
| // hostname verification is on by default (PIP-478); the default advertised address resolves | ||
| // to the machine's canonical hostname, which is not in the cert SAN (fails on CI runners). The two | ||
| // dedicated hostname-verification tests still fail as intended because their server presents a no-SAN | ||
| // certificate, which cannot match any host regardless of the advertised address. |
|
Consolidating the sweep into one checklist, now that the Copilot review has landed — it found two sites I missed, and chasing those turned up an eighth. Same defect in all of them: the retracted "a CN-only certificate is no longer accepted" claim, in the wording each file happens to use.
Six of the eight are in files this PR already edits, and two of those ( Why my original three missed the rest, in case it helps you check for a ninth: I grepped for the retracted phrasings — The last one is milder than the others and I'd treat it separately: Nothing here changes my read of the removal itself, which I still think is sound. |
Main Issue: #25890
PIP: #25890
Motivation
With the PIP-478 migration complete — the
PulsarTlsFactorySPI, the default file-based factory, the core migration and the v5-native auth inversion all landed — the PIP-337 TLS infrastructure it replaced has no callers left. So does theSecurityUtilitymonolith that PIP-478 decomposed into thecommon.util.tlshelpers, and the CN-based hostname-matching machinery whose only referent was that monolith.Removing them is the point of the exercise: PIP-478 justified its breaking changes partly on the maintenance cost of carrying two TLS stacks, and that cost is only actually paid down when the old one goes.
−3239 lines, +23.
Modifications
Three commits, each independently compiling, ordered so the reference graph unwinds cleanly:
PulsarSslFactory,DefaultPulsarSslFactory,PulsarSslConfiguration,KeyManagerProxy,TrustManagerProxy,KeyStoreSSLContext,SSLContextValidatorEngine, and the four tests that exist only to exercise them. Each verified unreachable, including by class-name string — the configuration keys that once named the default factory are gone and now sit on the removed-key tolerance lists, whichPulsarConfigurationLoaderTestandConfigurationDataUtilsTestalready pin.SecurityUtility.createAutoRefreshSslContextForClientgoes with them: it was the sole production referent of the two proxies and had no callers of its own.SecurityUtility. One live consumer remained — message crypto's BouncyCastle provider lookup — which moves toJcaProviders.requireBouncyCastleProvider(). Same FIPS-agnostic resolution (non-FIPSBCor FIPSBCFIPS, whichever is present), same initialization-on-demand timing, so a resolution failure still surfaces at first asymmetric-crypto use rather than at class loading. Conscrypt registration is unaffected: both classes register it in their static initializers.TlsHostnameVerifier,PublicSuffixMatcher,PublicSuffixList,SubjectName,DomainType.The compatibility statement, stated carefully
An earlier draft of this change announced that hostname verification becomes SAN-only and a CN-only certificate is no longer accepted. That is false, and shipping it would have been worse than saying nothing: operators would have reissued certificates for a break that does not exist, and the deployments that genuinely do break would have had no warning.
Verified with real JDK handshakes against this repository's own fixtures:
dNSNameSAN, and ignored once any is present. This holds on the JDK engine and on Netty's native OpenSSL/BoringSSL engine — the native engine does not verify hostnames itself, it delegates to the Java trust manager, so the JDK'sHostnameCheckergoverns both.SecurityUtility's static initializer installed the CN-tolerantTlsHostnameVerifieras Conscrypt's process-global default. With it gone, Conscrypt applies its own RFC 2818 verification and rejects a server certificate with nosubjectAltName. A deployment is affected only if it names Conscrypt as its client-side JSSE provider and still uses CN-only server certificates; the remedy is to reissue those with the hostname in the SAN. Conscrypt is never the client-side default — it is defaulted only forTlsPurpose.WEB, i.e. server listeners, which do not verify hostnames.Four documents asserted the wrong claim and are corrected, including the breaking-change bullet under Backward & Forward Compatibility that the other corrected passages route readers to. Eleven test comments and two certificate-generation scripts repeated the phrase and are swept too, so the corrected record is not contradicted by the next file a reader opens.
Deliberately no new "a CN-only certificate must be rejected" test: it would fail today on the default engines, which is the whole point.
What this PR does not remove
An earlier draft also removed
ClusterData.brokerClientSslFactoryPlugin/...Params. That is dropped, because its stated rationale — "factory selection is broker-level" — was refuted by the core migration, which made per-cluster selection real.pip-478.mdargues their retention as a deliberate compatibility decision in nine places, one of them promising that a 5.0 broker "preserves it rather than erasing it on rewrite". Removing them would also delete the only operator-facing signal for a stale value, and erase the fields from stored metadata on any peer-cluster or migration update — silently downgrading a 4.x broker in a mixed fleet to the default factory.Verifying this change
This change is a removal covered by existing tests:
sanityCheck(all modules, main and test sources) andquickCheckpass — the deletion's correctness is a reference-graph property, and a full compile is what checks it.:pulsar-common:test --tests '*tls*' --tests '*Crypto*'— the decomposed helpers and the migrated BouncyCastle lookup.AuthenticationTlsHostnameVerificationTestandProxyWithAuthorizationTest— the two suites carrying SAN-less negative fixtures, which would notice if hostname behaviour moved. Both still pass, and for the same reason as before: they fail on CN mismatch, not on SAN absence.Deadness was verified independently by review, including reflective and class-name-string loading, with the false positives triaged.
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
The public API: thirteen public classes in
pulsar-commonare removed, all of them the PIP-337 stack or its CN-matching helpers, and all inventoried in PIP-478's removal impact section.InetAddressUtilsandNoopHostnameVerifiersurvive in the same package.Deployment: see the compatibility statement above — a Conscrypt-pinned client using CN-only server certificates must reissue them.
Documentation
doc-requireddoc-not-neededdocdoc-completeThe operator-facing consequence is documented in
pip-478.md's compatibility section, which this PR corrects.Matching PR in forked repository
PR in forked repository: lhotari#254
Prepared with the assistance of Claude Code (Opus 5).
Stack created with GitHub Stacks CLI • Give Feedback 💬