feat(kernel): thread Azure Entra OAuth (U2M + SP M2M) through the auth bridge

CHANGELOG.md

@@ -1,7 +1,8 @@
 # Release History
 
 # Unreleased
-- Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision. `auth_type="azure-oauth"` (Azure AD) is not yet supported on the kernel path and raises `NotSupportedError` — use the Thrift backend for it (PECOBLR-4040; Azure tracked by PECOBLR-4120)
+- Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision (PECOBLR-4040)
+- Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) OAuth is now supported.** The kernel is the Azure-aware auth core (it owns the endpoints, scopes, app ids, and tenant discovery); the connector forwards the selector and Azure credentials unchanged, so `connect()` is byte-identical between the Thrift and `use_kernel=True` paths. `auth_type="azure-oauth"` (Azure AD U2M) runs the kernel's browser flow against the workspace v2.0 authorize/token endpoints with the Azure app client id (`96eecda7-…`), redirect port `8030`, and the `{app_id}/user_impersonation offline_access` scope. `auth_type="azure-sp-m2m"` (Azure service principal) forwards `azure_client_id` / `azure_client_secret`; the kernel builds the Entra v2.0 token endpoint and the `{app_id}/.default` scope, and **auto-discovers the tenant** from the workspace's `/aad/auth` redirect when `azure_tenant_id` is omitted (matching Thrift). `azure_workspace_resource_id` is an optional add-on: set it and the kernel additionally sends the Azure management-token header pair (`X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id`) so an SP that holds only an Azure RBAC role (not a workspace member) can authenticate; omit it and the SP authenticates with the data token alone (PECOBLR-4141; PECOBLR-4120)
 
 # 4.4.0 (2026-07-22)
 - Raised the minimum supported Python version to 3.10, dropping the end-of-life 3.8/3.9, to update the lockfile and clear CVE-flagged dependencies in the repo (databricks/databricks-sql-python#798)

CONNECTION_PARAMETERS.md

@@ -69,15 +69,16 @@ to change without notice.
 | Option                                              | Type                 | Thrift | Kernel | Default Value             | Note                                                                                                                                                            |
 | --------------------------------------------------- | -------------------- | :----: | :----: | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `access_token` (PAT)                                | `str`                |   ✅   |   ✅   | `None`                    | Personal Access Token / bearer token. The default auth mode when set; otherwise auth falls back to OAuth.                                                       |
-| `auth_type`                                         | `str`                |   ✅   |   ✅   | `None` ⇒ Databricks OAuth | `databricks-oauth` or `azure-oauth`.                                                                                                                            |
+| `auth_type`                                         | `str`                |   ✅   |   ✅   | `None` ⇒ Databricks OAuth | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three work on the kernel path; `azure-oauth` / `azure-sp-m2m` kernel support added in #919 (the connector forwards the selector + Azure credentials and the kernel owns Azure resolution). |
 | `oauth_client_id` (U2M)                             | `str`                |   ✅   |   ✅   | built-in client id        | Custom U2M client id. Forwarded on both; when absent, each path applies its own built-in default.                                                              |
 | `oauth_redirect_port` (U2M)                         | `int`                |   ✅   |   ✅   | `None`                    | Localhost redirect port for the browser flow. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) |
 | `oauth_client_secret` (OAuth M2M)                   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only in practice.** The Thrift auth path never reads `oauth_client_secret`; use `credentials_provider` or an Azure service principal for M2M on Thrift. |
 | `oauth_scopes`                                      | `List[str]`          |   ❌   |   ✅   | `["sql","offline_access"]`| **Thrift ignores custom scopes** — it always uses the built-in scope set. Only the kernel honors a custom `oauth_scopes`.                                       |
 | `credentials_provider`                              | `CredentialsProvider`|   ✅   |   ❌   | `None`                    | Custom external credentials provider. **Rejected on the kernel path** (`NotSupportedError`) — it is an opaque token source, so the kernel cannot own the token lifecycle; use `oauth_client_id` + `oauth_client_secret` for M2M, or the Thrift backend. |
 | `identity_federation_client_id`                     | `str`                |   ✅   |   ✅   | `None`                    | Workload identity / token-federation client id (kernel support added in #910).                                                                                 |
 | `experimental_oauth_persistence`                    | `OAuthPersistence`   |   ✅   |   ❌   | `None`                    | **Thrift-only.** The kernel owns its own token lifecycle and does not accept a persistence store.                                                              |
-| `azure_client_id` / `azure_client_secret` / `azure_tenant_id` / `azure_workspace_resource_id` | `str` | ✅ | ❌ | `None` | **Thrift-only.** The Azure service-principal (Entra ID M2M) fields are not forwarded to the kernel. (Azure *U2M* still works on the kernel via `auth_type="azure-oauth"`, the browser flow.) |
+| `azure_client_id` / `azure_client_secret` / `azure_tenant_id` | `str` | ✅ | ✅ | `None` | Azure service-principal (Entra ID M2M), selected by `auth_type="azure-sp-m2m"`. On the kernel path the connector forwards these to the kernel, which owns Azure resolution (Entra v2.0 token endpoint + the Databricks-resource `.default` scope) (#919). **`azure_tenant_id` is optional on the kernel path too** — like Thrift, the kernel auto-discovers it from the workspace's `/aad/auth` redirect when omitted. |
+| `azure_workspace_resource_id`                       | `str`                |   ✅   |   ✅   | `None`                    | Optional add-on for `azure-sp-m2m`. When set, the SP **management token** (`X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id`) is sent to authorize an SP that has an Azure RBAC role but is not a workspace member. On the kernel path the connector forwards it and the kernel fetches the management token and emits the header pair; omit it and the SP authenticates with the Databricks-audience data token alone. |
 | `_use_cert_as_auth` (+ `_tls_client_cert_file`)     | `bool`               |   ✅   |   ❌   | `False`                   | Authenticate with a TLS client certificate instead of a token. Thrift-only.                                                                                    |
 | `username` / `password`                             | `str`                |   ❌   |   ❌   | `None`                    | **Removed.** Basic auth is no longer supported; passing either raises `ValueError`.                                                                            |
 

src/databricks/sql/backend/kernel/auth_bridge.py

@@ -20,11 +20,26 @@
   app bundle (``client_id`` + ``redirect_ports`` list, with the optional
   ``oauth_client_id`` / ``oauth_redirect_port`` overriding it) is
   forwarded to the kernel's ``auth_type='oauth-u2m'`` and the kernel
-  runs the browser flow itself. ``azure-oauth`` (Azure AD) is **not yet
-  supported** on the kernel path and is rejected with
-  ``NotSupportedError`` — the kernel resolves OAuth endpoints only from
-  the workspace-native OIDC config and cannot drive the Azure AD flow
-  (PECOBLR-4120).
+  runs the browser flow itself.
+- **Azure Entra (Azure AD)** — both Azure auth types forward the selector
+  and Azure credentials to the KERNEL, which is the Azure-aware auth core
+  (it owns the endpoints, scopes, app ids, and tenant discovery). The
+  binding stays thin — it does not construct endpoints or scopes:
+
+  - ``azure-oauth`` (U2M) → forward ``auth_type='azure-oauth'`` (plus any
+    optional ``oauth_client_id`` / ``oauth_redirect_port`` passthrough). The
+    kernel pins the workspace v2.0 authorize/token endpoints, the Azure app
+    client id (``96eecda7-…``), port ``8030``, and the
+    ``{app_id}/user_impersonation offline_access`` scope (PECOBLR-4120).
+  - ``azure-sp-m2m`` (M2M) → forward ``auth_type='azure-sp-m2m'`` with the
+    Azure service-principal ``azure_client_id`` / ``azure_client_secret``.
+    The kernel builds the Entra v2.0 token endpoint and the
+    ``{effective_app_id}/.default`` scope, and auto-discovers the tenant from
+    the workspace's ``/aad/auth`` redirect when ``azure_tenant_id`` is omitted
+    (Thrift parity). ``azure_workspace_resource_id`` is an optional add-on:
+    forward it and the kernel additionally sends the Azure management-token
+    header pair, so an RBAC-only SP (not a workspace member) can authenticate
+    (PECOBLR-4141).
 
 ``identity_federation_client_id`` is forwarded with whichever auth shape
 wins resolution. It selects mandatory SP-wide workload-identity token
@@ -134,6 +149,7 @@ def _extract_bearer_token(auth_provider: Optional[AuthProvider]) -> Optional[str
 def kernel_auth_kwargs(
     auth_provider: Optional[AuthProvider],
     auth_options: Optional[Dict[str, Any]] = None,
+    hostname: Optional[str] = None,
 ) -> Dict[str, Any]:
     """Build the kwargs passed to ``databricks_sql_kernel.Session(...)``.
 
@@ -154,8 +170,9 @@ def kernel_auth_kwargs(
        - a U2M ``auth_type`` (``databricks-oauth``) *and*
          ``oauth_client_secret`` together.
 
-    (``azure-oauth`` is rejected as unsupported before these guards —
-    PECOBLR-4120.)
+    (The Azure Entra auth types — ``azure-oauth`` and ``azure-sp-m2m`` —
+    are forwarded to the kernel's Azure-aware flows up front, before these
+    guards; see the module docstring.)
     1. **OAuth M2M** — ``oauth_client_id`` + ``oauth_client_secret``
        both present → forward raw creds to the kernel's ``oauth-m2m``.
     2. **PAT** — the built provider is (or wraps) an
@@ -168,7 +185,6 @@ def kernel_auth_kwargs(
        forwarding the connector's own OAuth app rather than the kernel's
        ``databricks-sql-connector`` default (PECOBLR-4039/4040). Unlike the
        Thrift path, a caller-supplied ``oauth_scopes`` is honored here.
-       ``azure-oauth`` is rejected as unsupported (PECOBLR-4120).
     4. **Custom credentials_provider** → ``NotSupportedError`` (opaque
        token source; no raw creds for the kernel to own).
     5. Anything else → ``NotSupportedError``.
@@ -188,24 +204,71 @@ def kernel_auth_kwargs(
     auth_type = opts.get("auth_type")
     has_m2m = bool(client_id and client_secret)
 
-    # azure-oauth (Azure AD U2M) is not yet supported on the kernel path.
-    # Reject it up front — before any M2M/U2M routing — so ANY azure-oauth
-    # request gets a clear "not supported" error rather than being silently
-    # misrouted (e.g. azure-oauth + client_id + secret would otherwise look
-    # like M2M). The kernel resolves OAuth endpoints only from the
-    # workspace-native OIDC config and has no Azure AD path, so the Thrift
-    # azure-oauth flow (AAD token endpoint + /user_impersonation scope, see
-    # AzureOAuthEndpointCollection) cannot be reproduced here. Forwarding an
-    # azure bundle would authenticate against the wrong endpoints, so we fail
-    # loudly at session-open. Tracked by PECOBLR-4120.
+    # Azure Entra (Azure AD) auth types route to the kernel's GENERIC OAuth
+    # flows with Azure values supplied as overrides — the kernel needs no
+    # Azure-specific code. Handled up front, keyed on the explicit auth_type,
+    # before the generic M2M/PAT/U2M routing below (azure-sp-m2m carries its
+    # creds in azure_* kwargs, not oauth_client_id/secret, so it would
+    # otherwise fall through to the final "unsupported" error).
+
+    # azure-oauth (Azure AD U2M): forward the selector; the KERNEL owns Azure
+    # resolution (it is the auth core). The kernel pins the workspace v2.0
+    # authorize/token endpoints (`{host}/oidc/oauth2/v2.0/{authorize,token}` —
+    # NOT the discovered `/oidc/v1/authorize`, which the workspace redirects to
+    # a malformed Entra URL), the Azure app client id, port 8030, and the
+    # `{app_id}/user_impersonation offline_access` scope. So this binding does
+    # NOT construct endpoints/scopes — it just passes `auth_type='azure-oauth'`
+    # plus any optional client_id / redirect_port passthrough. PECOBLR-4120.
     if auth_type == "azure-oauth":
-        raise NotSupportedError(
-            "use_kernel=True does not support auth_type='azure-oauth' (Azure "
-            "AD U2M) yet: the kernel resolves OAuth endpoints only from the "
-            "workspace-native OIDC configuration and cannot drive the Azure AD "
-            "authorization/token flow. Use the Thrift backend (default) for "
-            "azure-oauth. Tracked by PECOBLR-4120."
-        )
+        kwargs = {"auth_type": "azure-oauth"}
+        if client_id:
+            kwargs["client_id"] = client_id
+            redirect_port = opts.get("oauth_redirect_port")
+            if redirect_port is not None:
+                kwargs["redirect_ports"] = [_coerce_redirect_port(redirect_port)]
+        if federation_client_id:
+            kwargs["identity_federation_client_id"] = federation_client_id
+        return kwargs
+
+    # azure-sp-m2m (Azure service principal, client-credentials): forward the
+    # selector + Azure SP credentials; the KERNEL owns Azure resolution (it is
+    # the auth core). The kernel builds the Entra v2.0 token endpoint
+    # (`{login}/{tenant}/oauth2/v2.0/token`) and the `{effective_app_id}/.default`
+    # scope, and — when azure_tenant_id is omitted — auto-discovers the tenant
+    # from the workspace's /aad/auth redirect, matching the Thrift backend
+    # (so connect() is byte-identical between Thrift and use_kernel=True).
+    # PECOBLR-4141.
+    #
+    # azure_workspace_resource_id is an optional add-on: forward it and the
+    # kernel additionally fetches an Azure-management token and sends the
+    # X-Databricks-Azure-SP-Management-Token + X-Databricks-Azure-Workspace-
+    # Resource-Id pair, so an SP that holds only an Azure RBAC role (not a
+    # workspace member) can authenticate. Omit it (the common case) and the SP
+    # authenticates with the Databricks-audience data token alone.
+    if auth_type == "azure-sp-m2m":
+        azure_client_id = opts.get("azure_client_id")
+        azure_client_secret = opts.get("azure_client_secret")
+        if not (azure_client_id and azure_client_secret):
+            raise ProgrammingError(
+                "auth_type='azure-sp-m2m' requires azure_client_id and "
+                "azure_client_secret."
+            )
+        kwargs = {
+            "auth_type": "azure-sp-m2m",
+            "azure_client_id": azure_client_id,
+            "azure_client_secret": azure_client_secret,
+        }
+        # Optional passthroughs: the kernel auto-discovers the tenant when
+        # absent, and sends the data token alone when no resource id is set.
+        azure_tenant_id = opts.get("azure_tenant_id")
+        if azure_tenant_id:
+            kwargs["azure_tenant_id"] = azure_tenant_id
+        azure_workspace_resource_id = opts.get("azure_workspace_resource_id")
+        if azure_workspace_resource_id:
+            kwargs["azure_workspace_resource_id"] = azure_workspace_resource_id
+        if federation_client_id:
+            kwargs["identity_federation_client_id"] = federation_client_id
+        return kwargs
 
     # 0. Ambiguity guards — fail before any flow is chosen.
     if client_secret and opts.get("credentials_provider") is not None:

src/databricks/sql/backend/kernel/client.py

@@ -309,7 +309,11 @@ def open_session(
         auth_kwargs: Dict[str, Any] = {}
         tls_kwargs: Dict[str, Any] = {}
         try:
-            auth_kwargs = kernel_auth_kwargs(self._auth_provider, self._auth_options)
+            auth_kwargs = kernel_auth_kwargs(
+                self._auth_provider,
+                self._auth_options,
+                hostname=self._server_hostname,
+            )
             # Translate the connector's SSLOptions into the kernel's
             # ``tls_*`` Session kwargs. Empty when TLS is at defaults.
             tls_kwargs = _kernel_tls_kwargs(self._ssl_options)

src/databricks/sql/session.py

@@ -177,6 +177,19 @@ def _create_backend(
                 "identity_federation_client_id": kwargs.get(
                     "identity_federation_client_id"
                 ),
+                # Azure Entra SP credentials for the azure-sp-m2m path. The
+                # kernel owns Azure resolution (endpoint/scope/tenant discovery),
+                # so these raw kwargs are the only source; without threading them
+                # the bridge would fail with "requires azure_client_id". The
+                # tenant and workspace-resource-id are optional (kernel
+                # auto-discovers the tenant; the resource id gates the optional
+                # management token). Kernel-only; Thrift / SEA are unaffected.
+                "azure_client_id": kwargs.get("azure_client_id"),
+                "azure_client_secret": kwargs.get("azure_client_secret"),
+                "azure_tenant_id": kwargs.get("azure_tenant_id"),
+                "azure_workspace_resource_id": kwargs.get(
+                    "azure_workspace_resource_id"
+                ),
             }
             # Forward the connector's retry-tuning kwargs so the kernel's
             # own retry policy honours them (the kernel owns the retry