diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md new file mode 100644 index 00000000..30853562 --- /dev/null +++ b/CONNECTION_PARAMETERS.md @@ -0,0 +1,192 @@ +# Connection parameter reference + +This document lists **every public connection / session parameter** the +Node.js connector accepts, and — because the driver ships two backends — +whether each parameter is honored on the **Thrift** backend (the default), +the **SEA / Kernel** backend (opt-in via `useKernel: true`), or both. + +The goal is to make protocol gaps explicit: a parameter honored on one +backend but ignored (or rejected) on the other is called out in the **Note** +column. + +> **Backend selection.** The connector defaults to Thrift. The SEA backend is +> selected by passing `useKernel: true`, an **internal, unstable (M0)** option +> that is intentionally absent from the published `.d.ts` and may be removed +> without notice (`lib/contracts/InternalConnectionOptions.ts`). Treat every +> "Kernel" column below as describing an experimental path. + +## Legend + +| Symbol | Meaning | +| ------ | ------------------------------------------------------------------- | +| ✅ | Honored — the option is read and forwarded to the backend. | +| ❌ | Ignored or rejected — see the Note column. | +| ⚠️ | Partially supported or behaves differently from the other backend. | +| — | Not applicable / no public equivalent / no default on this backend. | + +## Sources of truth + +- Public option shape ← `lib/contracts/IDBSQLClient.ts` (`ConnectionOptions`, + `AuthOptions`, `OpenSessionRequest`) and `lib/contracts/IDBSQLSession.ts` + (`ExecuteStatementOptions`). +- Default values ← `DBSQLClientDefaults` (`lib/DBSQLClient.ts`) and + `DEFAULT_TELEMETRY_CONFIG` (`lib/telemetry/types.ts`). +- Internal / kernel-only flags ← `lib/contracts/InternalConnectionOptions.ts`. +- Thrift wiring ← `lib/DBSQLClient.ts` (`getConnectionOptions`, + `createAuthProvider`), `lib/thrift-backend/ThriftBackend.ts`, + `lib/connection/connections/HttpConnection.ts`. +- Kernel wiring ← `lib/kernel/KernelAuth.ts` (`buildKernelConnectionOptions`, + `buildKernelTlsOptions`, `buildKernelHttpOptions`, `buildKernelProxyOptions`, + `buildKernelRetryOptions`), `lib/kernel/KernelBackend.ts`, + `lib/kernel/KernelSessionBackend.ts`. +- Kernel-core parameter semantics ← databricks-sql-kernel + [`docs/connection-parameters.md`](https://github.com/databricks/databricks-sql-kernel/pull/184). + +--- + +## Connection identity + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------- | -------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `host` | `string` | ✅ | ✅ | — (required) | Required on both. | +| `path` | `string` | ✅ | ✅ | — (required) | HTTP path; on the kernel path the org id is auto-parsed from a `?o=` query param and sent as `x-databricks-org-id`. | +| `port` | `number` | ✅ | ⚠️ | `443` | Thrift defaults to `443` (`options.port \|\| 443`). The kernel derives host/port from `host` + `path`; a standalone `port` is not separately threaded. | +| `userAgentEntry` | `string` | ✅ | ✅ | — | Folded into the composed `User-Agent` on both. | + +## Authentication + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------------------------- | -------------------------------------------------------------------------- | :------: | :------: | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `authType` — supported on both backends | `'access-token'` \| `'databricks-oauth'` | ✅ | ✅ | `'access-token'` | The two auth modes both backends accept. `access-token` uses `token` (PAT) and is the default when `authType` is omitted. `databricks-oauth` covers M2M (`oauthClientId` + `oauthClientSecret`; kernel runs OIDC discovery + client-credentials internally) and U2M (browser; no secret — kernel U2M differs slightly, see the OAuth sub-option rows below). | +| `authType` — Thrift-only | `'custom'` \| `'token-provider'` \| `'external-token'` \| `'static-token'` | ✅ | ❌ | — | **Thrift-only.** `custom` (`provider: IAuthentication`), `token-provider` (`tokenProvider: ITokenProvider`), `external-token` (`getToken: TokenCallback`), `static-token` (`staticToken`). The kernel throws `unsupported auth mode` for all four — it supports only the two modes above. | +| `oauthScopes` | `Array` | ❌ | ✅ | U2M `['sql','offline_access']`, M2M `['all-apis']` | **Thrift ignores `oauthScopes`** — `createAuthProvider` never threads it into `DatabricksOAuth`, so `authenticate()` always falls back to `defaultOAuthScopes` (`['sql','offline_access']`). Only the kernel honors a custom `oauthScopes`; its defaults happen to match Thrift's fallback. | +| `oauthClientId` (U2M) | `string` | ✅ | ✅ | napi default `client_id` when absent | The kernel adapter (`buildKernelConnectionOptions`) forwards a custom `oauthClientId` verbatim on the U2M arm; when it is absent the napi binding applies its own default `client_id`. Whether the native binding then honors or rejects a custom id is not observable from this repo — the TypeScript layer neither hardcodes an id nor rejects one. | +| `oauthClientId` + no secret | `string` | ✅ (U2M) | ✅ (U2M) | — | **Parity.** The kernel keys flow selection off `oauthClientSecret` presence exactly like Thrift, so `oauthClientId` + no secret routes to **U2M** (with the id forwarded) — it does **not** throw an M2M "secret required" error. | +| `azureTenantId` / `useDatabricksOAuthInAzure` | `string` / `boolean` | ✅ | ❌ | — | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | +| `persistence` (custom OAuth token store) | `OAuthPersistence` | ✅ | ❌ | — | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | +| `enableTokenFederation` / `federationClientId` | `boolean` / `string` | ✅ | ❌ | `false` / — | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | + +## HTTP client, proxy, retries + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------- | ------------------------ | :----: | :----: | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` (15 min) | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | +| `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | +| `noProxy` (internal) | `string` | ❌ | ✅ | — | **Thrift ignores `noProxy`** — `getConnectionOptions` never threads it, and `getProxyForUrl: () => proxyUrl` returns the proxy for every URL (no bypass-list logic). Only the kernel honors it, forwarded as `bypassHosts`. | +| `customHeaders` | `Record` | ⚠️ | ✅ | — | **Thrift: out-of-band requests only.** `getConnectionOptions` never threads `customHeaders` into the Thrift query transport (it sets only `User-Agent`); the map is consumed by `buildCustomHeaders` for driver-owned telemetry POSTs / feature-flag GETs and SPOG `x-databricks-org-id` injection, not the primary transport or OAuth/OIDC token requests. Kernel applies them to every request (dropping reserved `Authorization` / `x-databricks-org-id`, rejecting CR/LF/NUL, appending the connector `User-Agent` last). | +| `retryMaxAttempts` | `number` | ✅ | ✅ | `5` | Total-attempt semantics on both; kernel converts to retries-after-first. | +| `retriesTimeout` | `number` (ms) | ✅ | ✅ | `900000` (15 min) | Kernel converts ms → whole seconds. | +| `retryDelayMin` | `number` (ms) | ✅ | ✅ | `1000` (1 s) | Kernel converts ms → seconds. | +| `retryDelayMax` | `number` (ms) | ✅ | ✅ | `60000` (60 s) | Kernel converts ms → seconds. | +| `maxConnections` (pool size) | `number` (internal) | ❌ | ✅ | kernel default | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | + +## TLS / SSL + +> **Thrift TLS is public and secure-by-default.** `checkServerCertificate`, +> `customCaCert`, `clientCert`, and `clientKey` are declared on the public +> `ConnectionOptions` (`lib/contracts/IDBSQLClient.ts`) and honored on the +> Thrift (default) backend. `getConnectionOptions` (`lib/DBSQLClient.ts`) maps +> `customCaCert` → `ca` (**additively**, on top of `tls.rootCertificates` + +> `NODE_EXTRA_CA_CERTS`), `clientCert` → `cert`, and `clientKey` → `key`, with a +> both-or-neither mTLS guard and `normalizePemBytes` PEM validation. It sets +> `rejectUnauthorized: options.checkServerCertificate ?? true`, and +> `HttpConnection` reads `this.options.rejectUnauthorized ?? true` — i.e. +> **verification is on by default**; you must pass `checkServerCertificate: +false` to accept any certificate. + +| Option | Type | Thrift | Kernel | Default Value | Note | +| -------------------------------- | ------------------ | :----: | :----: | ------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| `checkServerCertificate` | `boolean` | ✅ | ✅ | `true` | Master verify toggle on both, secure-by-default: Thrift uses `options.checkServerCertificate ?? true`. Set `false` to accept-anything. | +| `checkServerCertificateHostname` | `boolean` (kernel) | ❌ | ✅ | `true` | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. No Thrift equivalent. | +| `customCaCert` | `Buffer \| string` | ✅ | ✅ | — | PEM. Honored on both; added on top of system roots. Thrift maps it to `ca` additively (`tls.rootCertificates` + `NODE_EXTRA_CA_CERTS`). | +| `clientCert` (mTLS) | `Buffer \| string` | ✅ | ✅ | — | Honored on both; maps to `cert`. Must be paired with `clientKey`; supplying one alone throws a client-side error. | +| `clientKey` (mTLS) | `Buffer \| string` | ✅ | ✅ | — | Honored on both; maps to `key`. Must be paired with `clientCert`. PKCS#8 recommended. | + +## Results & type rendering + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ----------------------------- | --------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `preserveBigNumericPrecision` | `boolean` | ✅ | ✅ | `false` | DECIMAL → exact string, BIGINT → `bigint` on both. | +| `enableMetricViewMetadata` | `boolean` | ✅ | ⚠️ | `false` | **Auto-injected for both backends** in `DBSQLClient.openSession`, which sets `spark.sql.thriftserver.metadata.metricview.enabled=true` on `request.configuration` before dispatch. `KernelBackend` folds that into `sessionOptions.sessionConf`, so the conf **does** reach the kernel session config. (`ThriftBackend.ts` performs a second, redundant injection on the Thrift path.) The kernel-side gap is that the key is a non-allowlisted session conf, so it is likely dropped by the kernel's case-insensitive allowlist (see "Session defaults") — not that it is never injected. | + +## Session defaults (`openSession(request)`) + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ------------------------------- | --------------------------------------------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `initialCatalog` | `string` | ✅ | ✅ | — | Kernel → `DefaultOpts.catalog` on `CreateSession`. | +| `initialSchema` | `string` | ✅ | ✅ | — | Kernel → `DefaultOpts.schema`. | +| `configuration` (session confs) | `{ [key: string]: string }` | ✅ | ⚠️ | — | Kernel matches keys **case-insensitively against an allowlist** and uppercases them; **non-allowlisted keys are dropped with a warning**. Thrift forwards the map more freely. | +| `queryTags` | `Record` | ✅ | ✅ | — | Both serialize into the reserved `QUERY_TAGS` conf; `queryTags` takes precedence over `configuration.QUERY_TAGS`. | + +## Telemetry + +All `telemetry*` options live in the driver-layer `ClientConfig`, not in either +backend, so they are read regardless of `useKernel`. Defaults are sourced from +`DEFAULT_TELEMETRY_CONFIG` (`lib/telemetry/types.ts`). + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------------- | ------------- | :----: | :----: | ------------- | --------------------------------------------------- | +| `telemetryEnabled` | `boolean` | ✅ | ✅ | `true` | Enabled by default, gated by a server feature flag. | +| `telemetryBatchSize` | `number` | ✅ | ✅ | `100` | Metrics per export batch. | +| `telemetryFlushIntervalMs` | `number` (ms) | ✅ | ✅ | `5000` | Periodic flush interval. | +| `telemetryMaxRetries` | `number` | ✅ | ✅ | `3` | Export retry attempts. | +| `telemetryAuthenticatedExport` | `boolean` | ✅ | ✅ | `true` | Export via the authenticated endpoint. | +| `telemetryCircuitBreakerThreshold` | `number` | ✅ | ✅ | `5` | Consecutive failures before the breaker opens. | +| `telemetryCircuitBreakerTimeout` | `number` (ms) | ✅ | ✅ | `60000` | Breaker open duration. | +| `telemetryCloseTimeoutMs` | `number` (ms) | ✅ | ✅ | `2000` | Caps `client.close()` shutdown latency. | +| `telemetryMaxStatementMetrics` | `number` | ✅ | ✅ | `5000` | Hard cap for the per-statement aggregation map. | +| `telemetryMaxPendingMetrics` | `number` | ✅ | ✅ | `500` | Cap on buffered, not-yet-exported metrics. | + +> **Telemetry _events_ differ by backend.** The config knobs above are +> backend-agnostic (driver layer), but the kernel owns result fetching +> internally, so it emits fewer per-statement / CloudFetch telemetry events than +> the Thrift path. + +## Per-statement options (`session.executeStatement(sql, options)`) + +| Option | Type | Thrift | Kernel | Default Value | Note | +| -------------------------------------- | ----------------------------------- | :----: | :----: | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `maxRows` | `number \| bigint \| Int64 \| null` | ✅ | ✅ | `100000` | Kernel applies it at fetch time in the facade rather than on the request. | +| `useCloudFetch` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | +| `useLZ4Compression` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | +| `stagingAllowedLocalPath` (volume ops) | `string \| string[]` | ✅ | ❌ | — | **Thrift-only.** Not supported on the kernel path. | +| `runAsync` | `boolean` | ⚠️ | ✅ | `false` | Deprecated. **Thrift:** no-op — the path always submits async (`runAsync: true` on the wire) and polls during fetch; the option is not read. **Kernel:** selects the execution path — `false`/unset (default) runs the blocking direct-results path (cancellable mid-compute); `true` submits and polls (returns a pending handle). | + +--- + +## Summary of gaps + +### Supported on Thrift, missing / ignored on Kernel + +1. `enableMetricViewMetadata` — auto-injected for both backends in + `DBSQLClient.openSession`, but the conf key is likely dropped by the + kernel's session-conf allowlist, so it has no effect on the kernel path. +2. Auth types `custom`, `token-provider`, `external-token`, `static-token`, + plus `enableTokenFederation` / `federationClientId`. +3. `azureTenantId` / `useDatabricksOAuthInAzure` (Azure-direct OAuth). +4. `persistence` (custom OAuth token store). +5. SOCKS proxies. +6. Per-statement `useCloudFetch`, `useLZ4Compression`, + `stagingAllowedLocalPath`. + +### Supported on Kernel, no Thrift public equivalent + +1. `maxConnections` (connection-pool sizing). +2. `checkServerCertificateHostname` — the independent hostname-vs-SNI check has + no public Thrift equivalent. (The other TLS controls — + `checkServerCertificate`, `customCaCert`, `clientCert`, `clientKey` — **are** + public and honored on the Thrift backend, which verifies certificates by + default via `checkServerCertificate ?? true`; see the TLS / SSL section.) + +### Behavioral divergences to watch + +- **U2M flow selection** keys off `oauthClientSecret` presence on the kernel + path, matching Thrift: no secret ⇒ U2M, secret present ⇒ M2M. A custom + `oauthClientId` (with no secret) is forwarded on the U2M arm rather than + triggering an M2M "secret required" error. +- **`socketTimeout: 0`** means "indefinite" on Thrift but is dropped on the + kernel path (kernel default kept). +- **`configuration`** is allowlist-filtered on the kernel path but forwarded + more freely on Thrift. + +> All kernel-path behavior reflects the **M0 stub** and is subject to change. diff --git a/README.md b/README.md index 5d31e138..0eab0fab 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,12 @@ client }); ``` +## Configuration + +See [CONNECTION_PARAMETERS.md](CONNECTION_PARAMETERS.md) for every connection, +session, and per-statement parameter the driver accepts, and whether each one +applies to the Thrift backend (default), the SEA backend, or both. + ## Telemetry The driver emits connection, statement, and CloudFetch metrics plus diff --git a/lib/kernel/KernelAuth.ts b/lib/kernel/KernelAuth.ts index 289c3e47..7cf99afa 100644 --- a/lib/kernel/KernelAuth.ts +++ b/lib/kernel/KernelAuth.ts @@ -451,21 +451,18 @@ export function buildKernelHttpOptions(options: ConnectionOptions): KernelHttpOp * a browser, listens on localhost:8030, exchanges the code, persists * to `~/.config/databricks-sql-kernel/oauth/{sha256}.json`). * - * **Flow selection — DELIBERATE DIVERGENCE FROM THRIFT.** Thrift's + * **Flow selection — MIRRORS THRIFT.** Thrift's * `DBSQLClient.createAuthProvider` (`DBSQLClient.ts:216`) keys off the * *secret* (`oauthClientSecret === undefined ? U2M : M2M`), so a custom - * `oauthClientId` with no secret runs U2M with that id. kernel instead keys - * off `oauthClientId` *presence* (id present → M2M, absent → U2M). The - * trade-off: keying off the id means a caller who set an id but - * typoed/forgot the secret gets the actionable M2M "secret is required" - * error instead of being silently routed to U2M (which would hide their - * intent). The cost is two real behavioural gaps vs Thrift: - * 1. `oauthClientId` + no secret → Thrift runs U2M; kernel throws - * `AuthenticationError` (M2M secret required). - * 2. kernel U2M has NO custom-client-id support — the kernel hardcodes - * `client_id = "databricks-cli"`, and kernel rejects any `oauthClientId` - * on the U2M arm. Thrift U2M honours a custom `clientId`. - * Both are documented limitations of the M0 kernel OAuth surface, not bugs. + * `oauthClientId` with no secret runs U2M with that id. This adapter keys + * off the same signal: `oauthClientSecret === undefined` ⇒ U2M, else M2M + * (see `buildKernelConnectionOptions` below). A custom `oauthClientId` is + * forwarded verbatim on the U2M arm; when absent the napi binding applies + * its own default `client_id`. The adapter therefore does NOT throw an M2M + * "secret required" error for `oauthClientId` + no secret, and does NOT + * reject a custom `oauthClientId` on U2M — those decisions, if the native + * binding makes them, happen below the TypeScript layer and are not + * observable from this repo. * * Out of scope on the OAuth paths (rejected with a clear error): * - `azureTenantId` / `useDatabricksOAuthInAzure` → Microsoft Entra diff --git a/lib/kernel/KernelBackend.ts b/lib/kernel/KernelBackend.ts index e0ee420f..221e7beb 100644 --- a/lib/kernel/KernelBackend.ts +++ b/lib/kernel/KernelBackend.ts @@ -57,7 +57,8 @@ export interface KernelBackendOptions { * (slash-prepended httpPath, AuthenticationError on missing token or * blank OAuth credentials, HiveDriverError on unsupported authType / * Azure-direct / ambiguous credential combinations). M2M and U2M - * routing key off `oauthClientId` presence; see KernelAuth.ts. + * routing key off `oauthClientSecret` presence (mirroring Thrift); see + * KernelAuth.ts. * * **Why we don't use IClientContext's connectionProvider here:** that * provider is the Thrift HTTP transport. The kernel owns its own